From fc81ecafd62de4daf310b0185b23c6d43603d237 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Mateusz=20Gruszczy=C5=84ski?=
 <mateusz.gruszczynski@firma.interia.pl>
Date: Sat, 31 Oct 2026 09:26:44 +0100
Subject: [PATCH] todo

---
 TODO.md | 806 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 806 insertions(+)
 create mode 100644 TODO.md

diff --git a/TODO.md b/TODO.md
new file mode 100644
index 0000000..82b952f
--- /dev/null
+++ b/TODO.md
@@ -0,0 +1,806 @@
+# TODO — cluster-safe storage and external search backends
+
+## Goal
+
+Remove the hard dependency on a shared local `data_dir` for multi-node RustBox deployments and make filesystem-backed data explicitly pluggable.
+
+The target architecture should allow fully stateless RustBox application nodes using:
+
+- a shared MySQL/PostgreSQL database for mailbox metadata and cached message bodies,
+- optional Redis for shared sessions/events,
+- shared blob/object storage for draft attachments and other binary data,
+- an optional external full-text indexer,
+- no writable node-local state required for normal cluster operation.
+
+This is a follow-up to the 0.1.2 database/cluster audit. It is not implemented in 0.1.2.
+
+---
+
+## Current state in 0.1.2
+
+Shared through SQL:
+
+- mailbox/folder metadata,
+- cached message bodies,
+- application data stored in MySQL/PostgreSQL,
+- persistent sessions when SQL is selected.
+
+Filesystem-backed under `data_dir/<user_hash>/`:
+
+- `attachments/<draft_id>/<attachment_id>` — draft attachments,
+- `tantivy/` — local full-text index,
+- `pgp/private/` — OpenPGP private-key material,
+- legacy `db.sqlite` — migration source only.
+
+Current cluster restrictions:
+
+```toml
+cluster_mode = true
+cluster_shared_data_dir = true
+search_index_backend = "disabled"
+pgp_enabled = false
+```
+
+`cluster_shared_data_dir = true` is currently required because draft attachments are path-based and local Tantivy/PGP state is not cluster-safe.
+
+---
+
+# 1. Shared data / blob storage abstraction
+
+## Required design
+
+Introduce a storage interface independent from `data_dir`.
+
+Conceptual API:
+
+```text
+BlobStorage
+  put(key, stream, metadata)
+  get(key) -> stream
+  delete(key)
+  exists(key)
+  stat(key)
+```
+
+Application code must store a logical object key in SQL, not an absolute filesystem path.
+
+Suggested key layout:
+
+```text
+users/<user_hash>/drafts/<draft_id>/attachments/<attachment_id>
+```
+
+The backend is responsible for translating that logical key to a filesystem path or object-store key.
+
+## Variant A — local filesystem
+
+Target: single-node installations.
+
+```toml
+[storage]
+backend = "filesystem"
+path = "/var/lib/rustbox/data"
+```
+
+Properties:
+
+- simplest deployment,
+- no external dependency,
+- valid for one RustBox node,
+- not considered cluster-safe unless the path is actually on shared storage.
+
+## Variant B — shared POSIX filesystem
+
+Target: existing clusters with NFS/CephFS/GlusterFS or another shared mount.
+
+```toml
+[storage]
+backend = "filesystem"
+path = "/srv/rustbox-shared"
+shared = true
+```
+
+Supported use:
+
+- draft attachments,
+- other blob-like application files.
+
+Do **not** treat a shared filesystem as a supported way to run multiple concurrent Tantivy writers. Tantivy indexing must remain a separate concern.
+
+Operational notes:
+
+- identical mount path is not required once SQL stores logical keys rather than absolute paths,
+- application startup should check read/write access,
+- health endpoint should expose storage availability,
+- temporary uploads must be written atomically and cleaned after failure.
+
+## Variant C — S3-compatible object storage
+
+Target: recommended multi-node/stateless deployment.
+
+Supported implementations should include any sufficiently compatible S3 API, for example:
+
+- AWS S3,
+- MinIO,
+- Ceph RGW,
+- other S3-compatible services.
+
+Suggested configuration:
+
+```toml
+[storage]
+backend = "s3"
+bucket = "rustbox"
+region = "eu-central-1"
+endpoint = "https://s3.example.net" # optional for AWS, required for many compatible services
+prefix = "rustbox/"
+path_style = false
+
+# Credentials should preferably come from environment/instance identity,
+# not plaintext TOML.
+```
+
+Required behavior:
+
+- streaming upload/download,
+- multipart upload for large objects where useful,
+- checksum/ETag validation where reliable,
+- abort incomplete multipart uploads,
+- retry transient errors,
+- configurable request/connect timeouts,
+- TLS verification enabled by default,
+- optional server-side encryption configuration,
+- lifecycle support for abandoned temporary objects,
+- no public bucket requirement.
+
+The database should keep object metadata and logical object keys. RustBox nodes must not depend on any local copy being present.
+
+## Variant D — additional cloud/object backends
+
+Future adapters may add:
+
+```toml
+backend = "azure_blob"
+```
+
+or:
+
+```toml
+backend = "gcs"
+```
+
+Do not hard-code cloud-specific behavior into attachment code. Keep it behind the storage interface.
+
+Priority should be:
+
+1. `filesystem`,
+2. `s3`,
+3. additional adapters only when there is demand.
+
+---
+
+# 2. Cluster behavior after storage abstraction
+
+When `storage.backend = "s3"`, RustBox should allow:
+
+```toml
+cluster_mode = true
+cluster_shared_data_dir = false
+```
+
+The cluster validation should become capability-based instead of checking only `cluster_shared_data_dir`.
+
+Expected rules:
+
+| Configuration | Cluster allowed | Notes |
+|---|---:|---|
+| local filesystem | no | unless explicitly marked/verified as shared |
+| shared POSIX filesystem | yes | attachments/blobs only |
+| S3-compatible storage | yes | preferred stateless mode |
+| external index + S3 | yes | full cluster mode |
+| local Tantivy | no | single-node only |
+| index disabled | yes | SQL search only |
+
+Suggested validation logic:
+
+```text
+cluster_mode = false
+    -> filesystem + Tantivy allowed
+
+cluster_mode = true
+    -> blob storage must be cluster-safe
+    -> local Tantivy forbidden
+    -> PGP requires a cluster-safe key backend or must be disabled
+```
+
+---
+
+# 3. Search/index backend abstraction
+
+Search must not be coupled to the blob storage backend.
+
+Introduce a separate interface such as:
+
+```text
+SearchBackend
+  index_message(document)
+  delete_message(message_id)
+  search(query, filters, paging)
+  health()
+  rebuild(...)
+```
+
+Suggested configuration:
+
+```toml
+[search]
+backend = "disabled"
+```
+
+Possible values:
+
+```text
+disabled
+local_tantivy
+quickwit
+opensearch
+solr
+```
+
+Additional adapters can be added later without changing mail/cache storage.
+
+---
+
+## Search variant 0 — disabled
+
+```toml
+[search]
+backend = "disabled"
+```
+
+Use cases:
+
+- cluster without external search service,
+- minimal deployments,
+- recovery/degraded mode.
+
+Behavior:
+
+- SQL-backed metadata/header search remains available,
+- deep body full-text search is unavailable,
+- no index files on application nodes.
+
+This must remain a fully supported configuration.
+
+---
+
+## Search variant 1 — local Tantivy
+
+```toml
+[search]
+backend = "local_tantivy"
+path = "/var/lib/rustbox/data"
+```
+
+Use case:
+
+- single-node RustBox.
+
+Rules:
+
+- allowed only when `cluster_mode = false`,
+- current Tantivy behavior can be retained behind the new `SearchBackend` interface,
+- index remains rebuildable from authoritative data,
+- the index must never be treated as the only copy of mailbox data.
+
+Do not support multi-node writers against one shared Tantivy directory.
+
+---
+
+## Search variant 2 — Quickwit
+
+```toml
+[search]
+backend = "quickwit"
+url = "http://quickwit:7280"
+index = "rustbox-mails"
+```
+
+Why evaluate it:
+
+- written in Rust,
+- based on Tantivy,
+- designed around distributed search and object storage,
+- conceptually close to the current local Tantivy implementation.
+
+Required evaluation before implementation:
+
+- message update/delete semantics,
+- latency for newly indexed mail,
+- tenant/user isolation strategy,
+- reindex cost,
+- index schema evolution,
+- attachment/body size limits,
+- failure/retry behavior,
+- suitability for mutable mailbox state rather than only append-heavy workloads.
+
+Do not select Quickwit solely because it uses Tantivy; validate mail-specific update/delete behavior first.
+
+---
+
+## Search variant 3 — OpenSearch
+
+```toml
+[search]
+backend = "opensearch"
+urls = ["https://search-1.example.net:9200", "https://search-2.example.net:9200"]
+index_prefix = "rustbox"
+```
+
+Why support it:
+
+- mature distributed indexing/search,
+- common in existing infrastructure,
+- strong filtering and full-text capabilities,
+- suitable for larger installations.
+
+Trade-offs:
+
+- heavier operational footprint,
+- JVM memory requirements,
+- index lifecycle/sharding needs administration.
+
+OpenSearch is a good enterprise/existing-infrastructure option, but should not become mandatory for RustBox.
+
+---
+
+## Search variant 4 — Solr / SolrCloud
+
+```toml
+[search]
+backend = "solr"
+url = "https://solr.example.net/solr/rustbox"
+```
+
+Why keep it as an option:
+
+- mature Lucene full-text engine,
+- still used in established environments,
+- supports distributed SolrCloud deployments.
+
+Trade-offs:
+
+- JVM-based,
+- cluster operation is not simpler than lightweight alternatives,
+- adds a separate operational stack.
+
+Treat Solr as an optional adapter for environments that already operate it, not as the default RustBox cluster indexer.
+
+---
+
+# 4. Recommended deployment variants
+
+## Variant 1 — simple single node
+
+```text
+RustBox
+ ├─ SQLite/MySQL/PostgreSQL
+ ├─ local filesystem attachments
+ └─ local Tantivy
+```
+
+Example:
+
+```toml
+cluster_mode = false
+
+[storage]
+backend = "filesystem"
+path = "/var/lib/rustbox/data"
+
+[search]
+backend = "local_tantivy"
+```
+
+Best for small installations.
+
+---
+
+## Variant 2 — cluster with shared filesystem
+
+```text
+RustBox node 1 ─┐
+RustBox node 2 ─┼── MySQL/PostgreSQL
+RustBox node 3 ─┘
+       │
+       └──────── shared NFS/CephFS for attachments
+
+Search: disabled or external
+```
+
+Example:
+
+```toml
+cluster_mode = true
+
+[storage]
+backend = "filesystem"
+path = "/srv/rustbox-shared"
+shared = true
+
+[search]
+backend = "disabled"
+```
+
+This is the lowest-change migration path from 0.1.2.
+
+---
+
+## Variant 3 — stateless cluster with S3, no external indexer
+
+```text
+RustBox node 1 ─┐
+RustBox node 2 ─┼── MySQL/PostgreSQL
+RustBox node 3 ─┘
+       │
+       └──────── S3 / MinIO / Ceph RGW
+
+Search: SQL-only
+```
+
+Example:
+
+```toml
+cluster_mode = true
+
+[storage]
+backend = "s3"
+bucket = "rustbox"
+endpoint = "https://minio.example.net"
+
+[search]
+backend = "disabled"
+```
+
+This should be a first-class supported production configuration.
+
+---
+
+## Variant 4 — stateless cluster with S3 + external indexer
+
+```text
+                    ┌── MySQL/PostgreSQL
+RustBox nodes ──────┼── S3-compatible blob storage
+                    └── external search cluster
+```
+
+Example with Quickwit:
+
+```toml
+cluster_mode = true
+
+[storage]
+backend = "s3"
+bucket = "rustbox"
+endpoint = "https://s3.example.net"
+
+[search]
+backend = "quickwit"
+url = "http://quickwit:7280"
+index = "rustbox-mails"
+```
+
+Example with OpenSearch:
+
+```toml
+[search]
+backend = "opensearch"
+urls = ["https://search.example.net:9200"]
+index_prefix = "rustbox"
+```
+
+This is the preferred architecture for larger multi-node installations requiring deep full-text search.
+
+---
+
+# 5. Durable indexing pipeline
+
+Do not make correctness depend on an HTTP request successfully updating the external index synchronously.
+
+Add a durable indexing queue/outbox in the shared SQL database.
+
+Conceptual flow:
+
+```text
+mail/cache transaction
+      │
+      ├── update authoritative SQL data
+      └── append search_outbox event
+                    │
+                    v
+             index worker(s)
+                    │
+                    v
+       Quickwit/OpenSearch/Solr
+```
+
+Required properties:
+
+- idempotent events,
+- retry with backoff,
+- dead-letter/error visibility,
+- safe concurrent workers,
+- delete events,
+- full reindex command,
+- index version/schema version,
+- ability to rebuild the index from authoritative SQL/IMAP data,
+- index lag metric.
+
+The external index must be treated as derived/rebuildable state, never as the source of truth.
+
+---
+
+# 6. PGP/key storage in cluster mode
+
+PGP private keys should not automatically follow the generic attachment/blob backend.
+
+Current safe rule remains:
+
+```toml
+pgp_enabled = false
+```
+
+for cluster mode until a dedicated key-storage design exists.
+
+Future variants to evaluate:
+
+1. encrypted keys in the shared SQL database,
+2. encrypted objects in S3 with a separate master-key/KMS strategy,
+3. Vault/HSM/KMS-backed key handling,
+4. dedicated secrets backend.
+
+Requirements:
+
+- encryption at rest independent of ordinary object-storage ACLs,
+- no plaintext private key in shared object storage,
+- identical behavior from every RustBox node,
+- rotation/recovery procedure,
+- auditability.
+
+Do not make `cluster_shared_data_dir` the solution for private-key consistency.
+
+---
+
+# 7. WebSocket / IMAP IDLE cluster events
+
+Separate from storage and search, the current in-process event bus is node-local.
+
+TODO:
+
+- add optional cluster event backend,
+- initial candidate: Redis Pub/Sub or Redis Streams,
+- propagate mailbox/notification invalidations across nodes,
+- retain local fast-path for single-node mode,
+- define behavior during broker outages.
+
+This is not required for SQL cache consistency, but is required for reliable real-time UI updates across nodes.
+
+---
+
+# 8. Configuration compatibility
+
+Keep backward compatibility where possible.
+
+Legacy:
+
+```toml
+data_dir = "/var/lib/rustbox/data"
+search_index_backend = "tantivy"
+```
+
+Migration target:
+
+```toml
+[storage]
+backend = "filesystem"
+path = "/var/lib/rustbox/data"
+
+[search]
+backend = "local_tantivy"
+```
+
+Suggested transition:
+
+- accept legacy keys for at least one release,
+- emit deprecation warnings,
+- map legacy values internally to the new backend configuration,
+- document exact removal version before deleting old settings.
+
+---
+
+# 9. Migration of existing attachments
+
+Provide an explicit migration command rather than silently moving data during normal startup.
+
+Example target CLI:
+
+```text
+rustbox storage migrate \
+  --from filesystem:/var/lib/rustbox/data \
+  --to s3:rustbox \
+  --verify
+```
+
+Requirements:
+
+- resumable,
+- idempotent,
+- checksum/size verification,
+- dry-run mode,
+- progress reporting,
+- do not delete source objects until verification passes,
+- update SQL references transactionally or through a safe staged migration,
+- rollback/retry documentation.
+
+---
+
+# 10. Health, metrics and operations
+
+Expose at least:
+
+```text
+storage backend status
+storage request failures
+storage latency
+index backend status
+index queue depth
+index lag
+index failures/retries
+last successful index operation
+```
+
+Startup behavior:
+
+- invalid cluster/storage combination -> hard configuration error,
+- external index unavailable -> application may start in degraded mode if configured,
+- shared SQL unavailable -> application startup/request handling must fail clearly,
+- object storage unavailable -> mail reading from SQL may continue, but attachment operations must fail explicitly.
+
+Suggested option:
+
+```toml
+[search]
+required = false
+```
+
+When `required = false`, RustBox can run with SQL search while the external index is unavailable.
+
+---
+
+# 11. Security requirements
+
+For object storage:
+
+- private buckets/containers only,
+- least-privilege credentials,
+- environment/instance/workload identity preferred over TOML secrets,
+- TLS by default,
+- optional custom CA bundle for private S3-compatible services,
+- prevent object-key traversal/injection,
+- server-side encryption option,
+- never expose backend credentials to the browser.
+
+For external search:
+
+- authenticated TLS connection,
+- per-user/tenant filtering must be enforced server-side by RustBox,
+- never trust client-provided tenant/user IDs,
+- index documents must not allow cross-user search leakage,
+- document deletion must propagate to the index.
+
+---
+
+# 12. Implementation order
+
+## Phase 1 — storage abstraction
+
+- [ ] introduce `BlobStorage` interface,
+- [ ] convert draft attachments from absolute paths to logical object keys,
+- [ ] implement filesystem backend,
+- [ ] retain existing behavior for single-node installations,
+- [ ] add storage health checks.
+
+## Phase 2 — S3 backend
+
+- [ ] implement S3-compatible backend,
+- [ ] streaming upload/download,
+- [ ] retries/timeouts/checksums,
+- [ ] MinIO integration validation,
+- [ ] AWS S3 integration validation,
+- [ ] Ceph RGW compatibility validation,
+- [ ] migration command from filesystem to S3,
+- [ ] remove hard `cluster_shared_data_dir` requirement when storage backend is cluster-safe.
+
+## Phase 3 — search abstraction
+
+- [ ] introduce `SearchBackend` interface,
+- [ ] move current Tantivy code behind `local_tantivy`,
+- [ ] preserve `disabled` backend,
+- [ ] reject `local_tantivy` in cluster mode,
+- [ ] define canonical indexed message document/schema.
+
+## Phase 4 — durable index events
+
+- [ ] add SQL search outbox/queue,
+- [ ] add index worker locking/concurrency,
+- [ ] retries/backoff,
+- [ ] delete events,
+- [ ] reindex command,
+- [ ] index lag/error metrics.
+
+## Phase 5 — external index adapters
+
+Recommended evaluation order:
+
+- [ ] Quickwit proof of concept,
+- [ ] OpenSearch adapter,
+- [ ] compare update/delete/reindex behavior and operational footprint,
+- [ ] choose documented recommended backend,
+- [ ] Solr adapter only if there is real deployment demand.
+
+## Phase 6 — cluster PGP
+
+- [ ] design dedicated encrypted key backend,
+- [ ] keep PGP disabled in cluster mode until completed,
+- [ ] evaluate Vault/KMS/SQL-encrypted/S3-encrypted variants,
+- [ ] add migration procedure for existing local keys.
+
+## Phase 7 — cross-node events
+
+- [ ] introduce cluster event interface,
+- [ ] Redis implementation,
+- [ ] WebSocket/IMAP IDLE fan-out,
+- [ ] reconnection/degraded-mode behavior.
+
+---
+
+# 13. Acceptance criteria
+
+The work is complete when all of the following are true:
+
+- [ ] three or more RustBox nodes can run with no writable shared POSIX filesystem,
+- [ ] nodes can be destroyed/recreated without losing attachments or search state,
+- [ ] S3-compatible storage works for draft attachments,
+- [ ] local filesystem remains supported for single-node mode,
+- [ ] `cluster_mode = true` accepts S3-backed storage without `cluster_shared_data_dir = true`,
+- [ ] local Tantivy is rejected in cluster mode,
+- [ ] external index failure does not corrupt SQL/cache state,
+- [ ] full index rebuild is possible,
+- [ ] index events are durable and retryable,
+- [ ] SQL remains the authoritative cache/data source,
+- [ ] no PGP private key depends on node-local filesystem in supported cluster mode,
+- [ ] configuration and migration paths are documented for every supported deployment variant.
+
+---
+
+# Recommended target
+
+For the long-term cluster architecture, prefer:
+
+```text
+RustBox nodes (stateless)
+       │
+       ├── PostgreSQL/MySQL      authoritative metadata + mailbox cache
+       ├── Redis (optional)      sessions + future cross-node events
+       ├── S3-compatible storage draft attachments / blob data
+       └── external indexer      optional deep full-text search
+```
+
+Recommended supported profiles:
+
+1. **Single node:** filesystem + local Tantivy.
+2. **Small cluster:** shared filesystem or S3 + search disabled.
+3. **Preferred stateless cluster:** S3-compatible storage + external indexer.
+4. **Enterprise/existing infrastructure:** S3/shared storage + OpenSearch or optional Solr adapter.
+
+The key design rule is to keep **SQL cache**, **blob storage**, **search index**, and **private-key storage** as separate concerns. None of them should force all application nodes to share the same writable local directory.