RustBox/TODO.md

TODO — cluster-safe storage and external search backends

Goal

Remove the hard dependency on a shared local data_dir for multi-node RustBox deployments and make filesystem-backed data explicitly pluggable.

The target architecture should allow fully stateless RustBox application nodes using:

• a shared MySQL/PostgreSQL database for mailbox metadata and cached message bodies,
• optional Redis for shared sessions/events,
• shared blob/object storage for draft attachments and other binary data,
• an optional external full-text indexer,
• no writable node-local state required for normal cluster operation.

This is a follow-up to the 0.1.2 database/cluster audit. RustBox 0.2.0 implements the storage abstraction, S3/shared-POSIX support, OpenSearch backend and durable indexing pipeline. Live interoperability tests against SILO/MinIO/AWS S3/Ceph RGW and a real multi-node cluster remain release-validation work.

---

Current state in 0.1.2

Shared through SQL:

• mailbox/folder metadata,
• cached message bodies,
• application data stored in MySQL/PostgreSQL,
• persistent sessions when SQL is selected.

Node-local data may still exist under data_dir/<user_hash>/ for single-node features such as local Tantivy and OpenPGP private-key material. Draft attachments use the configured [storage] backend.

Cluster configuration is explicit and capability-based:

cluster_mode = true
pgp_enabled = false

[storage]
backend = "s3"

[search]
backend = "opensearch"

Shared POSIX storage is also supported with storage.backend="filesystem" and storage.shared=true. Local Tantivy remains single-node only.

---

1. Shared data / blob storage abstraction

Required design

Introduce a storage interface independent from data_dir.

Conceptual API:

BlobStorage
  put(key, stream, metadata)
  get(key) -> stream
  delete(key)
  exists(key)
  stat(key)

Application code must store a logical object key in SQL, not an absolute filesystem path.

Suggested key layout:

users/<user_hash>/drafts/<draft_id>/attachments/<attachment_id>

The backend is responsible for translating that logical key to a filesystem path or object-store key.

Variant A — local filesystem

Target: single-node installations.

[storage]
backend = "filesystem"
path = "/var/lib/rustbox/data"

Properties:

• simplest deployment,
• no external dependency,
• valid for one RustBox node,
• not considered cluster-safe unless the path is actually on shared storage.

Variant B — shared POSIX filesystem

Target: existing clusters with NFS/CephFS/GlusterFS or another shared mount.

[storage]
backend = "filesystem"
path = "/srv/rustbox-shared"
shared = true

Supported use:

• draft attachments,
• other blob-like application files.

Do not treat a shared filesystem as a supported way to run multiple concurrent Tantivy writers. Tantivy indexing must remain a separate concern.

Operational notes:

• identical mount path is not required once SQL stores logical keys rather than absolute paths,
• application startup should check read/write access,
• health endpoint should expose storage availability,
• temporary uploads must be written atomically and cleaned after failure.

Variant C — S3-compatible object storage

Target: recommended multi-node/stateless deployment.

Supported implementations should include any sufficiently compatible S3 API, for example:

• AWS S3,
• MinIO,
• Ceph RGW,
• other S3-compatible services.

Suggested configuration:

[storage]
backend = "s3"
bucket = "rustbox"
region = "eu-central-1"
endpoint = "https://s3.example.net" # optional for AWS, required for many compatible services
prefix = "rustbox/"
path_style = false

# Credentials should preferably come from environment/instance identity,
# not plaintext TOML.

Required behavior:

• streaming upload/download,
• multipart upload for large objects where useful,
• checksum/ETag validation where reliable,
• abort incomplete multipart uploads,
• retry transient errors,
• configurable request/connect timeouts,
• TLS verification enabled by default,
• optional server-side encryption configuration,
• lifecycle support for abandoned temporary objects,
• no public bucket requirement.

The database should keep object metadata and logical object keys. RustBox nodes must not depend on any local copy being present.

Variant D — additional cloud/object backends

Future adapters may add:

backend = "azure_blob"

or:

backend = "gcs"

Do not hard-code cloud-specific behavior into attachment code. Keep it behind the storage interface.

Priority should be:

1. filesystem,
2. s3,
3. additional adapters only when there is demand.

---

2. Cluster behavior after storage abstraction

When storage.backend = "s3", RustBox allows cluster mode without a writable shared data_dir. Cluster validation is capability-based and depends on the selected storage/search backends.

Expected rules:

Configuration	Cluster allowed	Notes
local filesystem	no	unless explicitly marked/verified as shared
shared POSIX filesystem	yes	attachments/blobs only
S3-compatible storage	yes	preferred stateless mode
external index + S3	yes	full cluster mode
local Tantivy	no	single-node only
index disabled	yes	SQL search only

Suggested validation logic:

cluster_mode = false
    -> filesystem + Tantivy allowed

cluster_mode = true
    -> blob storage must be cluster-safe
    -> local Tantivy forbidden
    -> PGP requires a cluster-safe key backend or must be disabled

---

3. Search/index backend abstraction

Search must not be coupled to the blob storage backend.

Introduce a separate interface such as:

SearchBackend
  index_message(document)
  delete_message(message_id)
  search(query, filters, paging)
  health()
  rebuild(...)

Suggested configuration:

[search]
backend = "disabled"

Possible values:

disabled
local_tantivy
quickwit
opensearch
solr

Additional adapters can be added later without changing mail/cache storage.

---

Search variant 0 — disabled

[search]
backend = "disabled"

Use cases:

• cluster without external search service,
• minimal deployments,
• recovery/degraded mode.

Behavior:

• SQL-backed metadata/header search remains available,
• deep body full-text search is unavailable,
• no index files on application nodes.

This must remain a fully supported configuration.

---

Search variant 1 — local Tantivy

[search]
backend = "local_tantivy"
path = "/var/lib/rustbox/data"

Use case:

• single-node RustBox.

Rules:

• allowed only when cluster_mode = false,
• current Tantivy behavior can be retained behind the new SearchBackend interface,
• index remains rebuildable from authoritative data,
• the index must never be treated as the only copy of mailbox data.

Do not support multi-node writers against one shared Tantivy directory.

---

Search variant 2 — Quickwit

[search]
backend = "quickwit"
url = "http://quickwit:7280"
index = "rustbox-mails"

Why evaluate it:

• written in Rust,
• based on Tantivy,
• designed around distributed search and object storage,
• conceptually close to the current local Tantivy implementation.

Required evaluation before implementation:

• message update/delete semantics,
• latency for newly indexed mail,
• tenant/user isolation strategy,
• reindex cost,
• index schema evolution,
• attachment/body size limits,
• failure/retry behavior,
• suitability for mutable mailbox state rather than only append-heavy workloads.

Do not select Quickwit solely because it uses Tantivy; validate mail-specific update/delete behavior first.

---

Search variant 3 — OpenSearch

[search]
backend = "opensearch"
urls = ["https://search-1.example.net:9200", "https://search-2.example.net:9200"]
index_prefix = "rustbox"

Why support it:

• mature distributed indexing/search,
• common in existing infrastructure,
• strong filtering and full-text capabilities,
• suitable for larger installations.

Trade-offs:

• heavier operational footprint,
• JVM memory requirements,
• index lifecycle/sharding needs administration.

OpenSearch is a good enterprise/existing-infrastructure option, but should not become mandatory for RustBox.

---

Search variant 4 — Solr / SolrCloud

[search]
backend = "solr"
url = "https://solr.example.net/solr/rustbox"

Why keep it as an option:

• mature Lucene full-text engine,
• still used in established environments,
• supports distributed SolrCloud deployments.

Trade-offs:

• JVM-based,
• cluster operation is not simpler than lightweight alternatives,
• adds a separate operational stack.

Treat Solr as an optional adapter for environments that already operate it, not as the default RustBox cluster indexer.

---

4. Recommended deployment variants

Variant 1 — simple single node

RustBox
 ├─ SQLite/MySQL/PostgreSQL
 ├─ local filesystem attachments
 └─ local Tantivy

Example:

cluster_mode = false

[storage]
backend = "filesystem"
path = "/var/lib/rustbox/data"

[search]
backend = "local_tantivy"

Best for small installations.

---

Variant 2 — cluster with shared filesystem

RustBox node 1 ─┐
RustBox node 2 ─┼── MySQL/PostgreSQL
RustBox node 3 ─┘
       │
       └──────── shared NFS/CephFS for attachments

Search: disabled or external

Example:

cluster_mode = true

[storage]
backend = "filesystem"
path = "/srv/rustbox-shared"
shared = true

[search]
backend = "disabled"

This is the lowest-change migration path from 0.1.2.

---

Variant 3 — stateless cluster with S3, no external indexer

RustBox node 1 ─┐
RustBox node 2 ─┼── MySQL/PostgreSQL
RustBox node 3 ─┘
       │
       └──────── S3 / MinIO / Ceph RGW

Search: SQL-only

Example:

cluster_mode = true

[storage]
backend = "s3"
bucket = "rustbox"
endpoint = "https://minio.example.net"

[search]
backend = "disabled"

This should be a first-class supported production configuration.

---

Variant 4 — stateless cluster with S3 + external indexer

                    ┌── MySQL/PostgreSQL
RustBox nodes ──────┼── S3-compatible blob storage
                    └── external search cluster

Example with Quickwit:

cluster_mode = true

[storage]
backend = "s3"
bucket = "rustbox"
endpoint = "https://s3.example.net"

[search]
backend = "quickwit"
url = "http://quickwit:7280"
index = "rustbox-mails"

Example with OpenSearch:

[search]
backend = "opensearch"
urls = ["https://search.example.net:9200"]
index_prefix = "rustbox"

This is the preferred architecture for larger multi-node installations requiring deep full-text search.

---

5. Durable indexing pipeline

Do not make correctness depend on an HTTP request successfully updating the external index synchronously.

Add a durable indexing queue/outbox in the shared SQL database.

Conceptual flow:

mail/cache transaction
      │
      ├── update authoritative SQL data
      └── append search_outbox event
                    │
                    v
             index worker(s)
                    │
                    v
       Quickwit/OpenSearch/Solr

Required properties:

• idempotent events,
• retry with backoff,
• dead-letter/error visibility,
• safe concurrent workers,
• delete events,
• full reindex command,
• index version/schema version,
• ability to rebuild the index from authoritative SQL/IMAP data,
• index lag metric.

The external index must be treated as derived/rebuildable state, never as the source of truth.

---

6. PGP/key storage in cluster mode

PGP private keys should not automatically follow the generic attachment/blob backend.

Current safe rule remains:

pgp_enabled = false

for cluster mode until a dedicated key-storage design exists.

Future variants to evaluate:

1. encrypted keys in the shared SQL database,
2. encrypted objects in S3 with a separate master-key/KMS strategy,
3. Vault/HSM/KMS-backed key handling,
4. dedicated secrets backend.

Requirements:

• encryption at rest independent of ordinary object-storage ACLs,
• no plaintext private key in shared object storage,
• identical behavior from every RustBox node,
• rotation/recovery procedure,
• auditability.

Do not treat shared blob storage as a solution for private-key consistency.

---

7. WebSocket / IMAP IDLE cluster events

Separate from storage and search, the current in-process event bus is node-local.

TODO:

• add optional cluster event backend,
• initial candidate: Redis Pub/Sub or Redis Streams,
• propagate mailbox/notification invalidations across nodes,
• retain local fast-path for single-node mode,
• define behavior during broker outages.

This is not required for SQL cache consistency, but is required for reliable real-time UI updates across nodes.

---

8. Configuration model

RustBox 0.2.0 uses one configuration model only. There are no compatibility aliases for pre-0.2 storage/search keys.

[storage]
backend = "filesystem"
path = "/var/lib/rustbox/data"
shared = false

[search]
backend = "local_tantivy"

Cluster deployments must select cluster-safe backends explicitly (storage.shared=true for shared POSIX or storage.backend="s3"; search.backend="opensearch" or disabled).

---

9. Migration of existing attachments

Provide an explicit migration command rather than silently moving data during normal startup.

Example target CLI:

rustbox storage migrate \
  --from filesystem:/var/lib/rustbox/data \
  --to s3:rustbox \
  --verify

Requirements:

• resumable,
• idempotent,
• checksum/size verification,
• dry-run mode,
• progress reporting,
• do not delete source objects until verification passes,
• update SQL references transactionally or through a safe staged migration,
• rollback/retry documentation.

---

10. Health, metrics and operations

Expose at least:

storage backend status
storage request failures
storage latency
index backend status
index queue depth
index lag
index failures/retries
last successful index operation

Startup behavior:

• invalid cluster/storage combination -> hard configuration error,
• external index unavailable -> application may start in degraded mode if configured,
• shared SQL unavailable -> application startup/request handling must fail clearly,
• object storage unavailable -> mail reading from SQL may continue, but attachment operations must fail explicitly.

Suggested option:

[search]
required = false

When required = false, RustBox can run with SQL search while the external index is unavailable.

---

11. Security requirements

For object storage:

• private buckets/containers only,
• least-privilege credentials,
• environment/instance/workload identity preferred over TOML secrets,
• TLS by default,
• optional custom CA bundle for private S3-compatible services,
• prevent object-key traversal/injection,
• server-side encryption option,
• never expose backend credentials to the browser.

For external search:

• authenticated TLS connection,
• per-user/tenant filtering must be enforced server-side by RustBox,
• never trust client-provided tenant/user IDs,
• index documents must not allow cross-user search leakage,
• document deletion must propagate to the index.

---

12. Implementation order

Phase 1 — storage abstraction

• introduce BlobStorage interface,
• convert draft attachments from absolute paths to logical object keys,
• implement filesystem backend,
• retain existing behavior for single-node installations,
• add storage health checks.

Phase 2 — S3 backend

• implement S3-compatible backend,
• streaming upload/download,
• retries/timeouts/checksums,
• SILO Compose integration validation,
• MinIO integration validation,
• AWS S3 integration validation,
• Ceph RGW compatibility validation,
• migration command from filesystem to S3,
• validate cluster safety directly from storage.backend and storage.shared.

Phase 3 — search abstraction

• introduce SearchBackend interface,
• move current Tantivy code behind local_tantivy,
• preserve disabled backend,
• reject local_tantivy in cluster mode,
• define canonical indexed message document/schema.

Phase 4 — durable index events

• add SQL search outbox/queue,
• add index worker locking/concurrency,
• retries/backoff,
• delete events,
• reindex command,
• index lag/error metrics.

Phase 5 — external index adapters

Recommended evaluation order:

RustBox 0.2.0 selects OpenSearch as the supported external adapter because its update/delete/reindex semantics map directly to mutable mailbox documents. Quickwit and Solr remain optional future adapters.

• Quickwit proof of concept,
• OpenSearch adapter,
• compare update/delete/reindex behavior and operational footprint,
• choose documented recommended backend,
• Solr adapter only if there is real deployment demand.

Phase 6 — cluster PGP

• design dedicated encrypted key backend,
• keep PGP disabled in cluster mode until completed,
• evaluate Vault/KMS/SQL-encrypted/S3-encrypted variants,
• add migration procedure for existing local keys.

Phase 7 — cross-node events

• introduce cluster event interface,
• Redis implementation,
• WebSocket/IMAP IDLE fan-out,
• reconnection/degraded-mode behavior.

---

13. Acceptance criteria

The work is complete when all of the following are true:

• three or more RustBox nodes can run with no writable shared POSIX filesystem,
• nodes can be destroyed/recreated without losing attachments or search state,
• S3-compatible storage works for draft attachments (live SILO/MinIO/AWS validation pending),
• local filesystem remains supported for single-node mode,
• cluster_mode = true accepts S3-backed storage without any legacy shared-data-dir switch,
• local Tantivy is rejected in cluster mode,
• external index failure does not corrupt SQL/cache state,
• full index rebuild is possible,
• index events are durable and retryable,
• SQL remains the authoritative cache/data source,
• no PGP private key depends on node-local filesystem in supported cluster mode,
• configuration and migration paths are documented for every supported deployment variant.

---

Recommended target

For the long-term cluster architecture, prefer:

RustBox nodes (stateless)
       │
       ├── PostgreSQL/MySQL      authoritative metadata + mailbox cache
       ├── Redis (optional)      sessions + future cross-node events
       ├── S3-compatible storage draft attachments / blob data
       └── external indexer      optional deep full-text search

Recommended supported profiles:

1. Single node: filesystem + local Tantivy.
2. Small cluster: shared filesystem or S3 + search disabled.
3. Preferred stateless cluster: S3-compatible storage + external indexer.
4. Enterprise/existing infrastructure: S3/shared storage + OpenSearch or optional Solr adapter.

The key design rule is to keep SQL cache, blob storage, search index, and private-key storage as separate concerns. None of them should force all application nodes to share the same writable local directory.