# TODO — cluster-safe storage and external search backends

## Goal

Remove the hard dependency on a shared local `data_dir` for multi-node RustBox deployments and make filesystem-backed data explicitly pluggable.

The target architecture should allow fully stateless RustBox application nodes using:

- a shared MySQL/PostgreSQL database for mailbox metadata and cached message bodies,
- optional Redis for shared sessions/events,
- shared blob/object storage for draft attachments and other binary data,
- an optional external full-text indexer,
- no writable node-local state required for normal cluster operation.

This is a follow-up to the 0.1.2 database/cluster audit. RustBox 0.2.0 implements the storage abstraction, S3/shared-POSIX support, OpenSearch backend and durable indexing pipeline. Live interoperability tests against SILO/MinIO/AWS S3/Ceph RGW and a real multi-node cluster remain release-validation work.

---

## Current state in 0.1.2

Shared through SQL:

- mailbox/folder metadata,
- cached message bodies,
- application data stored in MySQL/PostgreSQL,
- persistent sessions when SQL is selected.

Node-local data may still exist under `data_dir/<user_hash>/` for single-node features such as local Tantivy and OpenPGP private-key material. Draft attachments use the configured `[storage]` backend.

Cluster configuration is explicit and capability-based:

```toml
cluster_mode = true
pgp_enabled = false

[storage]
backend = "s3"

[search]
backend = "opensearch"
```

Shared POSIX storage is also supported with `storage.backend="filesystem"` and `storage.shared=true`. Local Tantivy remains single-node only.

---

# 1. Shared data / blob storage abstraction

## Required design

Introduce a storage interface independent from `data_dir`.

Conceptual API:

```text
BlobStorage
  put(key, stream, metadata)
  get(key) -> stream
  delete(key)
  exists(key)
  stat(key)
```

Application code must store a logical object key in SQL, not an absolute filesystem path.

Suggested key layout:

```text
users/<user_hash>/drafts/<draft_id>/attachments/<attachment_id>
```

The backend is responsible for translating that logical key to a filesystem path or object-store key.

## Variant A — local filesystem

Target: single-node installations.

```toml
[storage]
backend = "filesystem"
path = "/var/lib/rustbox/data"
```

Properties:

- simplest deployment,
- no external dependency,
- valid for one RustBox node,
- not considered cluster-safe unless the path is actually on shared storage.

## Variant B — shared POSIX filesystem

Target: existing clusters with NFS/CephFS/GlusterFS or another shared mount.

```toml
[storage]
backend = "filesystem"
path = "/srv/rustbox-shared"
shared = true
```

Supported use:

- draft attachments,
- other blob-like application files.

Do **not** treat a shared filesystem as a supported way to run multiple concurrent Tantivy writers. Tantivy indexing must remain a separate concern.

Operational notes:

- identical mount path is not required once SQL stores logical keys rather than absolute paths,
- application startup should check read/write access,
- health endpoint should expose storage availability,
- temporary uploads must be written atomically and cleaned after failure.

## Variant C — S3-compatible object storage

Target: recommended multi-node/stateless deployment.

Supported implementations should include any sufficiently compatible S3 API, for example:

- AWS S3,
- MinIO,
- Ceph RGW,
- other S3-compatible services.

Suggested configuration:

```toml
[storage]
backend = "s3"
bucket = "rustbox"
region = "eu-central-1"
endpoint = "https://s3.example.net" # optional for AWS, required for many compatible services
prefix = "rustbox/"
path_style = false

# Credentials should preferably come from environment/instance identity,
# not plaintext TOML.
```

Required behavior:

- streaming upload/download,
- multipart upload for large objects where useful,
- checksum/ETag validation where reliable,
- abort incomplete multipart uploads,
- retry transient errors,
- configurable request/connect timeouts,
- TLS verification enabled by default,
- optional server-side encryption configuration,
- lifecycle support for abandoned temporary objects,
- no public bucket requirement.

The database should keep object metadata and logical object keys. RustBox nodes must not depend on any local copy being present.

## Variant D — additional cloud/object backends

Future adapters may add:

```toml
backend = "azure_blob"
```

or:

```toml
backend = "gcs"
```

Do not hard-code cloud-specific behavior into attachment code. Keep it behind the storage interface.

Priority should be:

1. `filesystem`,
2. `s3`,
3. additional adapters only when there is demand.

---

# 2. Cluster behavior after storage abstraction

When `storage.backend = "s3"`, RustBox allows cluster mode without a writable shared `data_dir`. Cluster validation is capability-based and depends on the selected storage/search backends.

Expected rules:

| Configuration | Cluster allowed | Notes |
|---|---:|---|
| local filesystem | no | unless explicitly marked/verified as shared |
| shared POSIX filesystem | yes | attachments/blobs only |
| S3-compatible storage | yes | preferred stateless mode |
| external index + S3 | yes | full cluster mode |
| local Tantivy | no | single-node only |
| index disabled | yes | SQL search only |

Suggested validation logic:

```text
cluster_mode = false
    -> filesystem + Tantivy allowed

cluster_mode = true
    -> blob storage must be cluster-safe
    -> local Tantivy forbidden
    -> PGP requires a cluster-safe key backend or must be disabled
```

---

# 3. Search/index backend abstraction

Search must not be coupled to the blob storage backend.

Introduce a separate interface such as:

```text
SearchBackend
  index_message(document)
  delete_message(message_id)
  search(query, filters, paging)
  health()
  rebuild(...)
```

Suggested configuration:

```toml
[search]
backend = "disabled"
```

Possible values:

```text
disabled
local_tantivy
quickwit
opensearch
solr
```

Additional adapters can be added later without changing mail/cache storage.

---

## Search variant 0 — disabled

```toml
[search]
backend = "disabled"
```

Use cases:

- cluster without external search service,
- minimal deployments,
- recovery/degraded mode.

Behavior:

- SQL-backed metadata/header search remains available,
- deep body full-text search is unavailable,
- no index files on application nodes.

This must remain a fully supported configuration.

---

## Search variant 1 — local Tantivy

```toml
[search]
backend = "local_tantivy"
path = "/var/lib/rustbox/data"
```

Use case:

- single-node RustBox.

Rules:

- allowed only when `cluster_mode = false`,
- current Tantivy behavior can be retained behind the new `SearchBackend` interface,
- index remains rebuildable from authoritative data,
- the index must never be treated as the only copy of mailbox data.

Do not support multi-node writers against one shared Tantivy directory.

---

## Search variant 2 — Quickwit

```toml
[search]
backend = "quickwit"
url = "http://quickwit:7280"
index = "rustbox-mails"
```

Why evaluate it:

- written in Rust,
- based on Tantivy,
- designed around distributed search and object storage,
- conceptually close to the current local Tantivy implementation.

Required evaluation before implementation:

- message update/delete semantics,
- latency for newly indexed mail,
- tenant/user isolation strategy,
- reindex cost,
- index schema evolution,
- attachment/body size limits,
- failure/retry behavior,
- suitability for mutable mailbox state rather than only append-heavy workloads.

Do not select Quickwit solely because it uses Tantivy; validate mail-specific update/delete behavior first.

---

## Search variant 3 — OpenSearch

```toml
[search]
backend = "opensearch"
urls = ["https://search-1.example.net:9200", "https://search-2.example.net:9200"]
index_prefix = "rustbox"
```

Why support it:

- mature distributed indexing/search,
- common in existing infrastructure,
- strong filtering and full-text capabilities,
- suitable for larger installations.

Trade-offs:

- heavier operational footprint,
- JVM memory requirements,
- index lifecycle/sharding needs administration.

OpenSearch is a good enterprise/existing-infrastructure option, but should not become mandatory for RustBox.

---

## Search variant 4 — Solr / SolrCloud

```toml
[search]
backend = "solr"
url = "https://solr.example.net/solr/rustbox"
```

Why keep it as an option:

- mature Lucene full-text engine,
- still used in established environments,
- supports distributed SolrCloud deployments.

Trade-offs:

- JVM-based,
- cluster operation is not simpler than lightweight alternatives,
- adds a separate operational stack.

Treat Solr as an optional adapter for environments that already operate it, not as the default RustBox cluster indexer.

---

# 4. Recommended deployment variants

## Variant 1 — simple single node

```text
RustBox
 ├─ SQLite/MySQL/PostgreSQL
 ├─ local filesystem attachments
 └─ local Tantivy
```

Example:

```toml
cluster_mode = false

[storage]
backend = "filesystem"
path = "/var/lib/rustbox/data"

[search]
backend = "local_tantivy"
```

Best for small installations.

---

## Variant 2 — cluster with shared filesystem

```text
RustBox node 1 ─┐
RustBox node 2 ─┼── MySQL/PostgreSQL
RustBox node 3 ─┘
       │
       └──────── shared NFS/CephFS for attachments

Search: disabled or external
```

Example:

```toml
cluster_mode = true

[storage]
backend = "filesystem"
path = "/srv/rustbox-shared"
shared = true

[search]
backend = "disabled"
```

This is the lowest-change migration path from 0.1.2.

---

## Variant 3 — stateless cluster with S3, no external indexer

```text
RustBox node 1 ─┐
RustBox node 2 ─┼── MySQL/PostgreSQL
RustBox node 3 ─┘
       │
       └──────── S3 / MinIO / Ceph RGW

Search: SQL-only
```

Example:

```toml
cluster_mode = true

[storage]
backend = "s3"
bucket = "rustbox"
endpoint = "https://minio.example.net"

[search]
backend = "disabled"
```

This should be a first-class supported production configuration.

---

## Variant 4 — stateless cluster with S3 + external indexer

```text
                    ┌── MySQL/PostgreSQL
RustBox nodes ──────┼── S3-compatible blob storage
                    └── external search cluster
```

Example with Quickwit:

```toml
cluster_mode = true

[storage]
backend = "s3"
bucket = "rustbox"
endpoint = "https://s3.example.net"

[search]
backend = "quickwit"
url = "http://quickwit:7280"
index = "rustbox-mails"
```

Example with OpenSearch:

```toml
[search]
backend = "opensearch"
urls = ["https://search.example.net:9200"]
index_prefix = "rustbox"
```

This is the preferred architecture for larger multi-node installations requiring deep full-text search.

---

# 5. Durable indexing pipeline

Do not make correctness depend on an HTTP request successfully updating the external index synchronously.

Add a durable indexing queue/outbox in the shared SQL database.

Conceptual flow:

```text
mail/cache transaction
      │
      ├── update authoritative SQL data
      └── append search_outbox event
                    │
                    v
             index worker(s)
                    │
                    v
       Quickwit/OpenSearch/Solr
```

Required properties:

- idempotent events,
- retry with backoff,
- dead-letter/error visibility,
- safe concurrent workers,
- delete events,
- full reindex command,
- index version/schema version,
- ability to rebuild the index from authoritative SQL/IMAP data,
- index lag metric.

The external index must be treated as derived/rebuildable state, never as the source of truth.

---

# 6. PGP/key storage in cluster mode

PGP private keys should not automatically follow the generic attachment/blob backend.

Current safe rule remains:

```toml
pgp_enabled = false
```

for cluster mode until a dedicated key-storage design exists.

Future variants to evaluate:

1. encrypted keys in the shared SQL database,
2. encrypted objects in S3 with a separate master-key/KMS strategy,
3. Vault/HSM/KMS-backed key handling,
4. dedicated secrets backend.

Requirements:

- encryption at rest independent of ordinary object-storage ACLs,
- no plaintext private key in shared object storage,
- identical behavior from every RustBox node,
- rotation/recovery procedure,
- auditability.

Do not treat shared blob storage as a solution for private-key consistency.

---

# 7. WebSocket / IMAP IDLE cluster events

Separate from storage and search, the current in-process event bus is node-local.

TODO:

- add optional cluster event backend,
- initial candidate: Redis Pub/Sub or Redis Streams,
- propagate mailbox/notification invalidations across nodes,
- retain local fast-path for single-node mode,
- define behavior during broker outages.

This is not required for SQL cache consistency, but is required for reliable real-time UI updates across nodes.

---

# 8. Configuration model

RustBox 0.2.0 uses one configuration model only. There are no compatibility aliases for pre-0.2 storage/search keys.

```toml
[storage]
backend = "filesystem"
path = "/var/lib/rustbox/data"
shared = false

[search]
backend = "local_tantivy"
```

Cluster deployments must select cluster-safe backends explicitly (`storage.shared=true` for shared POSIX or `storage.backend="s3"`; `search.backend="opensearch"` or `disabled`).

---

# 9. Migration of existing attachments

Provide an explicit migration command rather than silently moving data during normal startup.

Example target CLI:

```text
rustbox storage migrate \
  --from filesystem:/var/lib/rustbox/data \
  --to s3:rustbox \
  --verify
```

Requirements:

- resumable,
- idempotent,
- checksum/size verification,
- dry-run mode,
- progress reporting,
- do not delete source objects until verification passes,
- update SQL references transactionally or through a safe staged migration,
- rollback/retry documentation.

---

# 10. Health, metrics and operations

Expose at least:

```text
storage backend status
storage request failures
storage latency
index backend status
index queue depth
index lag
index failures/retries
last successful index operation
```

Startup behavior:

- invalid cluster/storage combination -> hard configuration error,
- external index unavailable -> application may start in degraded mode if configured,
- shared SQL unavailable -> application startup/request handling must fail clearly,
- object storage unavailable -> mail reading from SQL may continue, but attachment operations must fail explicitly.

Suggested option:

```toml
[search]
required = false
```

When `required = false`, RustBox can run with SQL search while the external index is unavailable.

---

# 11. Security requirements

For object storage:

- private buckets/containers only,
- least-privilege credentials,
- environment/instance/workload identity preferred over TOML secrets,
- TLS by default,
- optional custom CA bundle for private S3-compatible services,
- prevent object-key traversal/injection,
- server-side encryption option,
- never expose backend credentials to the browser.

For external search:

- authenticated TLS connection,
- per-user/tenant filtering must be enforced server-side by RustBox,
- never trust client-provided tenant/user IDs,
- index documents must not allow cross-user search leakage,
- document deletion must propagate to the index.

---

# 12. Implementation order

## Phase 1 — storage abstraction

- [x] introduce `BlobStorage` interface,
- [x] convert draft attachments from absolute paths to logical object keys,
- [x] implement filesystem backend,
- [x] retain existing behavior for single-node installations,
- [x] add storage health checks.

## Phase 2 — S3 backend

- [x] implement S3-compatible backend,
- [x] streaming upload/download,
- [x] retries/timeouts/checksums,
- [ ] SILO Compose integration validation,
- [ ] MinIO integration validation,
- [ ] AWS S3 integration validation,
- [ ] Ceph RGW compatibility validation,
- [x] migration command from filesystem to S3,
- [x] validate cluster safety directly from `storage.backend` and `storage.shared`.

## Phase 3 — search abstraction

- [x] introduce `SearchBackend` interface,
- [x] move current Tantivy code behind `local_tantivy`,
- [x] preserve `disabled` backend,
- [x] reject `local_tantivy` in cluster mode,
- [x] define canonical indexed message document/schema.

## Phase 4 — durable index events

- [x] add SQL search outbox/queue,
- [x] add index worker locking/concurrency,
- [x] retries/backoff,
- [x] delete events,
- [x] reindex command,
- [x] index lag/error metrics.

## Phase 5 — external index adapters

Recommended evaluation order:

RustBox 0.2.0 selects **OpenSearch** as the supported external adapter because its update/delete/reindex semantics map directly to mutable mailbox documents. Quickwit and Solr remain optional future adapters.

- [ ] Quickwit proof of concept,
- [x] OpenSearch adapter,
- [x] compare update/delete/reindex behavior and operational footprint,
- [x] choose documented recommended backend,
- [ ] Solr adapter only if there is real deployment demand.

## Phase 6 — cluster PGP

- [ ] design dedicated encrypted key backend,
- [x] keep PGP disabled in cluster mode until completed,
- [ ] evaluate Vault/KMS/SQL-encrypted/S3-encrypted variants,
- [ ] add migration procedure for existing local keys.

## Phase 7 — cross-node events

- [ ] introduce cluster event interface,
- [ ] Redis implementation,
- [ ] WebSocket/IMAP IDLE fan-out,
- [ ] reconnection/degraded-mode behavior.

---

# 13. Acceptance criteria

The work is complete when all of the following are true:

- [ ] three or more RustBox nodes can run with no writable shared POSIX filesystem,
- [ ] nodes can be destroyed/recreated without losing attachments or search state,
- [ ] S3-compatible storage works for draft attachments (live SILO/MinIO/AWS validation pending),
- [x] local filesystem remains supported for single-node mode,
- [x] `cluster_mode = true` accepts S3-backed storage without any legacy shared-data-dir switch,
- [x] local Tantivy is rejected in cluster mode,
- [x] external index failure does not corrupt SQL/cache state,
- [x] full index rebuild is possible,
- [x] index events are durable and retryable,
- [x] SQL remains the authoritative cache/data source,
- [x] no PGP private key depends on node-local filesystem in supported cluster mode,
- [x] configuration and migration paths are documented for every supported deployment variant.

---

# Recommended target

For the long-term cluster architecture, prefer:

```text
RustBox nodes (stateless)
       │
       ├── PostgreSQL/MySQL      authoritative metadata + mailbox cache
       ├── Redis (optional)      sessions + future cross-node events
       ├── S3-compatible storage draft attachments / blob data
       └── external indexer      optional deep full-text search
```

Recommended supported profiles:

1. **Single node:** filesystem + local Tantivy.
2. **Small cluster:** shared filesystem or S3 + search disabled.
3. **Preferred stateless cluster:** S3-compatible storage + external indexer.
4. **Enterprise/existing infrastructure:** S3/shared storage + OpenSearch or optional Solr adapter.

The key design rule is to keep **SQL cache**, **blob storage**, **search index**, and **private-key storage** as separate concerns. None of them should force all application nodes to share the same writable local directory.