Mateusz Gruszczyński committed
1 parent
e3e844ef4f
commit
fc81ecafd6
1 file changed
+806
@@ -0,0 +1,806 @@
|
||||
# TODO — cluster-safe storage and external search backends
|
||||
|
||||
## Goal
|
||||
|
||||
Remove the hard dependency on a shared local `data_dir` for multi-node RustBox deployments and make filesystem-backed data explicitly pluggable.
|
||||
|
||||
The target architecture should allow fully stateless RustBox application nodes using:
|
||||
|
||||
- a shared MySQL/PostgreSQL database for mailbox metadata and cached message bodies,
|
||||
- optional Redis for shared sessions/events,
|
||||
- shared blob/object storage for draft attachments and other binary data,
|
||||
- an optional external full-text indexer,
|
||||
- no writable node-local state required for normal cluster operation.
|
||||
|
||||
This is a follow-up to the 0.1.2 database/cluster audit. It is not implemented in 0.1.2.
|
||||
|
||||
---
|
||||
|
||||
## Current state in 0.1.2
|
||||
|
||||
Shared through SQL:
|
||||
|
||||
- mailbox/folder metadata,
|
||||
- cached message bodies,
|
||||
- application data stored in MySQL/PostgreSQL,
|
||||
- persistent sessions when SQL is selected.
|
||||
|
||||
Filesystem-backed under `data_dir/<user_hash>/`:
|
||||
|
||||
- `attachments/<draft_id>/<attachment_id>` — draft attachments,
|
||||
- `tantivy/` — local full-text index,
|
||||
- `pgp/private/` — OpenPGP private-key material,
|
||||
- legacy `db.sqlite` — migration source only.
|
||||
|
||||
Current cluster restrictions:
|
||||
|
||||
```toml
|
||||
cluster_mode = true
|
||||
cluster_shared_data_dir = true
|
||||
search_index_backend = "disabled"
|
||||
pgp_enabled = false
|
||||
```
|
||||
|
||||
`cluster_shared_data_dir = true` is currently required because draft attachments are path-based and local Tantivy/PGP state is not cluster-safe.
|
||||
|
||||
---
|
||||
|
||||
# 1. Shared data / blob storage abstraction
|
||||
|
||||
## Required design
|
||||
|
||||
Introduce a storage interface independent from `data_dir`.
|
||||
|
||||
Conceptual API:
|
||||
|
||||
```text
|
||||
BlobStorage
|
||||
put(key, stream, metadata)
|
||||
get(key) -> stream
|
||||
delete(key)
|
||||
exists(key)
|
||||
stat(key)
|
||||
```
|
||||
|
||||
Application code must store a logical object key in SQL, not an absolute filesystem path.
|
||||
|
||||
Suggested key layout:
|
||||
|
||||
```text
|
||||
users/<user_hash>/drafts/<draft_id>/attachments/<attachment_id>
|
||||
```
|
||||
|
||||
The backend is responsible for translating that logical key to a filesystem path or object-store key.
|
||||
|
||||
## Variant A — local filesystem
|
||||
|
||||
Target: single-node installations.
|
||||
|
||||
```toml
|
||||
[storage]
|
||||
backend = "filesystem"
|
||||
path = "/var/lib/rustbox/data"
|
||||
```
|
||||
|
||||
Properties:
|
||||
|
||||
- simplest deployment,
|
||||
- no external dependency,
|
||||
- valid for one RustBox node,
|
||||
- not considered cluster-safe unless the path is actually on shared storage.
|
||||
|
||||
## Variant B — shared POSIX filesystem
|
||||
|
||||
Target: existing clusters with NFS/CephFS/GlusterFS or another shared mount.
|
||||
|
||||
```toml
|
||||
[storage]
|
||||
backend = "filesystem"
|
||||
path = "/srv/rustbox-shared"
|
||||
shared = true
|
||||
```
|
||||
|
||||
Supported use:
|
||||
|
||||
- draft attachments,
|
||||
- other blob-like application files.
|
||||
|
||||
Do **not** treat a shared filesystem as a supported way to run multiple concurrent Tantivy writers. Tantivy indexing must remain a separate concern.
|
||||
|
||||
Operational notes:
|
||||
|
||||
- identical mount path is not required once SQL stores logical keys rather than absolute paths,
|
||||
- application startup should check read/write access,
|
||||
- health endpoint should expose storage availability,
|
||||
- temporary uploads must be written atomically and cleaned after failure.
|
||||
|
||||
## Variant C — S3-compatible object storage
|
||||
|
||||
Target: recommended multi-node/stateless deployment.
|
||||
|
||||
Supported implementations should include any sufficiently compatible S3 API, for example:
|
||||
|
||||
- AWS S3,
|
||||
- MinIO,
|
||||
- Ceph RGW,
|
||||
- other S3-compatible services.
|
||||
|
||||
Suggested configuration:
|
||||
|
||||
```toml
|
||||
[storage]
|
||||
backend = "s3"
|
||||
bucket = "rustbox"
|
||||
region = "eu-central-1"
|
||||
endpoint = "https://s3.example.net" # optional for AWS, required for many compatible services
|
||||
prefix = "rustbox/"
|
||||
path_style = false
|
||||
|
||||
# Credentials should preferably come from environment/instance identity,
|
||||
# not plaintext TOML.
|
||||
```
|
||||
|
||||
Required behavior:
|
||||
|
||||
- streaming upload/download,
|
||||
- multipart upload for large objects where useful,
|
||||
- checksum/ETag validation where reliable,
|
||||
- abort incomplete multipart uploads,
|
||||
- retry transient errors,
|
||||
- configurable request/connect timeouts,
|
||||
- TLS verification enabled by default,
|
||||
- optional server-side encryption configuration,
|
||||
- lifecycle support for abandoned temporary objects,
|
||||
- no public bucket requirement.
|
||||
|
||||
The database should keep object metadata and logical object keys. RustBox nodes must not depend on any local copy being present.
|
||||
|
||||
## Variant D — additional cloud/object backends
|
||||
|
||||
Future adapters may add:
|
||||
|
||||
```toml
|
||||
backend = "azure_blob"
|
||||
```
|
||||
|
||||
or:
|
||||
|
||||
```toml
|
||||
backend = "gcs"
|
||||
```
|
||||
|
||||
Do not hard-code cloud-specific behavior into attachment code. Keep it behind the storage interface.
|
||||
|
||||
Priority should be:
|
||||
|
||||
1. `filesystem`,
|
||||
2. `s3`,
|
||||
3. additional adapters only when there is demand.
|
||||
|
||||
---
|
||||
|
||||
# 2. Cluster behavior after storage abstraction
|
||||
|
||||
When `storage.backend = "s3"`, RustBox should allow:
|
||||
|
||||
```toml
|
||||
cluster_mode = true
|
||||
cluster_shared_data_dir = false
|
||||
```
|
||||
|
||||
The cluster validation should become capability-based instead of checking only `cluster_shared_data_dir`.
|
||||
|
||||
Expected rules:
|
||||
|
||||
| Configuration | Cluster allowed | Notes |
|
||||
|---|---:|---|
|
||||
| local filesystem | no | unless explicitly marked/verified as shared |
|
||||
| shared POSIX filesystem | yes | attachments/blobs only |
|
||||
| S3-compatible storage | yes | preferred stateless mode |
|
||||
| external index + S3 | yes | full cluster mode |
|
||||
| local Tantivy | no | single-node only |
|
||||
| index disabled | yes | SQL search only |
|
||||
|
||||
Suggested validation logic:
|
||||
|
||||
```text
|
||||
cluster_mode = false
|
||||
-> filesystem + Tantivy allowed
|
||||
|
||||
cluster_mode = true
|
||||
-> blob storage must be cluster-safe
|
||||
-> local Tantivy forbidden
|
||||
-> PGP requires a cluster-safe key backend or must be disabled
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# 3. Search/index backend abstraction
|
||||
|
||||
Search must not be coupled to the blob storage backend.
|
||||
|
||||
Introduce a separate interface such as:
|
||||
|
||||
```text
|
||||
SearchBackend
|
||||
index_message(document)
|
||||
delete_message(message_id)
|
||||
search(query, filters, paging)
|
||||
health()
|
||||
rebuild(...)
|
||||
```
|
||||
|
||||
Suggested configuration:
|
||||
|
||||
```toml
|
||||
[search]
|
||||
backend = "disabled"
|
||||
```
|
||||
|
||||
Possible values:
|
||||
|
||||
```text
|
||||
disabled
|
||||
local_tantivy
|
||||
quickwit
|
||||
opensearch
|
||||
solr
|
||||
```
|
||||
|
||||
Additional adapters can be added later without changing mail/cache storage.
|
||||
|
||||
---
|
||||
|
||||
## Search variant 0 — disabled
|
||||
|
||||
```toml
|
||||
[search]
|
||||
backend = "disabled"
|
||||
```
|
||||
|
||||
Use cases:
|
||||
|
||||
- cluster without external search service,
|
||||
- minimal deployments,
|
||||
- recovery/degraded mode.
|
||||
|
||||
Behavior:
|
||||
|
||||
- SQL-backed metadata/header search remains available,
|
||||
- deep body full-text search is unavailable,
|
||||
- no index files on application nodes.
|
||||
|
||||
This must remain a fully supported configuration.
|
||||
|
||||
---
|
||||
|
||||
## Search variant 1 — local Tantivy
|
||||
|
||||
```toml
|
||||
[search]
|
||||
backend = "local_tantivy"
|
||||
path = "/var/lib/rustbox/data"
|
||||
```
|
||||
|
||||
Use case:
|
||||
|
||||
- single-node RustBox.
|
||||
|
||||
Rules:
|
||||
|
||||
- allowed only when `cluster_mode = false`,
|
||||
- current Tantivy behavior can be retained behind the new `SearchBackend` interface,
|
||||
- index remains rebuildable from authoritative data,
|
||||
- the index must never be treated as the only copy of mailbox data.
|
||||
|
||||
Do not support multi-node writers against one shared Tantivy directory.
|
||||
|
||||
---
|
||||
|
||||
## Search variant 2 — Quickwit
|
||||
|
||||
```toml
|
||||
[search]
|
||||
backend = "quickwit"
|
||||
url = "http://quickwit:7280"
|
||||
index = "rustbox-mails"
|
||||
```
|
||||
|
||||
Why evaluate it:
|
||||
|
||||
- written in Rust,
|
||||
- based on Tantivy,
|
||||
- designed around distributed search and object storage,
|
||||
- conceptually close to the current local Tantivy implementation.
|
||||
|
||||
Required evaluation before implementation:
|
||||
|
||||
- message update/delete semantics,
|
||||
- latency for newly indexed mail,
|
||||
- tenant/user isolation strategy,
|
||||
- reindex cost,
|
||||
- index schema evolution,
|
||||
- attachment/body size limits,
|
||||
- failure/retry behavior,
|
||||
- suitability for mutable mailbox state rather than only append-heavy workloads.
|
||||
|
||||
Do not select Quickwit solely because it uses Tantivy; validate mail-specific update/delete behavior first.
|
||||
|
||||
---
|
||||
|
||||
## Search variant 3 — OpenSearch
|
||||
|
||||
```toml
|
||||
[search]
|
||||
backend = "opensearch"
|
||||
urls = ["https://search-1.example.net:9200", "https://search-2.example.net:9200"]
|
||||
index_prefix = "rustbox"
|
||||
```
|
||||
|
||||
Why support it:
|
||||
|
||||
- mature distributed indexing/search,
|
||||
- common in existing infrastructure,
|
||||
- strong filtering and full-text capabilities,
|
||||
- suitable for larger installations.
|
||||
|
||||
Trade-offs:
|
||||
|
||||
- heavier operational footprint,
|
||||
- JVM memory requirements,
|
||||
- index lifecycle/sharding needs administration.
|
||||
|
||||
OpenSearch is a good enterprise/existing-infrastructure option, but should not become mandatory for RustBox.
|
||||
|
||||
---
|
||||
|
||||
## Search variant 4 — Solr / SolrCloud
|
||||
|
||||
```toml
|
||||
[search]
|
||||
backend = "solr"
|
||||
url = "https://solr.example.net/solr/rustbox"
|
||||
```
|
||||
|
||||
Why keep it as an option:
|
||||
|
||||
- mature Lucene full-text engine,
|
||||
- still used in established environments,
|
||||
- supports distributed SolrCloud deployments.
|
||||
|
||||
Trade-offs:
|
||||
|
||||
- JVM-based,
|
||||
- cluster operation is not simpler than lightweight alternatives,
|
||||
- adds a separate operational stack.
|
||||
|
||||
Treat Solr as an optional adapter for environments that already operate it, not as the default RustBox cluster indexer.
|
||||
|
||||
---
|
||||
|
||||
# 4. Recommended deployment variants
|
||||
|
||||
## Variant 1 — simple single node
|
||||
|
||||
```text
|
||||
RustBox
|
||||
├─ SQLite/MySQL/PostgreSQL
|
||||
├─ local filesystem attachments
|
||||
└─ local Tantivy
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```toml
|
||||
cluster_mode = false
|
||||
|
||||
[storage]
|
||||
backend = "filesystem"
|
||||
path = "/var/lib/rustbox/data"
|
||||
|
||||
[search]
|
||||
backend = "local_tantivy"
|
||||
```
|
||||
|
||||
Best for small installations.
|
||||
|
||||
---
|
||||
|
||||
## Variant 2 — cluster with shared filesystem
|
||||
|
||||
```text
|
||||
RustBox node 1 ─┐
|
||||
RustBox node 2 ─┼── MySQL/PostgreSQL
|
||||
RustBox node 3 ─┘
|
||||
│
|
||||
└──────── shared NFS/CephFS for attachments
|
||||
|
||||
Search: disabled or external
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```toml
|
||||
cluster_mode = true
|
||||
|
||||
[storage]
|
||||
backend = "filesystem"
|
||||
path = "/srv/rustbox-shared"
|
||||
shared = true
|
||||
|
||||
[search]
|
||||
backend = "disabled"
|
||||
```
|
||||
|
||||
This is the lowest-change migration path from 0.1.2.
|
||||
|
||||
---
|
||||
|
||||
## Variant 3 — stateless cluster with S3, no external indexer
|
||||
|
||||
```text
|
||||
RustBox node 1 ─┐
|
||||
RustBox node 2 ─┼── MySQL/PostgreSQL
|
||||
RustBox node 3 ─┘
|
||||
│
|
||||
└──────── S3 / MinIO / Ceph RGW
|
||||
|
||||
Search: SQL-only
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```toml
|
||||
cluster_mode = true
|
||||
|
||||
[storage]
|
||||
backend = "s3"
|
||||
bucket = "rustbox"
|
||||
endpoint = "https://minio.example.net"
|
||||
|
||||
[search]
|
||||
backend = "disabled"
|
||||
```
|
||||
|
||||
This should be a first-class supported production configuration.
|
||||
|
||||
---
|
||||
|
||||
## Variant 4 — stateless cluster with S3 + external indexer
|
||||
|
||||
```text
|
||||
┌── MySQL/PostgreSQL
|
||||
RustBox nodes ──────┼── S3-compatible blob storage
|
||||
└── external search cluster
|
||||
```
|
||||
|
||||
Example with Quickwit:
|
||||
|
||||
```toml
|
||||
cluster_mode = true
|
||||
|
||||
[storage]
|
||||
backend = "s3"
|
||||
bucket = "rustbox"
|
||||
endpoint = "https://s3.example.net"
|
||||
|
||||
[search]
|
||||
backend = "quickwit"
|
||||
url = "http://quickwit:7280"
|
||||
index = "rustbox-mails"
|
||||
```
|
||||
|
||||
Example with OpenSearch:
|
||||
|
||||
```toml
|
||||
[search]
|
||||
backend = "opensearch"
|
||||
urls = ["https://search.example.net:9200"]
|
||||
index_prefix = "rustbox"
|
||||
```
|
||||
|
||||
This is the preferred architecture for larger multi-node installations requiring deep full-text search.
|
||||
|
||||
---
|
||||
|
||||
# 5. Durable indexing pipeline
|
||||
|
||||
Do not make correctness depend on an HTTP request successfully updating the external index synchronously.
|
||||
|
||||
Add a durable indexing queue/outbox in the shared SQL database.
|
||||
|
||||
Conceptual flow:
|
||||
|
||||
```text
|
||||
mail/cache transaction
|
||||
│
|
||||
├── update authoritative SQL data
|
||||
└── append search_outbox event
|
||||
│
|
||||
v
|
||||
index worker(s)
|
||||
│
|
||||
v
|
||||
Quickwit/OpenSearch/Solr
|
||||
```
|
||||
|
||||
Required properties:
|
||||
|
||||
- idempotent events,
|
||||
- retry with backoff,
|
||||
- dead-letter/error visibility,
|
||||
- safe concurrent workers,
|
||||
- delete events,
|
||||
- full reindex command,
|
||||
- index version/schema version,
|
||||
- ability to rebuild the index from authoritative SQL/IMAP data,
|
||||
- index lag metric.
|
||||
|
||||
The external index must be treated as derived/rebuildable state, never as the source of truth.
|
||||
|
||||
---
|
||||
|
||||
# 6. PGP/key storage in cluster mode
|
||||
|
||||
PGP private keys should not automatically follow the generic attachment/blob backend.
|
||||
|
||||
Current safe rule remains:
|
||||
|
||||
```toml
|
||||
pgp_enabled = false
|
||||
```
|
||||
|
||||
for cluster mode until a dedicated key-storage design exists.
|
||||
|
||||
Future variants to evaluate:
|
||||
|
||||
1. encrypted keys in the shared SQL database,
|
||||
2. encrypted objects in S3 with a separate master-key/KMS strategy,
|
||||
3. Vault/HSM/KMS-backed key handling,
|
||||
4. dedicated secrets backend.
|
||||
|
||||
Requirements:
|
||||
|
||||
- encryption at rest independent of ordinary object-storage ACLs,
|
||||
- no plaintext private key in shared object storage,
|
||||
- identical behavior from every RustBox node,
|
||||
- rotation/recovery procedure,
|
||||
- auditability.
|
||||
|
||||
Do not make `cluster_shared_data_dir` the solution for private-key consistency.
|
||||
|
||||
---
|
||||
|
||||
# 7. WebSocket / IMAP IDLE cluster events
|
||||
|
||||
Separate from storage and search, the current in-process event bus is node-local.
|
||||
|
||||
TODO:
|
||||
|
||||
- add optional cluster event backend,
|
||||
- initial candidate: Redis Pub/Sub or Redis Streams,
|
||||
- propagate mailbox/notification invalidations across nodes,
|
||||
- retain local fast-path for single-node mode,
|
||||
- define behavior during broker outages.
|
||||
|
||||
This is not required for SQL cache consistency, but is required for reliable real-time UI updates across nodes.
|
||||
|
||||
---
|
||||
|
||||
# 8. Configuration compatibility
|
||||
|
||||
Keep backward compatibility where possible.
|
||||
|
||||
Legacy:
|
||||
|
||||
```toml
|
||||
data_dir = "/var/lib/rustbox/data"
|
||||
search_index_backend = "tantivy"
|
||||
```
|
||||
|
||||
Migration target:
|
||||
|
||||
```toml
|
||||
[storage]
|
||||
backend = "filesystem"
|
||||
path = "/var/lib/rustbox/data"
|
||||
|
||||
[search]
|
||||
backend = "local_tantivy"
|
||||
```
|
||||
|
||||
Suggested transition:
|
||||
|
||||
- accept legacy keys for at least one release,
|
||||
- emit deprecation warnings,
|
||||
- map legacy values internally to the new backend configuration,
|
||||
- document exact removal version before deleting old settings.
|
||||
|
||||
---
|
||||
|
||||
# 9. Migration of existing attachments
|
||||
|
||||
Provide an explicit migration command rather than silently moving data during normal startup.
|
||||
|
||||
Example target CLI:
|
||||
|
||||
```text
|
||||
rustbox storage migrate \
|
||||
--from filesystem:/var/lib/rustbox/data \
|
||||
--to s3:rustbox \
|
||||
--verify
|
||||
```
|
||||
|
||||
Requirements:
|
||||
|
||||
- resumable,
|
||||
- idempotent,
|
||||
- checksum/size verification,
|
||||
- dry-run mode,
|
||||
- progress reporting,
|
||||
- do not delete source objects until verification passes,
|
||||
- update SQL references transactionally or through a safe staged migration,
|
||||
- rollback/retry documentation.
|
||||
|
||||
---
|
||||
|
||||
# 10. Health, metrics and operations
|
||||
|
||||
Expose at least:
|
||||
|
||||
```text
|
||||
storage backend status
|
||||
storage request failures
|
||||
storage latency
|
||||
index backend status
|
||||
index queue depth
|
||||
index lag
|
||||
index failures/retries
|
||||
last successful index operation
|
||||
```
|
||||
|
||||
Startup behavior:
|
||||
|
||||
- invalid cluster/storage combination -> hard configuration error,
|
||||
- external index unavailable -> application may start in degraded mode if configured,
|
||||
- shared SQL unavailable -> application startup/request handling must fail clearly,
|
||||
- object storage unavailable -> mail reading from SQL may continue, but attachment operations must fail explicitly.
|
||||
|
||||
Suggested option:
|
||||
|
||||
```toml
|
||||
[search]
|
||||
required = false
|
||||
```
|
||||
|
||||
When `required = false`, RustBox can run with SQL search while the external index is unavailable.
|
||||
|
||||
---
|
||||
|
||||
# 11. Security requirements
|
||||
|
||||
For object storage:
|
||||
|
||||
- private buckets/containers only,
|
||||
- least-privilege credentials,
|
||||
- environment/instance/workload identity preferred over TOML secrets,
|
||||
- TLS by default,
|
||||
- optional custom CA bundle for private S3-compatible services,
|
||||
- prevent object-key traversal/injection,
|
||||
- server-side encryption option,
|
||||
- never expose backend credentials to the browser.
|
||||
|
||||
For external search:
|
||||
|
||||
- authenticated TLS connection,
|
||||
- per-user/tenant filtering must be enforced server-side by RustBox,
|
||||
- never trust client-provided tenant/user IDs,
|
||||
- index documents must not allow cross-user search leakage,
|
||||
- document deletion must propagate to the index.
|
||||
|
||||
---
|
||||
|
||||
# 12. Implementation order
|
||||
|
||||
## Phase 1 — storage abstraction
|
||||
|
||||
- [ ] introduce `BlobStorage` interface,
|
||||
- [ ] convert draft attachments from absolute paths to logical object keys,
|
||||
- [ ] implement filesystem backend,
|
||||
- [ ] retain existing behavior for single-node installations,
|
||||
- [ ] add storage health checks.
|
||||
|
||||
## Phase 2 — S3 backend
|
||||
|
||||
- [ ] implement S3-compatible backend,
|
||||
- [ ] streaming upload/download,
|
||||
- [ ] retries/timeouts/checksums,
|
||||
- [ ] MinIO integration validation,
|
||||
- [ ] AWS S3 integration validation,
|
||||
- [ ] Ceph RGW compatibility validation,
|
||||
- [ ] migration command from filesystem to S3,
|
||||
- [ ] remove hard `cluster_shared_data_dir` requirement when storage backend is cluster-safe.
|
||||
|
||||
## Phase 3 — search abstraction
|
||||
|
||||
- [ ] introduce `SearchBackend` interface,
|
||||
- [ ] move current Tantivy code behind `local_tantivy`,
|
||||
- [ ] preserve `disabled` backend,
|
||||
- [ ] reject `local_tantivy` in cluster mode,
|
||||
- [ ] define canonical indexed message document/schema.
|
||||
|
||||
## Phase 4 — durable index events
|
||||
|
||||
- [ ] add SQL search outbox/queue,
|
||||
- [ ] add index worker locking/concurrency,
|
||||
- [ ] retries/backoff,
|
||||
- [ ] delete events,
|
||||
- [ ] reindex command,
|
||||
- [ ] index lag/error metrics.
|
||||
|
||||
## Phase 5 — external index adapters
|
||||
|
||||
Recommended evaluation order:
|
||||
|
||||
- [ ] Quickwit proof of concept,
|
||||
- [ ] OpenSearch adapter,
|
||||
- [ ] compare update/delete/reindex behavior and operational footprint,
|
||||
- [ ] choose documented recommended backend,
|
||||
- [ ] Solr adapter only if there is real deployment demand.
|
||||
|
||||
## Phase 6 — cluster PGP
|
||||
|
||||
- [ ] design dedicated encrypted key backend,
|
||||
- [ ] keep PGP disabled in cluster mode until completed,
|
||||
- [ ] evaluate Vault/KMS/SQL-encrypted/S3-encrypted variants,
|
||||
- [ ] add migration procedure for existing local keys.
|
||||
|
||||
## Phase 7 — cross-node events
|
||||
|
||||
- [ ] introduce cluster event interface,
|
||||
- [ ] Redis implementation,
|
||||
- [ ] WebSocket/IMAP IDLE fan-out,
|
||||
- [ ] reconnection/degraded-mode behavior.
|
||||
|
||||
---
|
||||
|
||||
# 13. Acceptance criteria
|
||||
|
||||
The work is complete when all of the following are true:
|
||||
|
||||
- [ ] three or more RustBox nodes can run with no writable shared POSIX filesystem,
|
||||
- [ ] nodes can be destroyed/recreated without losing attachments or search state,
|
||||
- [ ] S3-compatible storage works for draft attachments,
|
||||
- [ ] local filesystem remains supported for single-node mode,
|
||||
- [ ] `cluster_mode = true` accepts S3-backed storage without `cluster_shared_data_dir = true`,
|
||||
- [ ] local Tantivy is rejected in cluster mode,
|
||||
- [ ] external index failure does not corrupt SQL/cache state,
|
||||
- [ ] full index rebuild is possible,
|
||||
- [ ] index events are durable and retryable,
|
||||
- [ ] SQL remains the authoritative cache/data source,
|
||||
- [ ] no PGP private key depends on node-local filesystem in supported cluster mode,
|
||||
- [ ] configuration and migration paths are documented for every supported deployment variant.
|
||||
|
||||
---
|
||||
|
||||
# Recommended target
|
||||
|
||||
For the long-term cluster architecture, prefer:
|
||||
|
||||
```text
|
||||
RustBox nodes (stateless)
|
||||
│
|
||||
├── PostgreSQL/MySQL authoritative metadata + mailbox cache
|
||||
├── Redis (optional) sessions + future cross-node events
|
||||
├── S3-compatible storage draft attachments / blob data
|
||||
└── external indexer optional deep full-text search
|
||||
```
|
||||
|
||||
Recommended supported profiles:
|
||||
|
||||
1. **Single node:** filesystem + local Tantivy.
|
||||
2. **Small cluster:** shared filesystem or S3 + search disabled.
|
||||
3. **Preferred stateless cluster:** S3-compatible storage + external indexer.
|
||||
4. **Enterprise/existing infrastructure:** S3/shared storage + OpenSearch or optional Solr adapter.
|
||||
|
||||
The key design rule is to keep **SQL cache**, **blob storage**, **search index**, and **private-key storage** as separate concerns. None of them should force all application nodes to share the same writable local directory.
|
||||
Reference in new issue
Block a user