101 lines
7.4 KiB
Markdown
101 lines
7.4 KiB
Markdown
# RustBox 0.2.16 storage/search operations
|
|
|
|
RustBox 0.2.16 treats SQL, blob storage and search as independent capabilities. The database engine does not decide whether `data_dir` is allowed.
|
|
|
|
## Deployment matrix
|
|
|
|
| Mode | SQL | Blob storage | Search | Persistent `data_dir` |
|
|
| --- | --- | --- | --- | --- |
|
|
| Single node | SQLite/MySQL/PostgreSQL | local filesystem | local Tantivy | normally yes |
|
|
| Single node | MySQL/PostgreSQL | S3 | OpenSearch/disabled | no, if PGP is disabled and no local path is selected |
|
|
| Cluster | MySQL/PostgreSQL | S3-compatible | OpenSearch/disabled | disabled by default |
|
|
| Cluster | MySQL/PostgreSQL | shared POSIX (NFS/CephFS/GlusterFS) | OpenSearch/disabled | disabled by default |
|
|
|
|
In cluster mode an implicit/default `data_dir` is cleared. An explicitly configured `data_dir` is still accepted for deliberate local/migration use, but cluster validation rejects local-only features. PostgreSQL/MySQL remain valid with `data_dir` in single-node deployments.
|
|
|
|
## Blob storage
|
|
|
|
`storage.backend="filesystem"` supports both local and shared POSIX filesystems. `storage.shared=true` is an operator assertion that every RustBox node sees the same path and the same files; RustBox cannot turn a local disk or a host-local Docker volume into shared storage.
|
|
|
|
For NFS/CephFS/GlusterFS:
|
|
|
|
- mount the same logical path on every node;
|
|
- keep ownership/permissions consistent;
|
|
- validate rename/read/write/delete behavior before production;
|
|
- monitor filesystem availability and capacity outside RustBox.
|
|
|
|
`storage.backend="s3"` supports S3-compatible object storage. The backend streams uploads, applies request/retry timeouts, records SHA-256 attachment metadata and supports S3 server-side encryption options. For multipart uploads interrupted by process/node loss, configure an object-store lifecycle rule to expire abandoned multipart uploads.
|
|
|
|
Attachment SQL rows store logical object keys. This keeps SQL independent from a node-specific filesystem path.
|
|
|
|
## Durable blob deletion
|
|
|
|
Attachment deletion is not best-effort. RustBox writes a `blob_delete_outbox` entry in the same SQL transaction that removes the attachment row or draft row, then attempts the physical delete immediately. A background worker retries failures with claim leases and exponential backoff until the object is gone; object-store `NotFound` is treated as success. The same path is used for local filesystem, shared POSIX/NFS and S3-compatible backends. Failed finalized uploads and failed migration destination objects are also queued for cleanup.
|
|
|
|
This avoids orphan accumulation when storage is temporarily unavailable while also avoiding the inverse failure mode where a blob is deleted first and the SQL transaction later fails.
|
|
|
|
## Search and outbox
|
|
|
|
OpenSearch is the cluster-capable indexer. Every indexed document is scoped by `user_hash`; queries add a server-side tenant filter and deterministic document IDs are routed by tenant.
|
|
|
|
Search changes use a durable SQL outbox:
|
|
|
|
1. message mutations that already use a SQL transaction record the indexing event in that same transaction; other derived-state updates enqueue through the same tenant connection;
|
|
2. workers claim events with a lease;
|
|
3. OpenSearch operations are idempotent;
|
|
4. failures are retried with backoff;
|
|
5. exhausted events remain visible as dead-letter state.
|
|
|
|
Search is derived state. A full rebuild is available with:
|
|
|
|
```bash
|
|
rustbox search reindex
|
|
```
|
|
|
|
The rebuild removes superseded outbox events older than the rebuild start and keeps events created or refreshed during the rebuild. In a multi-node deployment run a full reindex in a controlled maintenance window and quiesce search workers on the other RustBox nodes; otherwise a direct rebuild can race an already claimed event from another node. Normal outbox processing does not require this maintenance step.
|
|
|
|
If `search.required=false`, an unavailable OpenSearch backend may leave the application usable while the outbox grows; readiness reports degraded search and workers retry later. With `search.required=true`, search health is a startup/readiness requirement.
|
|
|
|
### Automatic body cache / deep index
|
|
|
|
Automatic full-body caching is controlled by `search.deep_index`. When it is `true` and the search backend is enabled, every periodic IMAP reconciliation may fetch full MIME bodies for messages that have headers cached but no body/index yet. `sync.deep_index_batch_size` caps the number of bodies fetched in one pass. Full bodies are stored in the per-account application database/cache and then queued/indexed, so enabling this increases IMAP bandwidth, application-database/search-storage use and indexing CPU/I/O.
|
|
|
|
`search.deep_index=false` is the production default. It prevents a server-wide forced full-mailbox download, but preserves the existing per-account `deep_index` opt-in. Development configuration sets `search.deep_index=true` so cache/index behavior is exercised without an additional preference change.
|
|
|
|
`sync.interval_secs` controls the periodic reconciliation cadence. Active folders still use IMAP IDLE for near-real-time events; the periodic pass is the fallback/background reconciliation for counters, flags and headers in other folders and is also the cadence for automatic deep-index batches. Lower values converge faster but create more IMAP requests.
|
|
|
|
## Failure behavior
|
|
|
|
- SQL unavailable: authoritative mailbox/session/account operations cannot proceed.
|
|
- S3/shared filesystem unavailable: attachment upload/download/delete and storage readiness fail; SQL data is not silently rewritten to another backend.
|
|
- OpenSearch unavailable: outbox events remain durable and retry; SQL remains the source for rebuilding the index.
|
|
- One RustBox node lost: S3/shared POSIX attachments and shared SQL remain available to other nodes; local-only Tantivy/SQLite are rejected in cluster mode.
|
|
- Shared POSIX mounted only on some nodes: configuration may look valid but runtime attachment operations will fail or diverge. `storage.shared=true` must only be used after infrastructure-level verification.
|
|
|
|
## Attachment migration
|
|
|
|
Migrate old filesystem attachments to the configured backend with a dry run first:
|
|
|
|
```bash
|
|
rustbox storage migrate --from filesystem:/old/data/root --to configured --dry-run
|
|
rustbox storage migrate --from filesystem:/old/data/root --to configured --verify
|
|
```
|
|
|
|
The migration streams each file, verifies size/SHA-256 when requested and changes the SQL row only after a successful destination write. Source files are not deleted automatically. Run the migration during a controlled maintenance window or otherwise prevent concurrent attachment mutation for the affected drafts.
|
|
|
|
## Cluster prerequisites
|
|
|
|
A supported cluster requires:
|
|
|
|
- MySQL or PostgreSQL as shared SQL;
|
|
- S3-compatible storage or a genuinely shared POSIX filesystem;
|
|
- OpenSearch or disabled full-text search;
|
|
- `pgp_enabled=false` until cluster-safe private-key storage exists;
|
|
- the same explicit `session_encryption_key` on every node.
|
|
|
|
Cross-node realtime event fan-out remains separate from blob/search storage and is not implemented by the 0.2.0 storage work.
|
|
|
|
## Validation status
|
|
|
|
The source package includes `scripts/validate-release-static.py` for offline structural checks. Full release validation still requires a Rust 1.99.1 build host and live service tests against the exact PostgreSQL/MySQL, S3/NFS and OpenSearch infrastructure used in production. Live SILO/MinIO/AWS/Ceph RGW and multi-node acceptance items remain intentionally unchecked in `TODO.md` until executed.
|