RustBox/docs/STORAGE_SEARCH_OPERATIONS.md

RustBox 0.2.16 storage/search operations

RustBox 0.2.16 treats SQL, blob storage and search as independent capabilities. The database engine does not decide whether data_dir is allowed.

Deployment matrix

Mode	SQL	Blob storage	Search	Persistent data_dir
Single node	SQLite/MySQL/PostgreSQL	local filesystem	local Tantivy	normally yes
Single node	MySQL/PostgreSQL	S3	OpenSearch/disabled	no, if PGP is disabled and no local path is selected
Cluster	MySQL/PostgreSQL	S3-compatible	OpenSearch/disabled	disabled by default
Cluster	MySQL/PostgreSQL	shared POSIX (NFS/CephFS/GlusterFS)	OpenSearch/disabled	disabled by default

In cluster mode an implicit/default data_dir is cleared. An explicitly configured data_dir is still accepted for deliberate local/migration use, but cluster validation rejects local-only features. PostgreSQL/MySQL remain valid with data_dir in single-node deployments.

Blob storage

storage.backend="filesystem" supports both local and shared POSIX filesystems. storage.shared=true is an operator assertion that every RustBox node sees the same path and the same files; RustBox cannot turn a local disk or a host-local Docker volume into shared storage.

For NFS/CephFS/GlusterFS:

• mount the same logical path on every node;
• keep ownership/permissions consistent;
• validate rename/read/write/delete behavior before production;
• monitor filesystem availability and capacity outside RustBox.

storage.backend="s3" supports S3-compatible object storage. The backend streams uploads, applies request/retry timeouts, records SHA-256 attachment metadata and supports S3 server-side encryption options. For multipart uploads interrupted by process/node loss, configure an object-store lifecycle rule to expire abandoned multipart uploads.

Attachment SQL rows store logical object keys. This keeps SQL independent from a node-specific filesystem path.

Durable blob deletion

Attachment deletion is not best-effort. RustBox writes a blob_delete_outbox entry in the same SQL transaction that removes the attachment row or draft row, then attempts the physical delete immediately. A background worker retries failures with claim leases and exponential backoff until the object is gone; object-store NotFound is treated as success. The same path is used for local filesystem, shared POSIX/NFS and S3-compatible backends. Failed finalized uploads and failed migration destination objects are also queued for cleanup.

This avoids orphan accumulation when storage is temporarily unavailable while also avoiding the inverse failure mode where a blob is deleted first and the SQL transaction later fails.

Search and outbox

OpenSearch is the cluster-capable indexer. Every indexed document is scoped by user_hash; queries add a server-side tenant filter and deterministic document IDs are routed by tenant.

Search changes use a durable SQL outbox:

1. message mutations that already use a SQL transaction record the indexing event in that same transaction; other derived-state updates enqueue through the same tenant connection;
2. workers claim events with a lease;
3. OpenSearch operations are idempotent;
4. failures are retried with backoff;
5. exhausted events remain visible as dead-letter state.

Search is derived state. A full rebuild is available with:

rustbox search reindex

The rebuild removes superseded outbox events older than the rebuild start and keeps events created or refreshed during the rebuild. In a multi-node deployment run a full reindex in a controlled maintenance window and quiesce search workers on the other RustBox nodes; otherwise a direct rebuild can race an already claimed event from another node. Normal outbox processing does not require this maintenance step.

If search.required=false, an unavailable OpenSearch backend may leave the application usable while the outbox grows; readiness reports degraded search and workers retry later. With search.required=true, search health is a startup/readiness requirement.

Automatic body cache / deep index

Automatic full-body caching is controlled by search.deep_index. When it is true and the search backend is enabled, every periodic IMAP reconciliation may fetch full MIME bodies for messages that have headers cached but no body/index yet. sync.deep_index_batch_size caps the number of bodies fetched in one pass. Full bodies are stored in the per-account application database/cache and then queued/indexed, so enabling this increases IMAP bandwidth, application-database/search-storage use and indexing CPU/I/O.

search.deep_index=false is the production default. It prevents a server-wide forced full-mailbox download, but preserves the existing per-account deep_index opt-in. Development configuration sets search.deep_index=true so cache/index behavior is exercised without an additional preference change.

sync.interval_secs controls the periodic reconciliation cadence. Active folders still use IMAP IDLE for near-real-time events; the periodic pass is the fallback/background reconciliation for counters, flags and headers in other folders and is also the cadence for automatic deep-index batches. Lower values converge faster but create more IMAP requests.

Failure behavior

• SQL unavailable: authoritative mailbox/session/account operations cannot proceed.
• S3/shared filesystem unavailable: attachment upload/download/delete and storage readiness fail; SQL data is not silently rewritten to another backend.
• OpenSearch unavailable: outbox events remain durable and retry; SQL remains the source for rebuilding the index.
• One RustBox node lost: S3/shared POSIX attachments and shared SQL remain available to other nodes; local-only Tantivy/SQLite are rejected in cluster mode.
• Shared POSIX mounted only on some nodes: configuration may look valid but runtime attachment operations will fail or diverge. storage.shared=true must only be used after infrastructure-level verification.

Attachment migration

Migrate old filesystem attachments to the configured backend with a dry run first:

rustbox storage migrate --from filesystem:/old/data/root --to configured --dry-run
rustbox storage migrate --from filesystem:/old/data/root --to configured --verify

The migration streams each file, verifies size/SHA-256 when requested and changes the SQL row only after a successful destination write. Source files are not deleted automatically. Run the migration during a controlled maintenance window or otherwise prevent concurrent attachment mutation for the affected drafts.

Cluster prerequisites

A supported cluster requires:

• MySQL or PostgreSQL as shared SQL;
• S3-compatible storage or a genuinely shared POSIX filesystem;
• OpenSearch or disabled full-text search;
• pgp_enabled=false until cluster-safe private-key storage exists;
• the same explicit session_encryption_key on every node.

Cross-node realtime event fan-out remains separate from blob/search storage and is not implemented by the 0.2.0 storage work.

Validation status

The source package includes scripts/validate-release-static.py for offline structural checks. Full release validation still requires a Rust 1.99.1 build host and live service tests against the exact PostgreSQL/MySQL, S3/NFS and OpenSearch infrastructure used in production. Live SILO/MinIO/AWS/Ceph RGW and multi-node acceptance items remain intentionally unchecked in TODO.md until executed.