RustBox/docs/STORAGE_SEARCH_OPERATIONS.md

RustBox 0.2.3 storage/search operations

RustBox 0.2.3 treats SQL, blob storage and search as independent capabilities. The database engine does not decide whether data_dir is allowed.

Deployment matrix

Mode	SQL	Blob storage	Search	Persistent data_dir
Single node	SQLite/MySQL/PostgreSQL	local filesystem	local Tantivy	normally yes
Single node	MySQL/PostgreSQL	S3	OpenSearch/disabled	no, if PGP is disabled and no local path is selected
Cluster	MySQL/PostgreSQL	S3-compatible	OpenSearch/disabled	disabled by default
Cluster	MySQL/PostgreSQL	shared POSIX (NFS/CephFS/GlusterFS)	OpenSearch/disabled	disabled by default

In cluster mode an implicit/default data_dir is cleared. An explicitly configured data_dir is still accepted for deliberate local/migration use, but cluster validation rejects local-only features. PostgreSQL/MySQL remain valid with data_dir in single-node deployments.

Blob storage

storage.backend="filesystem" supports both local and shared POSIX filesystems. storage.shared=true is an operator assertion that every RustBox node sees the same path and the same files; RustBox cannot turn a local disk or a host-local Docker volume into shared storage.

For NFS/CephFS/GlusterFS:

• mount the same logical path on every node;
• keep ownership/permissions consistent;
• validate rename/read/write/delete behavior before production;
• monitor filesystem availability and capacity outside RustBox.

storage.backend="s3" supports S3-compatible object storage. The backend streams uploads, applies request/retry timeouts, records SHA-256 attachment metadata and supports S3 server-side encryption options. For multipart uploads interrupted by process/node loss, configure an object-store lifecycle rule to expire abandoned multipart uploads.

Attachment SQL rows store logical object keys. This keeps SQL independent from a node-specific filesystem path.

Search and outbox

OpenSearch is the cluster-capable indexer. Every indexed document is scoped by user_hash; queries add a server-side tenant filter and deterministic document IDs are routed by tenant.

Search changes use a durable SQL outbox:

1. message mutations that already use a SQL transaction record the indexing event in that same transaction; other derived-state updates enqueue through the same tenant connection;
2. workers claim events with a lease;
3. OpenSearch operations are idempotent;
4. failures are retried with backoff;
5. exhausted events remain visible as dead-letter state.

Search is derived state. A full rebuild is available with:

rustbox search reindex

The rebuild removes superseded outbox events older than the rebuild start and keeps events created or refreshed during the rebuild. In a multi-node deployment run a full reindex in a controlled maintenance window and quiesce search workers on the other RustBox nodes; otherwise a direct rebuild can race an already claimed event from another node. Normal outbox processing does not require this maintenance step.

If search.required=false, an unavailable OpenSearch backend may leave the application usable while the outbox grows; readiness reports degraded search and workers retry later. With search.required=true, search health is a startup/readiness requirement.

Failure behavior

• SQL unavailable: authoritative mailbox/session/account operations cannot proceed.
• S3/shared filesystem unavailable: attachment upload/download/delete and storage readiness fail; SQL data is not silently rewritten to another backend.
• OpenSearch unavailable: outbox events remain durable and retry; SQL remains the source for rebuilding the index.
• One RustBox node lost: S3/shared POSIX attachments and shared SQL remain available to other nodes; local-only Tantivy/SQLite are rejected in cluster mode.
• Shared POSIX mounted only on some nodes: configuration may look valid but runtime attachment operations will fail or diverge. storage.shared=true must only be used after infrastructure-level verification.

Attachment migration

Migrate old filesystem attachments to the configured backend with a dry run first:

rustbox storage migrate --from filesystem:/old/data/root --to configured --dry-run
rustbox storage migrate --from filesystem:/old/data/root --to configured --verify

The migration streams each file, verifies size/SHA-256 when requested and changes the SQL row only after a successful destination write. Source files are not deleted automatically. Run the migration during a controlled maintenance window or otherwise prevent concurrent attachment mutation for the affected drafts.

Cluster prerequisites

A supported cluster requires:

• MySQL or PostgreSQL as shared SQL;
• S3-compatible storage or a genuinely shared POSIX filesystem;
• OpenSearch or disabled full-text search;
• pgp_enabled=false until cluster-safe private-key storage exists;
• the same explicit session_encryption_key on every node.

Cross-node realtime event fan-out remains separate from blob/search storage and is not implemented by the 0.2.0 storage work.

Validation status

The source package includes scripts/validate-release-static.py for offline structural checks. Full release validation still requires a Rust 1.88 build host and live service tests against the exact PostgreSQL/MySQL, S3/NFS and OpenSearch infrastructure used in production. Live SILO/MinIO/AWS/Ceph RGW and multi-node acceptance items remain intentionally unchecked in TODO.md until executed.