# RustBox 0.2.16 storage/search operations

RustBox 0.2.16 treats SQL, blob storage and search as independent capabilities. The database engine does not decide whether `data_dir` is allowed.

## Deployment matrix

| Mode | SQL | Blob storage | Search | Persistent `data_dir` |
| --- | --- | --- | --- | --- |
| Single node | SQLite/MySQL/PostgreSQL | local filesystem | local Tantivy | normally yes |
| Single node | MySQL/PostgreSQL | S3 | OpenSearch/disabled | no, if PGP is disabled and no local path is selected |
| Cluster | MySQL/PostgreSQL | S3-compatible | OpenSearch/disabled | disabled by default |
| Cluster | MySQL/PostgreSQL | shared POSIX (NFS/CephFS/GlusterFS) | OpenSearch/disabled | disabled by default |

In cluster mode an implicit/default `data_dir` is cleared. An explicitly configured `data_dir` is still accepted for deliberate local/migration use, but cluster validation rejects local-only features. PostgreSQL/MySQL remain valid with `data_dir` in single-node deployments.

## Blob storage

`storage.backend="filesystem"` supports both local and shared POSIX filesystems. `storage.shared=true` is an operator assertion that every RustBox node sees the same path and the same files; RustBox cannot turn a local disk or a host-local Docker volume into shared storage.

For NFS/CephFS/GlusterFS:

- mount the same logical path on every node;
- keep ownership/permissions consistent;
- validate rename/read/write/delete behavior before production;
- monitor filesystem availability and capacity outside RustBox.

`storage.backend="s3"` supports S3-compatible object storage. The backend streams uploads, applies request/retry timeouts, records SHA-256 attachment metadata and supports S3 server-side encryption options. For multipart uploads interrupted by process/node loss, configure an object-store lifecycle rule to expire abandoned multipart uploads.

Attachment SQL rows store logical object keys. This keeps SQL independent from a node-specific filesystem path.

## Durable blob deletion

Attachment deletion is not best-effort. RustBox writes a `blob_delete_outbox` entry in the same SQL transaction that removes the attachment row or draft row, then attempts the physical delete immediately. A background worker retries failures with claim leases and exponential backoff until the object is gone; object-store `NotFound` is treated as success. The same path is used for local filesystem, shared POSIX/NFS and S3-compatible backends. Failed finalized uploads and failed migration destination objects are also queued for cleanup.

This avoids orphan accumulation when storage is temporarily unavailable while also avoiding the inverse failure mode where a blob is deleted first and the SQL transaction later fails.

## Search and outbox

OpenSearch is the cluster-capable indexer. Every indexed document is scoped by `user_hash`; queries add a server-side tenant filter and deterministic document IDs are routed by tenant.

Search changes use a durable SQL outbox:

1. message mutations that already use a SQL transaction record the indexing event in that same transaction; other derived-state updates enqueue through the same tenant connection;
2. workers claim events with a lease;
3. OpenSearch operations are idempotent;
4. failures are retried with backoff;
5. exhausted events remain visible as dead-letter state.

Search is derived state. A full rebuild is available with:

```bash
rustbox search reindex
```

The rebuild removes superseded outbox events older than the rebuild start and keeps events created or refreshed during the rebuild. In a multi-node deployment run a full reindex in a controlled maintenance window and quiesce search workers on the other RustBox nodes; otherwise a direct rebuild can race an already claimed event from another node. Normal outbox processing does not require this maintenance step.

If `search.required=false`, an unavailable OpenSearch backend may leave the application usable while the outbox grows; readiness reports degraded search and workers retry later. With `search.required=true`, search health is a startup/readiness requirement.

### Automatic body cache / deep index

Automatic full-body caching is controlled by `search.deep_index`. When it is `true` and the search backend is enabled, every periodic IMAP reconciliation may fetch full MIME bodies for messages that have headers cached but no body/index yet. `sync.deep_index_batch_size` caps the number of bodies fetched in one pass. Full bodies are stored in the per-account application database/cache and then queued/indexed, so enabling this increases IMAP bandwidth, application-database/search-storage use and indexing CPU/I/O.

`search.deep_index=false` is the production default. It prevents a server-wide forced full-mailbox download, but preserves the existing per-account `deep_index` opt-in. Development configuration sets `search.deep_index=true` so cache/index behavior is exercised without an additional preference change.

`sync.interval_secs` controls the periodic reconciliation cadence. Active folders still use IMAP IDLE for near-real-time events; the periodic pass is the fallback/background reconciliation for counters, flags and headers in other folders and is also the cadence for automatic deep-index batches. Lower values converge faster but create more IMAP requests.

## Failure behavior

- SQL unavailable: authoritative mailbox/session/account operations cannot proceed.
- S3/shared filesystem unavailable: attachment upload/download/delete and storage readiness fail; SQL data is not silently rewritten to another backend.
- OpenSearch unavailable: outbox events remain durable and retry; SQL remains the source for rebuilding the index.
- One RustBox node lost: S3/shared POSIX attachments and shared SQL remain available to other nodes; local-only Tantivy/SQLite are rejected in cluster mode.
- Shared POSIX mounted only on some nodes: configuration may look valid but runtime attachment operations will fail or diverge. `storage.shared=true` must only be used after infrastructure-level verification.

## Attachment migration

Migrate old filesystem attachments to the configured backend with a dry run first:

```bash
rustbox storage migrate --from filesystem:/old/data/root --to configured --dry-run
rustbox storage migrate --from filesystem:/old/data/root --to configured --verify
```

The migration streams each file, verifies size/SHA-256 when requested and changes the SQL row only after a successful destination write. Source files are not deleted automatically. Run the migration during a controlled maintenance window or otherwise prevent concurrent attachment mutation for the affected drafts.

## Cluster prerequisites

A supported cluster requires:

- MySQL or PostgreSQL as shared SQL;
- S3-compatible storage or a genuinely shared POSIX filesystem;
- OpenSearch or disabled full-text search;
- `pgp_enabled=false` until cluster-safe private-key storage exists;
- the same explicit `session_encryption_key` on every node.

Cross-node realtime event fan-out remains separate from blob/search storage and is not implemented by the 0.2.0 storage work.

## Validation status

The source package includes `scripts/validate-release-static.py` for offline structural checks. Full release validation still requires a Rust 1.99.1 build host and live service tests against the exact PostgreSQL/MySQL, S3/NFS and OpenSearch infrastructure used in production. Live SILO/MinIO/AWS/Ceph RGW and multi-node acceptance items remain intentionally unchecked in `TODO.md` until executed.