87 lines
5.4 KiB
Markdown
87 lines
5.4 KiB
Markdown
# RustBox 0.2.0 storage/search operations
|
|
|
|
RustBox 0.2.0 treats SQL, blob storage and search as independent capabilities. The database engine does not decide whether `data_dir` is allowed.
|
|
|
|
## Deployment matrix
|
|
|
|
| Mode | SQL | Blob storage | Search | Persistent `data_dir` |
|
|
| --- | --- | --- | --- | --- |
|
|
| Single node | SQLite/MySQL/PostgreSQL | local filesystem | local Tantivy | normally yes |
|
|
| Single node | MySQL/PostgreSQL | S3 | OpenSearch/disabled | no, if PGP is disabled and no local path is selected |
|
|
| Cluster | MySQL/PostgreSQL | S3-compatible | OpenSearch/disabled | disabled by default |
|
|
| Cluster | MySQL/PostgreSQL | shared POSIX (NFS/CephFS/GlusterFS) | OpenSearch/disabled | disabled by default |
|
|
|
|
In cluster mode an implicit/default `data_dir` is cleared. An explicitly configured `data_dir` is still accepted for deliberate local/migration use, but cluster validation rejects local-only features. PostgreSQL/MySQL remain valid with `data_dir` in single-node deployments.
|
|
|
|
## Blob storage
|
|
|
|
`storage.backend="filesystem"` supports both local and shared POSIX filesystems. `storage.shared=true` is an operator assertion that every RustBox node sees the same path and the same files; RustBox cannot turn a local disk or a host-local Docker volume into shared storage.
|
|
|
|
For NFS/CephFS/GlusterFS:
|
|
|
|
- mount the same logical path on every node;
|
|
- keep ownership/permissions consistent;
|
|
- validate rename/read/write/delete behavior before production;
|
|
- monitor filesystem availability and capacity outside RustBox.
|
|
|
|
`storage.backend="s3"` supports S3-compatible object storage. The backend streams uploads, applies request/retry timeouts, records SHA-256 attachment metadata and supports S3 server-side encryption options. For multipart uploads interrupted by process/node loss, configure an object-store lifecycle rule to expire abandoned multipart uploads.
|
|
|
|
Attachment SQL rows store logical object keys. This keeps SQL independent from a node-specific filesystem path.
|
|
|
|
## Search and outbox
|
|
|
|
OpenSearch is the cluster-capable indexer. Every indexed document is scoped by `user_hash`; queries add a server-side tenant filter and deterministic document IDs are routed by tenant.
|
|
|
|
Search changes use a durable SQL outbox:
|
|
|
|
1. message mutations that already use a SQL transaction record the indexing event in that same transaction; other derived-state updates enqueue through the same tenant connection;
|
|
2. workers claim events with a lease;
|
|
3. OpenSearch operations are idempotent;
|
|
4. failures are retried with backoff;
|
|
5. exhausted events remain visible as dead-letter state.
|
|
|
|
Search is derived state. A full rebuild is available with:
|
|
|
|
```bash
|
|
rustbox search reindex
|
|
```
|
|
|
|
The rebuild removes superseded outbox events older than the rebuild start and keeps events created or refreshed during the rebuild. In a multi-node deployment run a full reindex in a controlled maintenance window and quiesce search workers on the other RustBox nodes; otherwise a direct rebuild can race an already claimed event from another node. Normal outbox processing does not require this maintenance step.
|
|
|
|
If `search.required=false`, an unavailable OpenSearch backend may leave the application usable while the outbox grows; readiness reports degraded search and workers retry later. With `search.required=true`, search health is a startup/readiness requirement.
|
|
|
|
## Failure behavior
|
|
|
|
- SQL unavailable: authoritative mailbox/session/account operations cannot proceed.
|
|
- S3/shared filesystem unavailable: attachment upload/download/delete and storage readiness fail; SQL data is not silently rewritten to another backend.
|
|
- OpenSearch unavailable: outbox events remain durable and retry; SQL remains the source for rebuilding the index.
|
|
- One RustBox node lost: S3/shared POSIX attachments and shared SQL remain available to other nodes; local-only Tantivy/SQLite are rejected in cluster mode.
|
|
- Shared POSIX mounted only on some nodes: configuration may look valid but runtime attachment operations will fail or diverge. `storage.shared=true` must only be used after infrastructure-level verification.
|
|
|
|
## Attachment migration
|
|
|
|
Migrate old filesystem attachments to the configured backend with a dry run first:
|
|
|
|
```bash
|
|
rustbox storage migrate --from filesystem:/old/data/root --to configured --dry-run
|
|
rustbox storage migrate --from filesystem:/old/data/root --to configured --verify
|
|
```
|
|
|
|
The migration streams each file, verifies size/SHA-256 when requested and changes the SQL row only after a successful destination write. Source files are not deleted automatically. Run the migration during a controlled maintenance window or otherwise prevent concurrent attachment mutation for the affected drafts.
|
|
|
|
## Cluster prerequisites
|
|
|
|
A supported cluster requires:
|
|
|
|
- MySQL or PostgreSQL as shared SQL;
|
|
- S3-compatible storage or a genuinely shared POSIX filesystem;
|
|
- OpenSearch or disabled full-text search;
|
|
- `pgp_enabled=false` until cluster-safe private-key storage exists;
|
|
- the same explicit `session_encryption_key` on every node.
|
|
|
|
Cross-node realtime event fan-out remains separate from blob/search storage and is not implemented by the 0.2.0 storage work.
|
|
|
|
## Validation status
|
|
|
|
The source package includes `scripts/validate-release-static.py` for offline structural checks. Full release validation still requires a Rust 1.88 build host and live service tests against the exact PostgreSQL/MySQL, S3/NFS and OpenSearch infrastructure used in production. Live SILO/MinIO/AWS/Ceph RGW and multi-node acceptance items remain intentionally unchecked in `TODO.md` until executed.
|