v0.2.0

docs/IMPLEMENTATION_STATUS.md

@@ -1,6 +1,18 @@
-# RustBox 0.0.4 - implementation status
+# RustBox 0.2.0 - implementation status
 
-This file tracks the internal `0.0.4` baseline. The user-approved `docs/RUSTBOX_PLAN.md` remains authoritative. A feature is only listed as implemented when working code exists in this source tree; partial items are called out explicitly.
+This file tracks the `0.2.0` source snapshot. Storage/search scope and outstanding acceptance work are tracked in `TODO.md`. A feature is only listed as implemented when working code exists in this source tree; partial items are called out explicitly.
+
+## 0.2.0 storage/search architecture
+
+- SQL backend selection (`sqlite`/`mysql`/`pgsql`) is independent from blob storage and search backend selection.
+- Cluster mode clears an implicit/default `data_dir`; PostgreSQL/MySQL single-node deployments may still intentionally use `data_dir`-backed filesystem storage and local Tantivy.
+- Blob storage supports local/shared POSIX filesystems and S3-compatible object storage. Shared POSIX is an explicit capability (`storage.shared=true`) suitable for operator-provided NFS/CephFS/GlusterFS mounts.
+- Draft attachment upload/read/delete/snapshot/send paths use the `BlobStorage` abstraction and logical object keys instead of assuming node-local paths.
+- OpenSearch is available as the cluster-capable indexer. Local Tantivy remains single-node only; search may also be disabled.
+- Search mutations use a durable SQL outbox with claim/lease, retry/backoff and dead-letter state; the index remains rebuildable derived state.
+- V047 adds attachment checksum metadata and the search outbox across SQLite, MySQL and PostgreSQL.
+- Docker Compose examples cover standalone SQLite, PostgreSQL + SILO (S3-compatible) + OpenSearch, and MySQL + NFS + OpenSearch.
+- Live SILO/MinIO/AWS S3/Ceph RGW, live NFS multi-host, live OpenSearch and full multi-node failover acceptance remain release-environment validation items and are not represented as completed here.
 
 ## Implemented / wired in this working snapshot
 
@@ -39,7 +51,7 @@ This file tracks the internal `0.0.4` baseline. The user-approved `docs/RUSTBOX_
 - HTML, plain-text and Markdown compose modes. Markdown keeps its source text for the plain alternative and generates escaped/sanitized HTML for preview and delivery without accepting raw Markdown HTML.
 - Expanded HTML toolbar: bold, italic, underline, strike, lists, quote, alignment, links, font, size, color, clear formatting, undo/redo.
 - Draft autosave, explicit recovery of saved/browser-emergency drafts, and attachment upload/delete. Deliberate IMAP Drafts snapshots carry a stable RustBox draft identifier; the previous remote snapshot is removed only after the new UID is resolved unambiguously and UIDVALIDITY still matches.
-- Drag-and-drop attachments with visible upload progress. Attachment uploads stream multipart chunks into `.part` files with an enforced size limit, atomic final rename and stale/incomplete temporary-file cleanup instead of buffering the whole attachment in memory.
+- Drag-and-drop attachments with visible upload progress. Attachment uploads stream multipart chunks through `BlobStorage` with an enforced size limit instead of buffering the whole attachment in memory; filesystem and S3-compatible backends share the same logical-object flow.
 - Inline image upload using CID attachments, including paste and drag/drop of PNG/JPEG/GIF/WebP images into the composer. During editing, CID images use authenticated same-origin draft-attachment preview URLs and are serialized back to `cid:` references for saved/sent MIME content. Inline images can be resized, aligned left/center/right/inline and cropped in-app (GIF cropping is intentionally disabled to preserve animation); display-only resizing does not rewrite the source file. Unused CID attachments are cleaned when their image is removed from the editor.
 - Attachment reminder for common English/Polish wording.
 - Multiple identities.
@@ -104,7 +116,7 @@ This file tracks the internal `0.0.4` baseline. The user-approved `docs/RUSTBOX_
 
 - Fixed the `E0499` mutable-borrow failure in `mark_all_read`: the `uid_store()` stream is dropped before `session.logout()`.
 - Fixed a duplicate `else` introduced in the calendar delete handler during the calendar expansion.
-- All executable Rust SQL is centralized under `src/queries/{sqlite,mysql,pgsql}/queries.rs`, including session queries. 
+- All executable Rust SQL is centralized under `src/queries/{sqlite,mysql,pgsql}/queries.rs`, including session queries.
 ## Partially implemented - keep visible as incomplete
 
 - Sieve visual builder now supports a three-condition nested AND/OR group, negation, two actions, capability-aware disabling, managed-rule ordering and scheduled vacation dates. An arbitrary-depth condition tree, arbitrary-length action stack and full capability-aware condition/action matrix remain unfinished.
@@ -125,7 +137,7 @@ This file tracks the internal `0.0.4` baseline. The user-approved `docs/RUSTBOX_
 - Full CardDAV synchronization.
 - Password-change adapters remain unfinished. Session inventory/device metadata is now persistent across restarts and shared-session backends support cross-node listing/revocation.
 - Diagnostics now includes IMAP reconnect, SMTP connectivity, index rebuild and last local folder/message synchronization metadata; deeper background task/queue metrics and repair workflows remain.
-- Full security, acceptance and performance matrix from `docs/RUSTBOX_PLAN.md`.
+- Full live security, acceptance and performance matrix remains release-environment work tracked in `TODO.md`.
 
 ## UI reliability / workflow fixes in the current snapshot
 

docs/RELEASING.md

@@ -26,3 +26,9 @@ RUSTBOX_CONFIG=./config.dev.toml ./target/release/rustbox --check-config
 ## Browser asset build
 
 Browser JavaScript source lives in functional modules under `src/web/js/`, ordered by `src/web/js/manifest.txt`. `build.rs` concatenates those modules into the embedded `rustbox-ui.js` asset during every Cargo build. The repository has no duplicate `ui/` snapshot; `src/web/` is the only browser UI source.
+
+## Updating an existing checkout to 0.2.0
+
+Do not overlay the release with `cp *` or another command that skips dotfiles. That can retain stale `.github`, `.env.example`, or `Cargo.lock` files.
+
+Use a clean checkout/extraction, or a dotfile-aware copy such as `rsync -a` / `cp -a source/. destination/`. The source release intentionally does not ship `Cargo.lock`; CI and `scripts/build-release.sh` generate it before running `--locked` Cargo commands.

docs/STORAGE_SEARCH_OPERATIONS.md

@@ -0,0 +1,86 @@
+# RustBox 0.2.0 storage/search operations
+
+RustBox 0.2.0 treats SQL, blob storage and search as independent capabilities. The database engine does not decide whether `data_dir` is allowed.
+
+## Deployment matrix
+
+| Mode | SQL | Blob storage | Search | Persistent `data_dir` |
+| --- | --- | --- | --- | --- |
+| Single node | SQLite/MySQL/PostgreSQL | local filesystem | local Tantivy | normally yes |
+| Single node | MySQL/PostgreSQL | S3 | OpenSearch/disabled | no, if PGP is disabled and no local path is selected |
+| Cluster | MySQL/PostgreSQL | S3-compatible | OpenSearch/disabled | disabled by default |
+| Cluster | MySQL/PostgreSQL | shared POSIX (NFS/CephFS/GlusterFS) | OpenSearch/disabled | disabled by default |
+
+In cluster mode an implicit/default `data_dir` is cleared. An explicitly configured `data_dir` is still accepted for deliberate local/migration use, but cluster validation rejects local-only features. PostgreSQL/MySQL remain valid with `data_dir` in single-node deployments.
+
+## Blob storage
+
+`storage.backend="filesystem"` supports both local and shared POSIX filesystems. `storage.shared=true` is an operator assertion that every RustBox node sees the same path and the same files; RustBox cannot turn a local disk or a host-local Docker volume into shared storage.
+
+For NFS/CephFS/GlusterFS:
+
+- mount the same logical path on every node;
+- keep ownership/permissions consistent;
+- validate rename/read/write/delete behavior before production;
+- monitor filesystem availability and capacity outside RustBox.
+
+`storage.backend="s3"` supports S3-compatible object storage. The backend streams uploads, applies request/retry timeouts, records SHA-256 attachment metadata and supports S3 server-side encryption options. For multipart uploads interrupted by process/node loss, configure an object-store lifecycle rule to expire abandoned multipart uploads.
+
+Attachment SQL rows store logical object keys. This keeps SQL independent from a node-specific filesystem path.
+
+## Search and outbox
+
+OpenSearch is the cluster-capable indexer. Every indexed document is scoped by `user_hash`; queries add a server-side tenant filter and deterministic document IDs are routed by tenant.
+
+Search changes use a durable SQL outbox:
+
+1. message mutations that already use a SQL transaction record the indexing event in that same transaction; other derived-state updates enqueue through the same tenant connection;
+2. workers claim events with a lease;
+3. OpenSearch operations are idempotent;
+4. failures are retried with backoff;
+5. exhausted events remain visible as dead-letter state.
+
+Search is derived state. A full rebuild is available with:
+
+```bash
+rustbox search reindex
+```
+
+The rebuild removes superseded outbox events older than the rebuild start and keeps events created or refreshed during the rebuild. In a multi-node deployment run a full reindex in a controlled maintenance window and quiesce search workers on the other RustBox nodes; otherwise a direct rebuild can race an already claimed event from another node. Normal outbox processing does not require this maintenance step.
+
+If `search.required=false`, an unavailable OpenSearch backend may leave the application usable while the outbox grows; readiness reports degraded search and workers retry later. With `search.required=true`, search health is a startup/readiness requirement.
+
+## Failure behavior
+
+- SQL unavailable: authoritative mailbox/session/account operations cannot proceed.
+- S3/shared filesystem unavailable: attachment upload/download/delete and storage readiness fail; SQL data is not silently rewritten to another backend.
+- OpenSearch unavailable: outbox events remain durable and retry; SQL remains the source for rebuilding the index.
+- One RustBox node lost: S3/shared POSIX attachments and shared SQL remain available to other nodes; local-only Tantivy/SQLite are rejected in cluster mode.
+- Shared POSIX mounted only on some nodes: configuration may look valid but runtime attachment operations will fail or diverge. `storage.shared=true` must only be used after infrastructure-level verification.
+
+## Attachment migration
+
+Migrate old filesystem attachments to the configured backend with a dry run first:
+
+```bash
+rustbox storage migrate --from filesystem:/old/data/root --to configured --dry-run
+rustbox storage migrate --from filesystem:/old/data/root --to configured --verify
+```
+
+The migration streams each file, verifies size/SHA-256 when requested and changes the SQL row only after a successful destination write. Source files are not deleted automatically. Run the migration during a controlled maintenance window or otherwise prevent concurrent attachment mutation for the affected drafts.
+
+## Cluster prerequisites
+
+A supported cluster requires:
+
+- MySQL or PostgreSQL as shared SQL;
+- S3-compatible storage or a genuinely shared POSIX filesystem;
+- OpenSearch or disabled full-text search;
+- `pgp_enabled=false` until cluster-safe private-key storage exists;
+- the same explicit `session_encryption_key` on every node.
+
+Cross-node realtime event fan-out remains separate from blob/search storage and is not implemented by the 0.2.0 storage work.
+
+## Validation status
+
+The source package includes `scripts/validate-release-static.py` for offline structural checks. Full release validation still requires a Rust 1.88 build host and live service tests against the exact PostgreSQL/MySQL, S3/NFS and OpenSearch infrastructure used in production. Live SILO/MinIO/AWS/Ceph RGW and multi-node acceptance items remain intentionally unchecked in `TODO.md` until executed.