Mateusz Gruszczyński committed
1 parent
cfc74d9db6
commit
dacdf9c27d
76 files changed
+5209
-741
No files matched your search
@@ -1,8 +1,18 @@
|
||||
# RustBox 0.2.6 - implementation status
|
||||
# RustBox 0.2.8 - implementation status
|
||||
|
||||
This file tracks the `0.2.6` source snapshot. Storage/search scope and outstanding acceptance work are tracked in `TODO.md`. A feature is only listed as implemented when working code exists in this source tree; partial items are called out explicitly.
|
||||
This file tracks the `0.2.8` source snapshot. Storage/search scope and outstanding acceptance work are tracked in `TODO.md`. A feature is only listed as implemented when working code exists in this source tree; partial items are called out explicitly.
|
||||
|
||||
## 0.2.6 storage/search architecture
|
||||
## 0.2.8 browser security headers
|
||||
|
||||
- `security_headers_enabled` remains opt-in and disabled by default for compatibility; optional browser-security headers are centralized in this middleware, so disabling it removes the RustBox-added security profile while normal response semantics such as `Content-Type` and route-specific caching remain.
|
||||
- HTML/XHTML documents receive the full document profile: CSP, `X-Frame-Options: DENY`, `Permissions-Policy`, `Cross-Origin-Opener-Policy: same-origin`, `Referrer-Policy: no-referrer` and `X-Content-Type-Options: nosniff`.
|
||||
- The HTML CSP allows only same-origin scripts (`script-src 'self'`) and disables inline script attributes; the theme bootstrap is an external fingerprinted asset. Stylesheets are same-origin only (`style-src 'self'`), while `style-src-attr 'unsafe-inline'` is narrowly retained because the current UI still sets per-element style attributes dynamically.
|
||||
- Private `/api` responses receive `Cache-Control: no-store`, `X-Content-Type-Options: nosniff`, `Cross-Origin-Resource-Policy: same-origin`, `Content-Security-Policy: frame-ancestors 'none'` and legacy-compatible `X-Frame-Options: DENY`. API JSON does not receive the full document CSP, `Permissions-Policy`, COOP or `Referrer-Policy`.
|
||||
- CSS receives `Referrer-Policy: no-referrer` and `X-Content-Type-Options: nosniff`; application assets also receive same-origin CORP without redundant HTML document policies.
|
||||
- HSTS is deliberately not synthesized by RustBox because the application may sit behind a TLS-terminating reverse proxy and cannot reliably determine the public HTTPS boundary; HSTS belongs at that HTTPS edge.
|
||||
- The public `/api/auth/policy` is data-minimized to the single login-form capability `rememberMeEnabled`. Optional feature availability (OpenPGP, Sieve, diagnostics, About) is returned only from authenticated `/api/auth/session-policy`; session/remember durations are not exposed to the browser policy endpoints.
|
||||
|
||||
## 0.2.8 storage/search architecture
|
||||
|
||||
- SQL backend selection (`sqlite`/`mysql`/`pgsql`) is independent from blob storage and search backend selection.
|
||||
- Cluster mode clears an implicit/default `data_dir`; PostgreSQL/MySQL single-node deployments may still intentionally use `data_dir`-backed filesystem storage and local Tantivy.
|
||||
@@ -10,7 +20,7 @@ This file tracks the `0.2.6` source snapshot. Storage/search scope and outstandi
|
||||
- Draft attachment upload/read/delete/snapshot/send paths use the `BlobStorage` abstraction and logical object keys instead of assuming node-local paths.
|
||||
- OpenSearch is available as the cluster-capable indexer. Local Tantivy remains single-node only; search may also be disabled.
|
||||
- Search mutations use a durable SQL outbox with claim/lease, retry/backoff and dead-letter state; the index remains rebuildable derived state.
|
||||
- V047 adds attachment checksum metadata and the search outbox across SQLite, MySQL and PostgreSQL.
|
||||
- V047 adds attachment checksum metadata and the search outbox across SQLite, MySQL and PostgreSQL. V048 adds monotonic folder-counter generations used by compare-and-set reconciliation of authoritative IMAP counts; V049 is reserved as the next migration number.
|
||||
- SQL global-schema initialization is centralized in `db::pool::configure()`; database-backed sessions do not repeat startup DDL for SQLite, MySQL or PostgreSQL.
|
||||
- Docker Compose examples cover standalone SQLite, PostgreSQL + SILO (S3-compatible) + OpenSearch, and MySQL + NFS + OpenSearch.
|
||||
- Live SILO/MinIO/AWS S3/Ceph RGW, live NFS multi-host, live OpenSearch and full multi-node failover acceptance remain release-environment validation items and are not represented as completed here.
|
||||
@@ -26,6 +36,11 @@ This file tracks the `0.2.6` source snapshot. Storage/search scope and outstandi
|
||||
|
||||
### Mail and IMAP workflow
|
||||
|
||||
- Folder totals and unread counters use authoritative IMAP `STATUS (... MESSAGES UNSEEN ...)` data. If STATUS is unavailable for a selected mailbox, RustBox falls back to `UID SEARCH UNSEEN`; it never treats SELECT/EXAMINE `UNSEEN` (the first unseen sequence number) as an unread count.
|
||||
- Folder counter reconciliation runs independently from content synchronization and repairs `MESSAGES/UNSEEN` even when `HIGHESTMODSEQ` is unchanged. A manual counter refresh reuses the cached folder tree while it is fresh and batches STATUS calls on one IMAP session instead of forcing another LIST.
|
||||
- Read/unread/flag mutations update the affected row/badge optimistically, then reconcile against authoritative IMAP state and publish targeted flag/`FolderStatus` events without replacing the whole message list. The active folder is watched through IMAP IDLE; other folders are reconciled every 30 seconds.
|
||||
- Large folders may hydrate message headers incrementally, but the sidebar total/unread badge remains server-authoritative. IDLE/periodic reconciliation keeps the CONDSTORE cursor separate from counter refresh, falls back to a full `UID FLAGS` comparison when CONDSTORE is unavailable, and discards partial IMAP fetch streams rather than applying incomplete state. The top-right activity indicator distinguishes active synchronization/indexing from an incomplete local IMAP header cache and exposes backlog details in its tooltip.
|
||||
- Wide folder badges display `total / unread` while unread is non-zero and total alone at zero; narrow sidebars switch to the compact existing counter presentation.
|
||||
- IMAP folder list, hierarchy, create, create subfolder, rename, delete, subscribe/unsubscribe, plus collapsible branch navigation with per-account remembered expansion state.
|
||||
- Special folders stored per account: Sent, Drafts, Trash, Junk and Archive. Spam/Junk additionally supports a browser-wide default destination with a per-account override and IMAP special-use/name fallback.
|
||||
- Special-folder references follow folder renames and are cleared when a referenced folder is deleted.
|
||||
@@ -98,6 +113,7 @@ This file tracks the `0.2.6` source snapshot. Storage/search scope and outstandi
|
||||
- Account switch/remove/add workflow for multiple authenticated accounts in one browser session.
|
||||
- Active-session list across browsers while the RustBox process is running, with creation/last-active timestamps, friendly browser/device labels, full User-Agent hover detail, individual revoke and revoke-all-others.
|
||||
- Diagnostics panel exposing IMAP capabilities (including persistent custom-keyword support), authenticated SMTP connectivity, ManageSieve status, browser WebSocket state, local synchronization-cache timestamps/counts and live IMAP IDLE task/subscriber counts without returning server host names or credentials. Public liveness (`/api/health`) and local-storage readiness (`/api/ready`) endpoints are available for service supervision.
|
||||
- Optional administrator system diagnostics endpoint (disabled by default) supports explicit IP/CIDR allowlisting plus token/Basic authentication, redacts configured secrets, probes SQL/session/storage/search backends, reports active sessions and IMAP IDLE state, and surfaces bounded node-local security signals for failed logins, 403/429 responses and 5xx failures.
|
||||
- Search-index diagnostics expose cached/indexed counts, with a rebuild operation from the configured account-data backend.
|
||||
- IMAP reconnect control drops reusable request-path sessions and verifies a fresh authenticated connection.
|
||||
- Graceful SIGTERM/Ctrl+C shutdown stops accepting new requests and aborts dedicated IMAP IDLE tasks before process exit.
|
||||
|
||||
+1
-1
@@ -26,7 +26,7 @@ RUSTBOX_CONFIG=./config.dev.toml ./target/release/rustbox --check-config
|
||||
|
||||
Browser JavaScript source lives in functional modules under `src/web/js/`, ordered by `src/web/js/manifest.txt`. `build.rs` concatenates those modules into the embedded `rustbox-ui.js` asset during every Cargo build. The repository has no duplicate `ui/` snapshot; `src/web/` is the only browser UI source.
|
||||
|
||||
## Updating an existing checkout to 0.2.6
|
||||
## Updating an existing checkout to 0.2.8
|
||||
|
||||
Do not overlay the release with `cp *` or another command that skips dotfiles. That can retain stale `.github`, `.env.example`, or `Cargo.lock` files.
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# RustBox 0.2.6 storage/search operations
|
||||
# RustBox 0.2.8 storage/search operations
|
||||
|
||||
RustBox 0.2.6 treats SQL, blob storage and search as independent capabilities. The database engine does not decide whether `data_dir` is allowed.
|
||||
RustBox 0.2.8 treats SQL, blob storage and search as independent capabilities. The database engine does not decide whether `data_dir` is allowed.
|
||||
|
||||
## Deployment matrix
|
||||
|
||||
@@ -50,6 +50,14 @@ The rebuild removes superseded outbox events older than the rebuild start and ke
|
||||
|
||||
If `search.required=false`, an unavailable OpenSearch backend may leave the application usable while the outbox grows; readiness reports degraded search and workers retry later. With `search.required=true`, search health is a startup/readiness requirement.
|
||||
|
||||
### Automatic body cache / deep index
|
||||
|
||||
Automatic full-body caching is controlled by `search.deep_index`. When it is `true` and the search backend is enabled, every periodic IMAP reconciliation may fetch full MIME bodies for messages that have headers cached but no body/index yet. `sync.deep_index_batch_size` caps the number of bodies fetched in one pass. Full bodies are stored in the per-account application database/cache and then queued/indexed, so enabling this increases IMAP bandwidth, application-database/search-storage use and indexing CPU/I/O.
|
||||
|
||||
`search.deep_index=false` is the production default. It prevents a server-wide forced full-mailbox download, but preserves the existing per-account `deep_index` opt-in. Development configuration sets `search.deep_index=true` so cache/index behavior is exercised without an additional preference change.
|
||||
|
||||
`sync.interval_secs` controls the periodic reconciliation cadence. Active folders still use IMAP IDLE for near-real-time events; the periodic pass is the fallback/background reconciliation for counters, flags and headers in other folders and is also the cadence for automatic deep-index batches. Lower values converge faster but create more IMAP requests.
|
||||
|
||||
## Failure behavior
|
||||
|
||||
- SQL unavailable: authoritative mailbox/session/account operations cannot proceed.
|
||||
|
||||
@@ -0,0 +1,149 @@
|
||||
# Administrator system diagnostics endpoint
|
||||
|
||||
RustBox can expose a single administrator-only JSON endpoint for operational and security diagnostics. It is intended for local troubleshooting, monitoring agents and cluster operations. It is **disabled by default** and, when enabled without changing the allowlist, accepts only loopback clients.
|
||||
|
||||
The endpoint is separate from the user-facing `/api/diagnostics/*` routes. It does not require a RustBox mailbox session. Instead, access requires **both**:
|
||||
|
||||
1. the effective client IP to match `diagnostics_endpoint.allowed_networks`, and
|
||||
2. the configured diagnostics token or Basic Auth credentials.
|
||||
|
||||
Requests from a non-allowed address receive HTTP `403`. Requests from an allowed address with missing/invalid diagnostics credentials receive HTTP `401`. Critical backend probe failures return HTTP `503`; healthy or degraded reports return HTTP `200` with `status` set to `ok` or `degraded`.
|
||||
|
||||
## Configuration
|
||||
|
||||
```toml
|
||||
[diagnostics_endpoint]
|
||||
enabled = false
|
||||
path = "/api/system/diagnostics"
|
||||
allowed_networks = ["127.0.0.1/32", "::1/128"]
|
||||
auth = "token" # token | basic
|
||||
# token = "replace-with-at-least-24-random-characters"
|
||||
# basic_username = "rustbox-monitor"
|
||||
# basic_password = "replace-with-a-long-random-password"
|
||||
include_user_identifiers = false
|
||||
probe_timeout_secs = 5
|
||||
security_window_secs = 900
|
||||
```
|
||||
|
||||
`path` must remain below `/api/system/` so it cannot collide with normal application API routes.
|
||||
|
||||
### Token authentication
|
||||
|
||||
Prefer keeping the token outside TOML, for example in `.env` or an orchestrator secret:
|
||||
|
||||
```bash
|
||||
DIAGNOSTICS_ENDPOINT__ENABLED=true
|
||||
DIAGNOSTICS_ENDPOINT__AUTH=token
|
||||
DIAGNOSTICS_ENDPOINT__TOKEN='a-long-random-secret-at-least-24-characters'
|
||||
```
|
||||
|
||||
Example request:
|
||||
|
||||
```bash
|
||||
curl -fsS \
|
||||
-H "Authorization: Bearer $DIAGNOSTICS_TOKEN" \
|
||||
http://127.0.0.1:3001/api/system/diagnostics
|
||||
```
|
||||
|
||||
`X-RustBox-Diagnostics-Token` is accepted as an alternative to the Bearer header for monitoring systems that cannot set an `Authorization` header.
|
||||
|
||||
### Basic authentication
|
||||
|
||||
```toml
|
||||
[diagnostics_endpoint]
|
||||
enabled = true
|
||||
allowed_networks = ["127.0.0.1/32", "::1/128"]
|
||||
auth = "basic"
|
||||
```
|
||||
|
||||
```bash
|
||||
DIAGNOSTICS_ENDPOINT__BASIC_USERNAME='rustbox-monitor'
|
||||
DIAGNOSTICS_ENDPOINT__BASIC_PASSWORD='a-long-random-monitoring-password'
|
||||
```
|
||||
|
||||
Basic Auth must only be used over HTTPS/TLS (directly or at a trusted reverse proxy), because the credentials are not encrypted by the Basic scheme itself.
|
||||
|
||||
## Networks and reverse proxies
|
||||
|
||||
The network allowlist accepts IPv4/IPv6 addresses and CIDRs. Single addresses may also be specified in TOML.
|
||||
|
||||
Example monitoring network:
|
||||
|
||||
```toml
|
||||
[diagnostics_endpoint]
|
||||
enabled = true
|
||||
allowed_networks = [
|
||||
"127.0.0.1/32",
|
||||
"::1/128",
|
||||
"10.42.20.0/24",
|
||||
"2001:db8:42::/64",
|
||||
]
|
||||
auth = "token"
|
||||
```
|
||||
|
||||
For a reverse proxy, configure `trusted_proxies` separately. RustBox only accepts `X-Forwarded-For` for diagnostics when the directly connected proxy is trusted. If proxy headers arrive from an untrusted peer, diagnostics fails closed with HTTP `403` rather than treating the proxy address as the real client.
|
||||
|
||||
```toml
|
||||
trusted_proxies = "127.0.0.1/32,10.42.0.0/16"
|
||||
|
||||
[diagnostics_endpoint]
|
||||
enabled = true
|
||||
allowed_networks = ["10.42.20.0/24"]
|
||||
auth = "token"
|
||||
```
|
||||
|
||||
A trusted proxy should provide a valid `X-Forwarded-For`. `Forwarded` or `X-Real-IP` without `X-Forwarded-For` is deliberately rejected for this endpoint. In production, also restrict the diagnostics path at the firewall, ingress or reverse-proxy layer.
|
||||
|
||||
## Returned data
|
||||
|
||||
The report contains:
|
||||
|
||||
- application version, process uptime/start time and PID; host-wide CPU, memory, load-average and other machine statistics are deliberately not collected;
|
||||
- effective non-secret configuration and feature switches;
|
||||
- IMAP/SMTP/Sieve endpoint configuration without credentials;
|
||||
- SQL backend, engine version, query latency, account/schema versions, SQLite file size or SQL pool size/idle connection count;
|
||||
- session persistence health and aggregate active session/user/browser counts for SQL or Redis;
|
||||
- local-node cached sessions, timestamps and mail endpoints; e-mail/user hash identifiers appear only when `include_user_identifiers=true`;
|
||||
- filesystem/S3 configuration plus a live storage health probe;
|
||||
- Tantivy/OpenSearch configuration; OpenSearch adds cluster health, node/shard counts and index document count;
|
||||
- active IMAP IDLE task/subscriber counts;
|
||||
- request counters and a bounded in-memory security window for failed logins, access denials, rate-limit responses and server errors;
|
||||
- configuration warnings for risky settings such as disabled TLS verification, HTTP OpenSearch/S3, global diagnostics allowlists or production browser security headers being disabled.
|
||||
|
||||
## Secret handling
|
||||
|
||||
The endpoint never returns diagnostics credentials, mailbox passwords, session encryption keys, Redis passwords, S3 access/secret/session keys, OpenSearch passwords or full credential-bearing DSNs. URLs are reduced to scheme/host/port/path and query strings/userinfo are removed. Secret fields are represented only as `...Configured: true|false` where useful.
|
||||
|
||||
`include_user_identifiers=false` is the default. With that setting, local session details omit e-mail addresses and stable user hashes while aggregate user/session counts remain visible.
|
||||
|
||||
All endpoint responses, including `401` and `403`, carry `Cache-Control: no-store`, `Pragma: no-cache`, `X-Content-Type-Options: nosniff` and `Cross-Origin-Resource-Policy: same-origin` independently of the general browser security-header switch.
|
||||
|
||||
## Security signals
|
||||
|
||||
The telemetry buffer is bounded and in-memory; it stores only timestamp, event category, source IP and request path. It does **not** store credentials, Authorization headers, cookies, tokens, request bodies or mail content.
|
||||
|
||||
Within `security_window_secs` the report includes counts and top source IPs for:
|
||||
|
||||
- rejected login attempts;
|
||||
- HTTP `429` rate limiting;
|
||||
- HTTP `403` access denials;
|
||||
- HTTP `5xx` server failures;
|
||||
- denied diagnostics access.
|
||||
|
||||
`bruteForceSignal` becomes true after repeated login rejection/rate-limit activity, while `diagnosticsAttackSignal` flags repeated attempts against the management endpoint. These are preliminary operational signals, not an IDS or an account-lockout mechanism. In a multi-node deployment the counters are node-local; aggregate application/security logs in a central monitoring/SIEM system for cluster-wide detection.
|
||||
|
||||
## Process and host metrics
|
||||
|
||||
The diagnostics endpoint deliberately avoids host-wide metrics and platform-specific process probes. It does not read `/proc`, query system CPU/RAM/load averages, enumerate host resources or add a system-information dependency. The portable process section is limited to the RustBox PID, while process uptime/start time and request/security counters come from RustBox's own in-memory runtime state. This keeps the endpoint low-overhead and portable across supported operating systems.
|
||||
|
||||
## Probe cost
|
||||
|
||||
The endpoint performs live backend checks, so it should not be polled at sub-second frequency:
|
||||
|
||||
- SQL executes lightweight health/version/schema queries;
|
||||
- Redis uses `PING` plus bounded `SCAN`/`MGET` of at most 5000 session records and reports `truncated=true` if the cap is hit;
|
||||
- S3/filesystem health writes and deletes a tiny temporary `_health/...` object;
|
||||
- OpenSearch queries cluster health and the configured index count;
|
||||
- each backend probe has its own `probe_timeout_secs` timeout.
|
||||
|
||||
A polling interval around tens of seconds is appropriate for typical monitoring. Use the normal `/api/health` or `/api/ready` endpoints for very frequent liveness/readiness checks.
|
||||
Reference in new issue
Block a user