v0.2.1
RustBox CI / rustbox (push) Canceled after 0s

This commit is contained in:
Mateusz Gruszczyński committed 2026-10-31 15:55:47 +01:00
1 parent 0d9f3ff447
commit 95c0587c1e
26 files changed
+237 -52

No files matched your search

+10
View File
@@ -1,5 +1,15 @@
# RustBox changelog # RustBox changelog
### 0.2.1
- Fixed Docker development authentication by documenting and defaulting Compose to `ENVIRONMENT=development`, so session cookies work over direct HTTP while production keeps `Secure` cookies behind HTTPS.
- Exposed `pgp_enabled` through the auth policy and hid OpenPGP settings, identity controls, composer controls and reader UI when OpenPGP is disabled by administrator policy.
- Prevented disabled OpenPGP from producing repeated UI error toasts or unnecessary API requests.
- Added OpenSearch outbox worker backoff for HTTP 429/flood-stage/read-only conditions and stopped the current batch after the first backpressure failure, preventing log storms while preserving durable retries without consuming dead-letter attempts.
- Kept OpenSearch disk safety thresholds intact; Docker documentation explains that named volumes consume the Docker host filesystem and how to diagnose/recover flood-stage blocks.
- Added a stable `rustbox` Compose project name, documented Docker environment variables, and made application port mapping configurable through `PORT`.
- Reduced PostgreSQL startup log noise in the Docker example by suppressing repeated SQLx `NOTICE` messages at the default log filter.
- Bumped the application/source release to 0.2.1.
### 0.2.0 ### 0.2.0
- Fixed nested database transactions during message move by reusing the caller-owned transaction for cache removal, header-sync invalidation, folder invalidation, and search outbox writes. - Fixed nested database transactions during message move by reusing the caller-owned transaction for cache removal, header-sync invalidation, folder invalidation, and search outbox writes.
- Kept `search_outbox` in sync between tenant and legacy-import table sets so schema/import parity tests pass. - Kept `search_outbox` in sync between tenant and legacy-import table sets so schema/import parity tests pass.
Generated
+1 -1
View File
@@ -3508,7 +3508,7 @@ dependencies = [
[[package]] [[package]]
name = "rustbox" name = "rustbox"
version = "0.2.0" version = "0.2.1"
dependencies = [ dependencies = [
"aes-gcm", "aes-gcm",
"async-imap", "async-imap",
+1 -1
View File
@@ -1,6 +1,6 @@
[package] [package]
name = "rustbox" name = "rustbox"
version = "0.2.0" version = "0.2.1"
edition = "2024" edition = "2024"
rust-version = "1.88" rust-version = "1.88"
default-run = "rustbox" default-run = "rustbox"
+1 -1
View File
@@ -1,6 +1,6 @@
# RustBox attribution / notice # RustBox attribution / notice
RustBox 0.2.0 is derived from the Oxi webmail project: RustBox 0.2.1 is derived from the Oxi webmail project:
- Upstream: `c0h1b4/oxi` - Upstream: `c0h1b4/oxi`
- Base snapshot: `7f1507c486b78cf33e54beb701dd460d3bd2bd5d` - Base snapshot: `7f1507c486b78cf33e54beb701dd460d3bd2bd5d`
+4 -4
View File
@@ -1,4 +1,4 @@
# RustBox 0.2.0 # RustBox 0.2.1
RustBox is a webmail derived from `c0h1b4/oxi`. The consumed Oxi revision is recorded in `UPSTREAM_BASE`. RustBox is a webmail derived from `c0h1b4/oxi`. The consumed Oxi revision is recorded in `UPSTREAM_BASE`.
@@ -34,7 +34,7 @@ rustbox/
`build.rs` combines the CSS modules from `src/web/css/manifest.txt` and JavaScript modules from `src/web/js/manifest.txt`. Cargo embeds the generated CSS/JS bundles, HTML and SVG sprite in the RustBox executable. There is no standalone source `src/web/app.js`; `src/web/` is the single source of truth and Cargo generates the embedded bundles in `OUT_DIR`. `build.rs` combines the CSS modules from `src/web/css/manifest.txt` and JavaScript modules from `src/web/js/manifest.txt`. Cargo embeds the generated CSS/JS bundles, HTML and SVG sprite in the RustBox executable. There is no standalone source `src/web/app.js`; `src/web/` is the single source of truth and Cargo generates the embedded bundles in `OUT_DIR`.
## Storage and search in 0.2.0 ## Storage and search in 0.2.1
RustBox separates three concerns: SQL metadata, blob storage and search. The SQL engine does **not** decide whether `data_dir` may be used. A single-node PostgreSQL/MySQL deployment may still use filesystem blobs and local Tantivy. In cluster mode, local node state is rejected unless the configured backend is explicitly shared. RustBox separates three concerns: SQL metadata, blob storage and search. The SQL engine does **not** decide whether `data_dir` may be used. A single-node PostgreSQL/MySQL deployment may still use filesystem blobs and local Tantivy. In cluster mode, local node state is rejected unless the configured backend is explicitly shared.
@@ -148,7 +148,7 @@ rustbox search reindex
Ready Docker examples are in `docker/`: standalone SQLite, PostgreSQL + SILO (S3-compatible) + OpenSearch, and MySQL + NFS + OpenSearch. Ready Docker examples are in `docker/`: standalone SQLite, PostgreSQL + SILO (S3-compatible) + OpenSearch, and MySQL + NFS + OpenSearch.
All executable application SQL is centralized by dialect under `src/queries/{sqlite,mysql,pgsql}/queries.rs`. The squashed compatibility baseline remains V046; RustBox 0.2.0 adds runtime migration V047 for attachment checksums/logical blob keys and the durable search outbox. The next migration number is V048. See `migrations/README.md`. All executable application SQL is centralized by dialect under `src/queries/{sqlite,mysql,pgsql}/queries.rs`. The squashed compatibility baseline remains V046; RustBox 0.2.1 adds runtime migration V047 for attachment checksums/logical blob keys and the durable search outbox. The next migration number is V048. See `migrations/README.md`.
## Development ## Development
@@ -217,7 +217,7 @@ Initialize Git after extracting the release archive:
```bash ```bash
git init git init
git add . git add .
git commit -m "RustBox 0.2.0 release" git commit -m "RustBox 0.2.1 release"
./scripts/setup-upstream.sh ./scripts/setup-upstream.sh
``` ```
+3 -3
View File
@@ -1,4 +1,4 @@
RustBox 0.2.0 release manifest RustBox 0.2.1 release manifest
Architecture: Architecture:
- database_backend = sqlite | mysql | pgsql selects authoritative SQL only - database_backend = sqlite | mysql | pgsql selects authoritative SQL only
@@ -33,13 +33,13 @@ Docker examples:
- docker/compose-mysql-nfs-opensearch.yml (MySQL + real NFS volume + OpenSearch) - docker/compose-mysql-nfs-opensearch.yml (MySQL + real NFS volume + OpenSearch)
Operations/docs: Operations/docs:
- README.md documents 0.2.0 storage/search configuration - README.md documents 0.2.1 storage/search configuration
- docker/README.md documents Compose variants - docker/README.md documents Compose variants
- docs/STORAGE_SEARCH_OPERATIONS.md documents failure modes, shared-storage requirements and migration behavior - docs/STORAGE_SEARCH_OPERATIONS.md documents failure modes, shared-storage requirements and migration behavior
- TODO.md leaves live SILO/MinIO/AWS/Ceph and real multi-node acceptance validation explicitly pending - TODO.md leaves live SILO/MinIO/AWS/Ceph and real multi-node acceptance validation explicitly pending
Build/release metadata: Build/release metadata:
- Cargo.toml version = 0.2.0 - Cargo.toml version = 0.2.1
- rust-toolchain.toml pins Rust 1.88.0 - rust-toolchain.toml pins Rust 1.88.0
- Cargo.lock is intentionally not shipped because this environment could not regenerate it after dependency changes - Cargo.lock is intentionally not shipped because this environment could not regenerate it after dependency changes
- CI and scripts/build-release.sh generate Cargo.lock, then run locked tests/Clippy/release build - CI and scripts/build-release.sh generate Cargo.lock, then run locked tests/Clippy/release build
+7 -4
View File
@@ -1,7 +1,7 @@
RustBox 0.2.0 - final source validation RustBox 0.2.1 - final source validation
Completed in this environment: Completed in this environment:
- offline release validator: scripts/validate-release-static.py (139/139 checks passed) - offline release validator: scripts/validate-release-static.py (144/144 checks passed)
- all maintained TOML examples parse successfully - all maintained TOML examples parse successfully
- Docker Compose topology/config references are validated by the offline release validator - Docker Compose topology/config references are validated by the offline release validator
- JSON files parse successfully - JSON files parse successfully
@@ -11,9 +11,12 @@ Completed in this environment:
- Rust source delimiter sanity check passes after removing partial-patch leftovers - Rust source delimiter sanity check passes after removing partial-patch leftovers
- git diff --check passes on the final source tree - git diff --check passes on the final source tree
- no Cargo.lock, *.orig, *.rej, temporary FIX/APPLY scripts, or patch leftovers are shipped - no Cargo.lock, *.orig, *.rej, temporary FIX/APPLY scripts, or patch leftovers are shipped
- regression source checks cover every Clippy failure reported during the 0.2.0 release attempt - regression source checks retain the 0.2.0 release gate and add 0.2.1 policy/backpressure checks
0.2.0 fixes included: 0.2.1 fixes included:
- OpenPGP policy visibility follows pgp_enabled and disabled OpenPGP is hidden from the UI
- OpenSearch 429/flood-stage failures trigger worker backoff instead of per-event warning storms
- Docker development defaults session cookies for direct HTTP and documents the production HTTPS requirement
- cluster_mode controls the implicit data_dir default; PostgreSQL/MySQL remain valid with local data_dir backends in single-node deployments - cluster_mode controls the implicit data_dir default; PostgreSQL/MySQL remain valid with local data_dir backends in single-node deployments
- storage/search configuration comes only from [storage] and [search] - storage/search configuration comes only from [storage] and [search]
- removed pre-0.2 top-level storage/search switches and the tantivy backend alias - removed pre-0.2 top-level storage/search switches and the tantivy backend alias
+1 -1
View File
@@ -1,4 +1,4 @@
# RustBox 0.2.0 # RustBox 0.2.1
host = "127.0.0.1" host = "127.0.0.1"
port = 3001 port = 3001
environment = "production" environment = "production"
+31 -1
View File
@@ -112,7 +112,7 @@ The bundled SILO topology is intended for local/integration validation of the S3
### `compose-mysql-nfs-opensearch.yml` ### `compose-mysql-nfs-opensearch.yml`
MySQL, shared POSIX/NFS blob storage and OpenSearch. The NFS volume driver is deliberate: a normal local Docker named volume is not shared between Docker hosts and therefore is not a valid blob backend for a multi-host RustBox deployment. MySQL, shared POSIX/NFS blob storage and OpenSearch. The NFS volume driver is deliberate: a normal local Docker named volume is not shared between Docker hosts and therefore is not a valid blob backend for a multi-host RustBox deployment. These examples intentionally run one RustBox application container; the shared backing services demonstrate storage/database topology, not multi-app load balancing.
## Production notes ## Production notes
@@ -135,3 +135,33 @@ ENVIRONMENT=production docker compose -p rustbox -f compose-postgres-s3-opensear
In `production`, RustBox intentionally marks authentication cookies as `Secure`. Opening the application directly over plain HTTP in that mode causes the browser not to send the session cookie and protected requests return `Invalid or expired session`. In `production`, RustBox intentionally marks authentication cookies as `Secure`. Opening the application directly over plain HTTP in that mode causes the browser not to send the session cookie and protected requests return `Invalid or expired session`.
After changing from production/HTTP to development/HTTP, log in again. If the browser still has stale RustBox state, clear cookies/site data for the RustBox origin once and reload. After changing from production/HTTP to development/HTTP, log in again. If the browser still has stale RustBox state, clear cookies/site data for the RustBox origin once and reload.
## OpenSearch disk watermark / 429 errors
The `opensearch-data` named volume still consumes space on the filesystem used by Docker. Running OpenSearch in a container does not give it an independent disk. If the Docker host filesystem reaches OpenSearch's flood-stage threshold, OpenSearch protects indices with `read_only_allow_delete` and RustBox indexing receives HTTP 429 responses.
Check the actual filesystem seen by the container:
```bash
docker compose -p rustbox -f compose-postgres-s3-opensearch.yml exec opensearch \
df -h /usr/share/opensearch/data
```
Also inspect Docker usage on the host:
```bash
df -h
docker system df
docker volume inspect rustbox_opensearch-data
```
Free disk space first. Do not permanently lower the OpenSearch flood-stage safety threshold. If the index remains read-only after disk usage is back below the high watermark, clear the block explicitly:
```bash
docker compose -p rustbox -f compose-postgres-s3-opensearch.yml exec opensearch \
curl -fsS -X PUT http://localhost:9200/rustbox-messages-v1/_settings \
-H 'Content-Type: application/json' \
-d '{"index.blocks.read_only_allow_delete":null}'
```
RustBox 0.2.1 backs off its OpenSearch outbox worker after these 429/flood-stage responses, so a full disk no longer produces one warning per queued event.
+1 -1
View File
@@ -76,7 +76,7 @@ services:
SMTP_HOST: ${SMTP_HOST:-mail.example.com} SMTP_HOST: ${SMTP_HOST:-mail.example.com}
SMTP_PORT: ${SMTP_PORT:-587} SMTP_PORT: ${SMTP_PORT:-587}
TLS_ENABLED: ${TLS_ENABLED:-true} TLS_ENABLED: ${TLS_ENABLED:-true}
RUST_LOG: ${RUST_LOG:-info} RUST_LOG: ${RUST_LOG:-info,sqlx::postgres::notice=warn}
DATABASE_URL: postgres://rustbox:${POSTGRES_PASSWORD:-rustbox-dev-password}@postgres:5432/rustbox DATABASE_URL: postgres://rustbox:${POSTGRES_PASSWORD:-rustbox-dev-password}@postgres:5432/rustbox
SESSION_ENCRYPTION_KEY: ${SESSION_ENCRYPTION_KEY:-MDEyMzQ1Njc4OWFiY2RlZjAxMjM0NTY3ODlhYmNkZWY=} SESSION_ENCRYPTION_KEY: ${SESSION_ENCRYPTION_KEY:-MDEyMzQ1Njc4OWFiY2RlZjAxMjM0NTY3ODlhYmNkZWY=}
AWS_ACCESS_KEY_ID: ${S3_ACCESS_KEY:-rustbox} AWS_ACCESS_KEY_ID: ${S3_ACCESS_KEY:-rustbox}
+3 -3
View File
@@ -1,8 +1,8 @@
# RustBox 0.2.0 - implementation status # RustBox 0.2.1 - implementation status
This file tracks the `0.2.0` source snapshot. Storage/search scope and outstanding acceptance work are tracked in `TODO.md`. A feature is only listed as implemented when working code exists in this source tree; partial items are called out explicitly. This file tracks the `0.2.1` source snapshot. Storage/search scope and outstanding acceptance work are tracked in `TODO.md`. A feature is only listed as implemented when working code exists in this source tree; partial items are called out explicitly.
## 0.2.0 storage/search architecture ## 0.2.1 storage/search architecture
- SQL backend selection (`sqlite`/`mysql`/`pgsql`) is independent from blob storage and search backend selection. - SQL backend selection (`sqlite`/`mysql`/`pgsql`) is independent from blob storage and search backend selection.
- Cluster mode clears an implicit/default `data_dir`; PostgreSQL/MySQL single-node deployments may still intentionally use `data_dir`-backed filesystem storage and local Tantivy. - Cluster mode clears an implicit/default `data_dir`; PostgreSQL/MySQL single-node deployments may still intentionally use `data_dir`-backed filesystem storage and local Tantivy.
+1 -1
View File
@@ -27,7 +27,7 @@ RUSTBOX_CONFIG=./config.dev.toml ./target/release/rustbox --check-config
Browser JavaScript source lives in functional modules under `src/web/js/`, ordered by `src/web/js/manifest.txt`. `build.rs` concatenates those modules into the embedded `rustbox-ui.js` asset during every Cargo build. The repository has no duplicate `ui/` snapshot; `src/web/` is the only browser UI source. Browser JavaScript source lives in functional modules under `src/web/js/`, ordered by `src/web/js/manifest.txt`. `build.rs` concatenates those modules into the embedded `rustbox-ui.js` asset during every Cargo build. The repository has no duplicate `ui/` snapshot; `src/web/` is the only browser UI source.
## Updating an existing checkout to 0.2.0 ## Updating an existing checkout to 0.2.1
Do not overlay the release with `cp *` or another command that skips dotfiles. That can retain stale `.github`, `.env.example`, or `Cargo.lock` files. Do not overlay the release with `cp *` or another command that skips dotfiles. That can retain stale `.github`, `.env.example`, or `Cargo.lock` files.
+2 -2
View File
@@ -1,6 +1,6 @@
# RustBox 0.2.0 storage/search operations # RustBox 0.2.1 storage/search operations
RustBox 0.2.0 treats SQL, blob storage and search as independent capabilities. The database engine does not decide whether `data_dir` is allowed. RustBox 0.2.1 treats SQL, blob storage and search as independent capabilities. The database engine does not decide whether `data_dir` is allowed.
## Deployment matrix ## Deployment matrix
+3 -3
View File
@@ -1,6 +1,6 @@
# Database schemas and migrations # Database schemas and migrations
RustBox 0.2.0 keeps SQL, blob storage and search as independent concerns: RustBox 0.2.1 keeps SQL, blob storage and search as independent concerns:
```text ```text
database_backend = sqlite | mysql | pgsql database_backend = sqlite | mysql | pgsql
@@ -26,7 +26,7 @@ migrations/
└── V047__blob_keys_and_search_outbox.sql └── V047__blob_keys_and_search_outbox.sql
``` ```
Each `schema.sql` is the current complete schema for fresh installations. The historical compatibility baseline remains **V046**. RustBox 0.2.0 adds **V047**, which adds attachment SHA-256 metadata and the durable search outbox used by external indexing. Each `schema.sql` is the current complete schema for fresh installations. The historical compatibility baseline remains **V046**. RustBox 0.2.1 adds **V047**, which adds attachment SHA-256 metadata and the durable search outbox used by external indexing.
Account data is isolated inside the selected physical database with a deterministic, collision-checked table prefix derived from the account hash. The `{{p}}` marker in tenant SQL is replaced by RustBox at provisioning/migration time. Sessions are global and therefore are not prefixed. Account data is isolated inside the selected physical database with a deterministic, collision-checked table prefix derived from the account hash. The `{{p}}` marker in tenant SQL is replaced by RustBox at provisioning/migration time. Sessions are global and therefore are not prefixed.
@@ -34,7 +34,7 @@ Account data is isolated inside the selected physical database with a determinis
Every schema-changing release must add the same logical migration for all supported SQL engines and register it in `src/db/pool.rs`. Do not update only `schema.sql`: fresh installs use the baseline while existing installations use the numbered migration path. Every schema-changing release must add the same logical migration for all supported SQL engines and register it in `src/db/pool.rs`. Do not update only `schema.sql`: fresh installs use the baseline while existing installations use the numbered migration path.
The next migration number after RustBox 0.2.0 is **V048**. The next migration number after RustBox 0.2.1 is **V048**.
Redis and Redis Sentinel do not use SQL migrations. Redis and Redis Sentinel do not use SQL migrations.
+1 -1
View File
@@ -1,4 +1,4 @@
# RustBox 0.2.0 - packaged production configuration (DEB/RPM) # RustBox 0.2.1 - packaged production configuration (DEB/RPM)
# #
# This file contains all supported RustBox configuration options. # This file contains all supported RustBox configuration options.
# Commented options are optional and do not change the packaged defaults. # Commented options are optional and do not change the packaged defaults.
+11 -6
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Static/offline release checks for RustBox 0.2.0. """Static/offline release checks for RustBox 0.2.1.
These checks intentionally avoid pretending to replace `cargo test`: they validate These checks intentionally avoid pretending to replace `cargo test`: they validate
configuration, SQL migrations, Docker examples, query-catalog parity and release configuration, SQL migrations, Docker examples, query-catalog parity and release
@@ -49,7 +49,7 @@ def extract_section(text: str, begin: str, end: str) -> str:
# Release/toolchain metadata. # Release/toolchain metadata.
cargo = tomllib.loads(read("Cargo.toml")) cargo = tomllib.loads(read("Cargo.toml"))
require(cargo["package"]["version"] == "0.2.0", "Cargo package version is 0.2.0") require(cargo["package"]["version"] == "0.2.1", "Cargo package version is 0.2.1")
require(cargo["package"]["rust-version"] == "1.88", "Cargo rust-version is 1.88") require(cargo["package"]["rust-version"] == "1.88", "Cargo rust-version is 1.88")
toolchain = tomllib.loads(read("rust-toolchain.toml")) toolchain = tomllib.loads(read("rust-toolchain.toml"))
require(toolchain["toolchain"]["channel"] == "1.88.0", "rust-toolchain pins Rust 1.88.0") require(toolchain["toolchain"]["channel"] == "1.88.0", "rust-toolchain pins Rust 1.88.0")
@@ -58,13 +58,18 @@ require("cargo generate-lockfile" in read("scripts/build-release.sh"), "release
require("cargo generate-lockfile" in read(".github/workflows/ci.yml"), "CI regenerates Cargo.lock before --locked checks") require("cargo generate-lockfile" in read(".github/workflows/ci.yml"), "CI regenerates Cargo.lock before --locked checks")
require("validate-release-static.py" in read("scripts/build-release.sh"), "release script runs offline/static release gate") require("validate-release-static.py" in read("scripts/build-release.sh"), "release script runs offline/static release gate")
require("validate-release-static.py" in read(".github/workflows/ci.yml"), "CI runs offline/static release gate") require("validate-release-static.py" in read(".github/workflows/ci.yml"), "CI runs offline/static release gate")
require(read("README.md").startswith("# RustBox 0.2.0"), "README identifies RustBox 0.2.0") require(read("README.md").startswith("# RustBox 0.2.1"), "README identifies RustBox 0.2.1")
require(read("RELEASE-MANIFEST.txt").startswith("RustBox 0.2.0"), "release manifest identifies RustBox 0.2.0") require(read("RELEASE-MANIFEST.txt").startswith("RustBox 0.2.1"), "release manifest identifies RustBox 0.2.1")
require(read("RELEASE-VALIDATION.txt").startswith("RustBox 0.2.0"), "release validation identifies RustBox 0.2.0") require(read("RELEASE-VALIDATION.txt").startswith("RustBox 0.2.1"), "release validation identifies RustBox 0.2.1")
require(read("docs/IMPLEMENTATION_STATUS.md").startswith("# RustBox 0.2.0"), "implementation status identifies RustBox 0.2.0") require(read("docs/IMPLEMENTATION_STATUS.md").startswith("# RustBox 0.2.1"), "implementation status identifies RustBox 0.2.1")
require((ROOT / "docs/STORAGE_SEARCH_OPERATIONS.md").is_file(), "storage/search operations guide exists") require((ROOT / "docs/STORAGE_SEARCH_OPERATIONS.md").is_file(), "storage/search operations guide exists")
require('imap_host = ""' not in read("config.dev.toml"), "development config does not override IMAP host with an invalid empty value") require('imap_host = ""' not in read("config.dev.toml"), "development config does not override IMAP host with an invalid empty value")
require(not re.search(r"(?m)^IMAP_HOST=\s*$", read(".env.example")), "environment example does not set an invalid empty IMAP host") require(not re.search(r"(?m)^IMAP_HOST=\s*$", read(".env.example")), "environment example does not set an invalid empty IMAP host")
require("pgp_enabled: bool" in read("src/routes/auth.rs"), "auth policy exposes OpenPGP availability")
require("setOpenPgpAvailability" in read("src/web/js/72-openpgp-identities.js"), "UI hides administratively disabled OpenPGP")
require("is_opensearch_backpressure_error" in read("src/search/engine.rs"), "OpenSearch worker classifies backpressure failures")
require("worker pass failed; backing off" in read("src/search/engine.rs"), "OpenSearch worker backs off after failures")
require("ENVIRONMENT: ${ENVIRONMENT:-development}" in read("docker/compose-postgres-s3-opensearch.yml"), "PostgreSQL Compose defaults direct HTTP to development cookie mode")
# Parse all maintained TOML examples. # Parse all maintained TOML examples.
toml_files = [ toml_files = [
+39
View File
@@ -164,6 +164,45 @@ pub fn mark_failure(
Ok(changed > 0) Ok(changed > 0)
} }
pub fn defer_transient_failure(
conn: &Connection,
event: &OutboxEvent,
error: &str,
delay_secs: u64,
) -> Result<bool, String> {
let available_at = now_epoch().saturating_add(delay_secs.max(1) as i64);
let now = now_epoch();
let truncated: String = error.chars().take(2048).collect();
let changed = match crate::db::pool::configured_backend() {
crate::db::pool::DatabaseBackend::Mysql => conn.execute(
crate::dbq!(db_search_outbox::Q_004),
params![
event.attempts,
available_at,
false,
&truncated,
now,
&event.id,
&event.generation
],
),
_ => conn.execute(
crate::dbq!(db_search_outbox::Q_004),
params![
&event.id,
&event.generation,
event.attempts,
available_at,
false,
&truncated,
now
],
),
}
.map_err(|e| format!("failed to defer transient search event: {e}"))?;
Ok(changed > 0)
}
pub fn clear_before(conn: &Connection, updated_before: i64) -> Result<usize, String> { pub fn clear_before(conn: &Connection, updated_before: i64) -> Result<usize, String> {
conn.execute( conn.execute(
crate::dbq!(db_search_outbox::Q_007), crate::dbq!(db_search_outbox::Q_007),
+2
View File
@@ -20,6 +20,7 @@ pub struct AuthPolicyResponse {
remember_me_enabled: bool, remember_me_enabled: bool,
remember_me_days: u64, remember_me_days: u64,
session_timeout_hours: u64, session_timeout_hours: u64,
pgp_enabled: bool,
} }
pub async fn auth_policy(Extension(config): Extension<Arc<AppConfig>>) -> Json<AuthPolicyResponse> { pub async fn auth_policy(Extension(config): Extension<Arc<AppConfig>>) -> Json<AuthPolicyResponse> {
@@ -27,6 +28,7 @@ pub async fn auth_policy(Extension(config): Extension<Arc<AppConfig>>) -> Json<A
remember_me_enabled: config.remember_me_enabled, remember_me_enabled: config.remember_me_enabled,
remember_me_days: config.remember_me_days, remember_me_days: config.remember_me_days,
session_timeout_hours: config.session_timeout_hours, session_timeout_hours: config.session_timeout_hours,
pgp_enabled: config.pgp_enabled,
}) })
} }
+62 -8
View File
@@ -275,16 +275,38 @@ impl SearchEngine {
let batch_size = *worker_batch_size; let batch_size = *worker_batch_size;
let max_attempts = *worker_max_attempts; let max_attempts = *worker_max_attempts;
Some(tokio::spawn(async move { Some(tokio::spawn(async move {
let mut interval = tokio::time::interval(Duration::from_secs(interval_secs)); let mut failure_streak = 0u32;
loop { loop {
interval.tick().await; let delay_secs = match process_search_outbox(
match process_search_outbox(&backend, &data_dir, batch_size, max_attempts).await { &backend,
Ok(processed) if processed > 0 => { &data_dir,
tracing::debug!(processed, "processed OpenSearch outbox events"); batch_size,
max_attempts,
)
.await
{
Ok(processed) => {
failure_streak = 0;
if processed > 0 {
tracing::debug!(processed, "processed OpenSearch outbox events");
}
interval_secs
} }
Ok(_) => {} Err(error) => {
Err(error) => tracing::warn!(%error, "OpenSearch outbox worker pass failed"), failure_streak = failure_streak.saturating_add(1);
} let multiplier = 2u64.saturating_pow(failure_streak.min(5));
let retry_after_secs = interval_secs
.saturating_mul(multiplier)
.clamp(interval_secs, 60);
tracing::warn!(
%error,
retry_after_secs,
"OpenSearch outbox worker pass failed; backing off"
);
retry_after_secs
}
};
tokio::time::sleep(Duration::from_secs(delay_secs)).await;
} }
})) }))
} }
@@ -1033,6 +1055,14 @@ impl OpenSearchUserIndex {
} }
} }
fn is_opensearch_backpressure_error(error: &str) -> bool {
let lower = error.to_ascii_lowercase();
lower.contains("429 too many requests")
|| lower.contains("status 429")
|| lower.contains("flood-stage watermark")
|| lower.contains("read-only-allow-delete")
}
async fn process_search_outbox( async fn process_search_outbox(
backend: &OpenSearchBackend, backend: &OpenSearchBackend,
data_dir: &str, data_dir: &str,
@@ -1082,6 +1112,17 @@ async fn process_search_outbox(
processed += 1; processed += 1;
} }
Err(error) => { Err(error) => {
if is_opensearch_backpressure_error(&error) {
crate::db::search_outbox::defer_transient_failure(
&conn,
&event,
&error,
30,
)?;
return Err(format!(
"OpenSearch temporarily rejected indexing for tenant {user_hash}: {error}"
));
}
crate::db::search_outbox::mark_failure(&conn, &event, &error, max_attempts)?; crate::db::search_outbox::mark_failure(&conn, &event, &error, max_attempts)?;
tracing::warn!(%error, event_id = %event.id, %user_hash, "OpenSearch outbox event failed"); tracing::warn!(%error, event_id = %event.id, %user_hash, "OpenSearch outbox event failed");
} }
@@ -1105,6 +1146,19 @@ mod tests {
(tmp, user_index) (tmp, user_index)
} }
#[test]
fn opensearch_backpressure_classifier_detects_flood_stage_and_429() {
assert!(is_opensearch_backpressure_error(
"OpenSearch document upsert failed (429 Too Many Requests): flood-stage watermark"
));
assert!(is_opensearch_backpressure_error(
"index blocked by read-only-allow-delete block"
));
assert!(!is_opensearch_backpressure_error(
"OpenSearch document upsert failed (500 Internal Server Error)"
));
}
#[test] #[test]
fn opensearch_event_lease_covers_timeout_and_failover_budget() { fn opensearch_event_lease_covers_timeout_and_failover_budget() {
let backend = OpenSearchBackend { let backend = OpenSearchBackend {
+2 -2
View File
@@ -943,7 +943,7 @@
</div><button id="identity-new" class="button primary" type="button">New identity</button> </div><button id="identity-new" class="button primary" type="button">New identity</button>
</div> </div>
<div class="identity-overview"> <div class="identity-overview">
<div class="identity-overview-hint"><strong>Configured identities</strong><span class="muted">Select an <div class="identity-overview-hint"><strong>Configured identities</strong><span id="identity-overview-help" class="muted">Select an
identity to edit its sender data, text/HTML footer, image and OpenPGP defaults.</span></div> identity to edit its sender data, text/HTML footer, image and OpenPGP defaults.</span></div>
<div id="identity-list" class="identity-list"></div> <div id="identity-list" class="identity-list"></div>
</div> </div>
@@ -1249,7 +1249,7 @@
</div> </div>
</section> </section>
<section class="identity-modal-section identity-modal-options"> <section class="identity-modal-section identity-modal-options">
<div class="identity-modal-heading"><strong>Defaults</strong><span class="muted">Choose where this signature <div class="identity-modal-heading"><strong>Defaults</strong><span id="identity-defaults-help" class="muted">Choose where this signature
is inserted and, when OpenPGP is configured, its default behavior.</span></div> is inserted and, when OpenPGP is configured, its default behavior.</span></div>
<div class="identity-toggle-grid"> <div class="identity-toggle-grid">
<label class="checkbox-row"><input id="identity-signature-new" type="checkbox"><span>Signature on new <label class="checkbox-row"><input id="identity-signature-new" type="checkbox"><span>Signature on new
+1 -1
View File
@@ -33,7 +33,7 @@
pgpKeys: [], pgpKeys: [],
pgpPrefs: { wkdEnabled: true, hkpEnabled: false, requireTrustedEncryption: false }, pgpPrefs: { wkdEnabled: true, hkpEnabled: false, requireTrustedEncryption: false },
pgpRecipientStatus: [], pgpRecipientStatus: [],
authPolicy: { rememberMeEnabled: true, rememberMeDays: 30, sessionTimeoutHours: 24 }, authPolicy: { rememberMeEnabled: true, rememberMeDays: 30, sessionTimeoutHours: 24, pgpEnabled: true },
contacts: [], contacts: [],
trustedSenders: new Set(), trustedSenders: new Set(),
contactGroups: [], contactGroups: [],
+2 -1
View File
@@ -462,8 +462,9 @@ async function loadAuthPolicy() {
try { try {
state.authPolicy = await request("/auth/policy", { method: "GET" }); state.authPolicy = await request("/auth/policy", { method: "GET" });
} catch (_) { } catch (_) {
state.authPolicy = { rememberMeEnabled: true, rememberMeDays: 30, sessionTimeoutHours: 24 }; state.authPolicy = { rememberMeEnabled: true, rememberMeDays: 30, sessionTimeoutHours: 24, pgpEnabled: true };
} }
setOpenPgpAvailability(state.authPolicy.pgpEnabled !== false);
const enabled = state.authPolicy.rememberMeEnabled !== false; const enabled = state.authPolicy.rememberMeEnabled !== false;
["#login-remember-row", "#add-account-remember-row"].forEach((selector) => $(selector)?.classList.toggle("hidden", !enabled)); ["#login-remember-row", "#add-account-remember-row"].forEach((selector) => $(selector)?.classList.toggle("hidden", !enabled));
["#login-remember", "#add-account-remember"].forEach((selector) => { const input = $(selector); if (input && !enabled) { input.checked = false; input.disabled = true; } }); ["#login-remember", "#add-account-remember"].forEach((selector) => { const input = $(selector); if (input && !enabled) { input.checked = false; input.disabled = true; } });
+1 -1
View File
@@ -470,7 +470,7 @@ function renderPgpReaderState(detail) {
const title = $("#pgp-reader-title"); const title = $("#pgp-reader-title");
const find = $("#pgp-reader-discover"); const find = $("#pgp-reader-discover");
const kind = pgpMessageKind(detail); const kind = pgpMessageKind(detail);
const configured = state.pgpKeys.length > 0; const configured = state.authPolicy.pgpEnabled !== false && state.pgpKeys.length > 0;
bar.classList.toggle("hidden", !kind.pgp || !configured); bar.classList.toggle("hidden", !kind.pgp || !configured);
find.classList.add("hidden"); find.classList.add("hidden");
if (!kind.pgp || !configured) return; if (!kind.pgp || !configured) return;
+41 -2
View File
@@ -1,5 +1,41 @@
function setOpenPgpAvailability(enabled) {
const available = Boolean(enabled);
state.authPolicy.pgpEnabled = available;
document.querySelector('.settings-tab[data-tab="openpgp"]')?.classList.toggle("hidden", !available);
const settingsPanel = document.querySelector('.settings-panel[data-panel="openpgp"]');
if (settingsPanel && !available) settingsPanel.classList.add("hidden");
$("#compose-pgp-option")?.classList.toggle("hidden", !available || state.pgpKeys.length === 0);
$("#compose-pgp-panel")?.classList.toggle("hidden", !available || state.pgpKeys.length === 0);
if (!available) $("#pgp-reader-bar")?.classList.add("hidden");
$("#identity-pgp-key-row")?.classList.toggle("hidden", !available);
$$(".identity-pgp-default").forEach((row) => row.classList.toggle("hidden", !available));
const overview = $("#identity-overview-help");
if (overview) overview.textContent = available
? "Select an identity to edit its sender data, text/HTML footer, image and OpenPGP defaults."
: "Select an identity to edit its sender data, text/HTML footer and image.";
const defaultsHelp = $("#identity-defaults-help");
if (defaultsHelp) defaultsHelp.textContent = available
? "Choose where this signature is inserted and, when OpenPGP is configured, its default behavior."
: "Choose where this signature is inserted.";
if (!available) {
state.pgpKeys = [];
state.pgpRecipientStatus = [];
const mode = $("#compose-pgp-mode");
if (mode) mode.value = "none";
const panel = $("#compose-pgp-panel");
if (panel) panel.open = false;
}
}
async function loadOpenPgp() { async function loadOpenPgp() {
if (state.authPolicy.pgpEnabled === false) {
setOpenPgpAvailability(false);
updateComposePgpUi();
if (state.currentDetail) renderPgpReaderState(state.currentDetail);
return;
}
try { try {
const [keys, prefs] = await Promise.all([ const [keys, prefs] = await Promise.all([
request("/pgp/keys", { method: "GET" }), request("/pgp/keys", { method: "GET" }),
@@ -13,9 +49,11 @@ async function loadOpenPgp() {
if (state.currentDetail) renderPgpReaderState(state.currentDetail); if (state.currentDetail) renderPgpReaderState(state.currentDetail);
} catch (error) { } catch (error) {
state.pgpKeys = []; state.pgpKeys = [];
const disabled = String(error?.message || "").toLowerCase().includes("openpgp is disabled");
if (disabled) setOpenPgpAvailability(false);
updateComposePgpUi(); updateComposePgpUi();
if (state.currentDetail) renderPgpReaderState(state.currentDetail); if (state.currentDetail) renderPgpReaderState(state.currentDetail);
toast(error.message, "error"); if (!disabled) toast(error.message, "error");
} }
} }
@@ -177,7 +215,8 @@ function updateComposePgpUi(applyDefaults = false) {
const panel = $("#compose-pgp-panel"); const panel = $("#compose-pgp-panel");
const option = $("#compose-pgp-option"); const option = $("#compose-pgp-option");
if (!modeSelect || !panel) return; if (!modeSelect || !panel) return;
const configured = state.pgpKeys.length > 0; const available = state.authPolicy.pgpEnabled !== false;
const configured = available && state.pgpKeys.length > 0;
option?.classList.toggle("hidden", !configured); option?.classList.toggle("hidden", !configured);
panel.classList.toggle("hidden", !configured); panel.classList.toggle("hidden", !configured);
if (!configured) { if (!configured) {
+4 -2
View File
@@ -45,11 +45,12 @@ async function loadQuota() {
} }
function openSettings(tab = "appearance") { function openSettings(tab = "appearance") {
if (tab === "openpgp" && state.authPolicy.pgpEnabled === false) tab = "appearance";
selectSettingsTab(tab); selectSettingsTab(tab);
$("#settings-dialog").showModal(); $("#settings-dialog").showModal();
if (tab === "sieve") loadSieve(); if (tab === "sieve") loadSieve();
if (tab === "identities") { loadIdentities(); loadOpenPgp(); } if (tab === "identities") { loadIdentities(); if (state.authPolicy.pgpEnabled !== false) loadOpenPgp(); }
if (tab === "openpgp") loadOpenPgp(); if (tab === "openpgp" && state.authPolicy.pgpEnabled !== false) loadOpenPgp();
if (tab === "notifications") loadNotificationPreferences(); if (tab === "notifications") loadNotificationPreferences();
if (tab === "mailbox") loadMailPreferences(); if (tab === "mailbox") loadMailPreferences();
if (tab === "diagnostics") loadDiagnostics(); if (tab === "diagnostics") loadDiagnostics();
@@ -58,6 +59,7 @@ function openSettings(tab = "appearance") {
} }
function selectSettingsTab(name) { function selectSettingsTab(name) {
if (name === "openpgp" && state.authPolicy.pgpEnabled === false) name = "appearance";
$$(".settings-tab").forEach((button) => button.classList.toggle("active", button.dataset.tab === name)); $$(".settings-tab").forEach((button) => button.classList.toggle("active", button.dataset.tab === name));
$$(".settings-panel").forEach((panel) => panel.classList.toggle("hidden", panel.dataset.panel !== name)); $$(".settings-panel").forEach((panel) => panel.classList.toggle("hidden", panel.dataset.panel !== name));
} }
+2 -2
View File
@@ -292,7 +292,7 @@ function bindEvents() {
$("#pgp-import-file").addEventListener("click", () => $("#pgp-import-file-input").click()); $("#pgp-import-file").addEventListener("click", () => $("#pgp-import-file-input").click());
$("#pgp-import-file-input").addEventListener("change", async (event) => { const file = event.target.files?.[0]; if (file) await importPgpFile(file); event.target.value = ""; }); $("#pgp-import-file-input").addEventListener("change", async (event) => { const file = event.target.files?.[0]; if (file) await importPgpFile(file); event.target.value = ""; });
$("#pgp-discover").addEventListener("click", () => discoverPgpEmail()); $("#pgp-discover").addEventListener("click", () => discoverPgpEmail());
$$(".settings-tab").forEach((button) => button.addEventListener("click", () => { selectSettingsTab(button.dataset.tab); if (button.dataset.tab === "sieve") loadSieve(); if (button.dataset.tab === "identities") { loadIdentities(); loadOpenPgp(); } if (button.dataset.tab === "openpgp") loadOpenPgp(); if (button.dataset.tab === "notifications") loadNotificationPreferences(); if (button.dataset.tab === "mailbox") loadMailPreferences(); if (button.dataset.tab === "composing") loadResponseTemplates(); if (button.dataset.tab === "diagnostics") loadDiagnostics(); if (button.dataset.tab === "account") { renderAccountSettings(); loadActiveSessions(); } if (button.dataset.tab === "folders") { loadSpecialFolders(); loadQuota(); } })); $$(".settings-tab").forEach((button) => button.addEventListener("click", () => { selectSettingsTab(button.dataset.tab); if (button.dataset.tab === "sieve") loadSieve(); if (button.dataset.tab === "identities") { loadIdentities(); if (state.authPolicy.pgpEnabled !== false) loadOpenPgp(); } if (button.dataset.tab === "openpgp" && state.authPolicy.pgpEnabled !== false) loadOpenPgp(); if (button.dataset.tab === "notifications") loadNotificationPreferences(); if (button.dataset.tab === "mailbox") loadMailPreferences(); if (button.dataset.tab === "composing") loadResponseTemplates(); if (button.dataset.tab === "diagnostics") loadDiagnostics(); if (button.dataset.tab === "account") { renderAccountSettings(); loadActiveSessions(); } if (button.dataset.tab === "folders") { loadSpecialFolders(); loadQuota(); } }));
$("#pref-theme").addEventListener("change", (event) => savePreference("theme", event.target.value)); $("#pref-theme").addEventListener("change", (event) => savePreference("theme", event.target.value));
$("#pref-interface-skin").addEventListener("change", (event) => savePreference("interfaceSkin", event.target.value)); $("#pref-interface-skin").addEventListener("change", (event) => savePreference("interfaceSkin", event.target.value));
$("#pref-preset-scale-unlocked").addEventListener("change", (event) => savePreference("presetScaleUnlocked", event.target.checked)); $("#pref-preset-scale-unlocked").addEventListener("change", (event) => savePreference("presetScaleUnlocked", event.target.checked));
@@ -515,7 +515,7 @@ async function boot() {
await loadAuthPolicy(); await loadAuthPolicy();
const authenticated = await loadAccounts(); const authenticated = await loadAccounts();
if (authenticated) { if (authenticated) {
await loadPreferences(); await loadMailPreferences(); await loadOpenPgp(); await loadIdentities(); await loadResponseTemplates(); await loadFolders(true); await loadSpecialFolders(); await loadQuota(); await loadTags(); await loadNotificationPreferences(); await loadTrustedSenders(); startCalendarReminderWatcher(); connectRealtime(); await loadPreferences(); await loadMailPreferences(); if (state.authPolicy.pgpEnabled !== false) await loadOpenPgp(); else setOpenPgpAvailability(false); await loadIdentities(); await loadResponseTemplates(); await loadFolders(true); await loadSpecialFolders(); await loadQuota(); await loadTags(); await loadNotificationPreferences(); await loadTrustedSenders(); startCalendarReminderWatcher(); connectRealtime();
const params = new URLSearchParams(location.search); const params = new URLSearchParams(location.search);
const folder = params.get("mail_folder"); const folder = params.get("mail_folder");
const uid = Number(params.get("mail_uid")); const uid = Number(params.get("mail_uid"));