v0.2.2
RustBox CI / rustbox (push) Canceled after 0s

This commit is contained in:
Mateusz Gruszczyński committed 2026-10-31 16:27:41 +01:00
1 parent 95c0587c1e
commit 5b22add1f3
21 files changed
+292 -88

No files matched your search

+20
View File
@@ -1,5 +1,25 @@
# RustBox changelog
### 0.2.2
- Fixed the SQLite isolation regression test to initialize `AppConfig` with a struct literal plus `..Default::default()`, satisfying `cargo clippy --all-targets -- -D warnings` without suppressing `field_reassign_with_default`.
- Fixed Rust test isolation for SQLite: test-only `db::pool::configure()` still initializes the requested database schema but no longer pins the first temporary SQLite path in the process-wide `OnceLock`, preventing later tests from reusing a deleted `TempDir` and collapsing into HTTP 500/database-open failures.
- Added a regression test that configures SQLite in one temporary directory, removes it, then opens an account database in a second temporary directory.
- Strengthened the offline release contract to assert that `scripts/build-release.sh` requires the full locked test suite, warning-free Clippy, and a locked release build before a release is accepted.
- Fixed the PostgreSQL startup schema compile failure (`implementation of sqlx::Executor is not general enough`) by executing the multi-statement DDL batch through `sqlx::Executor::execute()` on the transaction connection instead of `sqlx::raw_sql(...).execute(&mut PgConnection)`.
- Preserved transactional startup schema initialization and the local `client_min_messages=WARNING` scope used to suppress expected idempotent PostgreSQL NOTICE output.
- Added a static regression gate that forbids the lifetime-problematic `raw_sql(&schema).execute(&mut *tx)` PostgreSQL startup path.
- No schema migration is required; runtime schema remains V047 and V048 remains the next migration number.
- Fixed PostgreSQL header-sync invalidation used by message mutations (including marking/moving messages as spam): the UPSERT now qualifies the existing tenant `user_meta.value`, avoiding `column reference "value" is ambiguous` and HTTP 500 responses.
- Added a regression check for tenant SQL rewriting of the qualified `user_meta.value` reference and a static audit covering all PostgreSQL `ON CONFLICT ... DO UPDATE` target-column reads.
- Confirmed the equivalent SQLite UPSERT is not ambiguous and MySQL uses `ON DUPLICATE KEY UPDATE`, so no matching backend fix is required.
- No schema migration is required; the runtime schema remains V047 and V048 remains the next migration number.
- Centralized SQL schema initialization in `db::pool::configure()` so the database session store no longer executes the global schema a second time.
- Fixed the duplicate startup DDL for all SQL backends: PostgreSQL no longer repeats `relation ... already exists, skipping` notices, while MySQL and SQLite also avoid the redundant second schema pass.
- Scoped PostgreSQL startup schema initialization to a transaction with `client_min_messages=WARNING`, suppressing expected idempotent `CREATE ... IF NOT EXISTS` notices without hiding application/database warnings globally.
- Removed the Docker-specific SQLx PostgreSQL notice filter workaround; normal `RUST_LOG=info` is sufficient.
- No schema migration is required; the runtime schema remains V047 and V048 remains the next migration number.
- Release number kept at 0.2.2 for this cumulative patch set.
### 0.2.1
- Fixed Docker development authentication by documenting and defaulting Compose to `ENVIRONMENT=development`, so session cookies work over direct HTTP while production keeps `Secure` cookies behind HTTPS.
- Exposed `pgp_enabled` through the auth policy and hid OpenPGP settings, identity controls, composer controls and reader UI when OpenPGP is disabled by administrator policy.
Generated
+1 -1
View File
@@ -3508,7 +3508,7 @@ dependencies = [
[[package]]
name = "rustbox"
version = "0.2.1"
version = "0.2.2"
dependencies = [
"aes-gcm",
"async-imap",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "rustbox"
version = "0.2.1"
version = "0.2.2"
edition = "2024"
rust-version = "1.88"
default-run = "rustbox"
+1 -1
View File
@@ -1,6 +1,6 @@
# RustBox attribution / notice
RustBox 0.2.1 is derived from the Oxi webmail project:
RustBox 0.2.2 is derived from the Oxi webmail project:
- Upstream: `c0h1b4/oxi`
- Base snapshot: `7f1507c486b78cf33e54beb701dd460d3bd2bd5d`
+3 -3
View File
@@ -1,4 +1,4 @@
# RustBox 0.2.1
# RustBox 0.2.2
RustBox is a webmail derived from `c0h1b4/oxi`. The consumed Oxi revision is recorded in `UPSTREAM_BASE`.
@@ -34,7 +34,7 @@ rustbox/
`build.rs` combines the CSS modules from `src/web/css/manifest.txt` and JavaScript modules from `src/web/js/manifest.txt`. Cargo embeds the generated CSS/JS bundles, HTML and SVG sprite in the RustBox executable. There is no standalone source `src/web/app.js`; `src/web/` is the single source of truth and Cargo generates the embedded bundles in `OUT_DIR`.
## Storage and search in 0.2.1
## Storage and search in 0.2.2
RustBox separates three concerns: SQL metadata, blob storage and search. The SQL engine does **not** decide whether `data_dir` may be used. A single-node PostgreSQL/MySQL deployment may still use filesystem blobs and local Tantivy. In cluster mode, local node state is rejected unless the configured backend is explicitly shared.
@@ -217,7 +217,7 @@ Initialize Git after extracting the release archive:
```bash
git init
git add .
git commit -m "RustBox 0.2.1 release"
git commit -m "RustBox 0.2.2 release"
./scripts/setup-upstream.sh
```
+28 -3
View File
@@ -1,4 +1,4 @@
RustBox 0.2.1 release manifest
RustBox 0.2.2 release manifest
Architecture:
- database_backend = sqlite | mysql | pgsql selects authoritative SQL only
@@ -27,19 +27,44 @@ Database/schema:
- runtime migration V047 adds attachment checksum metadata and search_outbox
- next migration number: V048
PostgreSQL message mutation fix (0.2.2):
- header_sync_generation UPSERT qualifies the current tenant user_meta.value on the UPDATE RHS
- this prevents PostgreSQL from treating `value` as ambiguous during ON CONFLICT DO UPDATE
- SQLite/MySQL equivalents were reviewed and require no matching SQL change
- no schema migration is required
PostgreSQL startup compile fix (0.2.2):
- multi-statement global schema DDL uses `sqlx::Executor::execute()` on the transaction connection
- avoids SQLx raw_sql mutable-connection lifetime inference failures (`Executor is not general enough`)
- retains the startup transaction and `SET LOCAL client_min_messages TO WARNING`
- no schema migration is required
Database/session startup (0.2.2):
- db::pool::configure() is the single owner of global schema initialization
- database-backed session persistence does not execute schema.sql a second time
- the fix applies uniformly to SQLite, MySQL and PostgreSQL
- PostgreSQL expected IF NOT EXISTS notices are suppressed only inside the startup DDL transaction
- no SQL migration is required; runtime schema remains V047
SQLite test isolation (0.2.2):
- test builds initialize each requested SQLite database but do not publish a TempDir-backed path into the process-wide database OnceLock
- account/database tests therefore remain isolated when Rust runs them in parallel
- a regression test covers deleting the first TempDir before opening a second test database
Docker examples:
- docker/compose-single-sqlite.yml
- docker/compose-postgres-s3-opensearch.yml (PostgreSQL + SILO/S3 + OpenSearch)
- docker/compose-mysql-nfs-opensearch.yml (MySQL + real NFS volume + OpenSearch)
Operations/docs:
- README.md documents 0.2.1 storage/search configuration
- README.md documents 0.2.2 storage/search configuration
- docker/README.md documents Compose variants
- docs/STORAGE_SEARCH_OPERATIONS.md documents failure modes, shared-storage requirements and migration behavior
- TODO.md leaves live SILO/MinIO/AWS/Ceph and real multi-node acceptance validation explicitly pending
Build/release metadata:
- Cargo.toml version = 0.2.1
- `scripts/package-release.sh` runs the full Rust release gate before creating ZIP/SHA256 artifacts
- Cargo.toml version = 0.2.2
- rust-toolchain.toml pins Rust 1.88.0
- Cargo.lock is intentionally not shipped because this environment could not regenerate it after dependency changes
- CI and scripts/build-release.sh generate Cargo.lock, then run locked tests/Clippy/release build
+17 -17
View File
@@ -1,30 +1,30 @@
RustBox 0.2.1 - final source validation
RustBox 0.2.2 - final source validation
Completed in this environment:
- offline release validator: scripts/validate-release-static.py (144/144 checks passed)
- offline release validator: scripts/validate-release-static.py (164/164 checks passed)
- all maintained TOML examples parse successfully
- Docker Compose topology/config references are validated by the offline release validator
- JSON files parse successfully
- browser JavaScript assembled in src/web/js/manifest.txt order passes node --check
- shell scripts pass bash -n
- Python helper scripts compile successfully
- Rust source delimiter sanity check passes after removing partial-patch leftovers
- git diff --check passes on the final source tree
- targeted source checks validate the PostgreSQL header-sync UPSERT and startup schema-initialization control flow
- the PostgreSQL startup DDL path now uses `Executor::execute()` instead of the lifetime-problematic `raw_sql(...).execute(&mut PgConnection)` path; compiler validation remains part of the Rust-host release gate
- whitespace/patch-artifact checks pass on the final source tree (this packaged worktree has no `.git` metadata)
- no Cargo.lock, *.orig, *.rej, temporary FIX/APPLY scripts, or patch leftovers are shipped
- regression source checks retain the 0.2.0 release gate and add 0.2.1 policy/backpressure checks
- regression source checks retain the PostgreSQL header-sync qualification audit and now reject the SQLx raw_sql mutable-connection lifetime regression in startup DDL
- SQLite unit-test configuration no longer leaks a temporary database path through the process-wide OnceLock; a dedicated regression test covers deleted TempDir reuse
- the static release contract now verifies that build-release.sh requires cargo test, Clippy with -D warnings, and a locked release build
- scripts/package-release.sh runs the full build gate before creating the source ZIP and SHA256, so packaging cannot succeed after a failed Rust test/Clippy/build gate
0.2.1 fixes included:
- OpenPGP policy visibility follows pgp_enabled and disabled OpenPGP is hidden from the UI
- OpenSearch 429/flood-stage failures trigger worker backoff instead of per-event warning storms
- Docker development defaults session cookies for direct HTTP and documents the production HTTPS requirement
- cluster_mode controls the implicit data_dir default; PostgreSQL/MySQL remain valid with local data_dir backends in single-node deployments
- storage/search configuration comes only from [storage] and [search]
- removed pre-0.2 top-level storage/search switches and the tantivy backend alias
- removed runtime legacy attachment read-through; attachment access uses logical BlobStorage keys
- fixed search_outbox tenant/import table parity
- fixed nested SQLite transactions in message move paths
- fixed the Clippy -D warnings failures reported for examples, config, storage, search, routes, realtime and formatting
- scripts/build-release.sh removes stale Cargo.lock, runs the static gate, generates a fresh lockfile for locked Cargo commands, and cleans it on exit
0.2.2 fixes included:
- PostgreSQL startup schema initialization no longer triggers SQLx `Executor is not general enough`; raw multi-statement DDL is executed directly through `Executor::execute()` inside the existing transaction
- PostgreSQL header-sync invalidation qualifies the existing tenant user_meta.value in ON CONFLICT DO UPDATE, eliminating the ambiguous-column 500 seen during message mutations such as marking spam
- tenant SQL rewriting has a regression test proving the qualified user_meta reference is rewritten to the tenant table
- SQLite and MySQL equivalents were reviewed and do not have the PostgreSQL ambiguity
- 0.2.2 single-schema-owner and PostgreSQL NOTICE-noise fixes remain in place
- no database migration is required; V047 remains current and V048 remains next
- 0.2.1 OpenPGP visibility, OpenSearch backpressure and HTTP/dev cookie fixes remain in place
Not executed in this packaging environment:
- cargo fmt/test/clippy/build with Rust 1.88, because Rust/Cargo is not installed here and outbound DNS is unavailable
+1 -1
View File
@@ -1,4 +1,4 @@
# RustBox 0.2.1
# RustBox 0.2.2
host = "127.0.0.1"
port = 3001
environment = "production"
+1 -1
View File
@@ -164,4 +164,4 @@ docker compose -p rustbox -f compose-postgres-s3-opensearch.yml exec opensearch
-d '{"index.blocks.read_only_allow_delete":null}'
```
RustBox 0.2.1 backs off its OpenSearch outbox worker after these 429/flood-stage responses, so a full disk no longer produces one warning per queued event.
RustBox 0.2.2 backs off its OpenSearch outbox worker after these 429/flood-stage responses, so a full disk no longer produces one warning per queued event.
+1 -1
View File
@@ -76,7 +76,7 @@ services:
SMTP_HOST: ${SMTP_HOST:-mail.example.com}
SMTP_PORT: ${SMTP_PORT:-587}
TLS_ENABLED: ${TLS_ENABLED:-true}
RUST_LOG: ${RUST_LOG:-info,sqlx::postgres::notice=warn}
RUST_LOG: ${RUST_LOG:-info}
DATABASE_URL: postgres://rustbox:${POSTGRES_PASSWORD:-rustbox-dev-password}@postgres:5432/rustbox
SESSION_ENCRYPTION_KEY: ${SESSION_ENCRYPTION_KEY:-MDEyMzQ1Njc4OWFiY2RlZjAxMjM0NTY3ODlhYmNkZWY=}
AWS_ACCESS_KEY_ID: ${S3_ACCESS_KEY:-rustbox}
+4 -3
View File
@@ -1,8 +1,8 @@
# RustBox 0.2.1 - implementation status
# RustBox 0.2.2 - implementation status
This file tracks the `0.2.1` source snapshot. Storage/search scope and outstanding acceptance work are tracked in `TODO.md`. A feature is only listed as implemented when working code exists in this source tree; partial items are called out explicitly.
This file tracks the `0.2.2` source snapshot. Storage/search scope and outstanding acceptance work are tracked in `TODO.md`. A feature is only listed as implemented when working code exists in this source tree; partial items are called out explicitly.
## 0.2.1 storage/search architecture
## 0.2.2 storage/search architecture
- SQL backend selection (`sqlite`/`mysql`/`pgsql`) is independent from blob storage and search backend selection.
- Cluster mode clears an implicit/default `data_dir`; PostgreSQL/MySQL single-node deployments may still intentionally use `data_dir`-backed filesystem storage and local Tantivy.
@@ -11,6 +11,7 @@ This file tracks the `0.2.1` source snapshot. Storage/search scope and outstandi
- OpenSearch is available as the cluster-capable indexer. Local Tantivy remains single-node only; search may also be disabled.
- Search mutations use a durable SQL outbox with claim/lease, retry/backoff and dead-letter state; the index remains rebuildable derived state.
- V047 adds attachment checksum metadata and the search outbox across SQLite, MySQL and PostgreSQL.
- SQL global-schema initialization is centralized in `db::pool::configure()`; database-backed sessions do not repeat startup DDL for SQLite, MySQL or PostgreSQL.
- Docker Compose examples cover standalone SQLite, PostgreSQL + SILO (S3-compatible) + OpenSearch, and MySQL + NFS + OpenSearch.
- Live SILO/MinIO/AWS S3/Ceph RGW, live NFS multi-host, live OpenSearch and full multi-node failover acceptance remain release-environment validation items and are not represented as completed here.
+5 -6
View File
@@ -3,15 +3,14 @@
1. Update the version in `Cargo.toml` and the RustBox package entry in `Cargo.lock`.
2. Confirm `UPSTREAM_BASE` is the Oxi revision actually consumed by the release.
3. Ensure no TypeScript sources or separate frontend build dependencies are present.
4. Run:
4. Run the release gate and package only after it passes:
```bash
cargo fmt --all -- --check
cargo test --locked
cargo clippy --locked --all-targets -- -D warnings
./scripts/build-release.sh
./scripts/package-release.sh
```
`package-release.sh` runs `scripts/build-release.sh` first. That gate regenerates the lockfile, runs the static validator, `cargo fmt --check`, the full `cargo test --locked` suite, Clippy with `-D warnings`, and the locked release build. The ZIP and SHA256 are created only after all of those steps succeed.
5. Verify:
```bash
@@ -27,7 +26,7 @@ RUSTBOX_CONFIG=./config.dev.toml ./target/release/rustbox --check-config
Browser JavaScript source lives in functional modules under `src/web/js/`, ordered by `src/web/js/manifest.txt`. `build.rs` concatenates those modules into the embedded `rustbox-ui.js` asset during every Cargo build. The repository has no duplicate `ui/` snapshot; `src/web/` is the only browser UI source.
## Updating an existing checkout to 0.2.1
## Updating an existing checkout to 0.2.2
Do not overlay the release with `cp *` or another command that skips dotfiles. That can retain stale `.github`, `.env.example`, or `Cargo.lock` files.
+2 -2
View File
@@ -1,6 +1,6 @@
# RustBox 0.2.1 storage/search operations
# RustBox 0.2.2 storage/search operations
RustBox 0.2.1 treats SQL, blob storage and search as independent capabilities. The database engine does not decide whether `data_dir` is allowed.
RustBox 0.2.2 treats SQL, blob storage and search as independent capabilities. The database engine does not decide whether `data_dir` is allowed.
## Deployment matrix
+2 -2
View File
@@ -1,6 +1,6 @@
# Database schemas and migrations
RustBox 0.2.1 keeps SQL, blob storage and search as independent concerns:
RustBox 0.2.2 keeps SQL, blob storage and search as independent concerns:
```text
database_backend = sqlite | mysql | pgsql
@@ -34,7 +34,7 @@ Account data is isolated inside the selected physical database with a determinis
Every schema-changing release must add the same logical migration for all supported SQL engines and register it in `src/db/pool.rs`. Do not update only `schema.sql`: fresh installs use the baseline while existing installations use the numbered migration path.
The next migration number after RustBox 0.2.1 is **V048**.
The next migration number after RustBox 0.2.2 is **V048**.
Redis and Redis Sentinel do not use SQL migrations.
+1 -1
View File
@@ -1,4 +1,4 @@
# RustBox 0.2.1 - packaged production configuration (DEB/RPM)
# RustBox 0.2.2 - packaged production configuration (DEB/RPM)
#
# This file contains all supported RustBox configuration options.
# Commented options are optional and do not change the packaged defaults.
+30
View File
@@ -0,0 +1,30 @@
#!/usr/bin/env bash
set -euo pipefail
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$ROOT"
./scripts/build-release.sh
VERSION="$(python3 - <<'PY'
import tomllib
from pathlib import Path
with Path('Cargo.toml').open('rb') as fh:
print(tomllib.load(fh)['package']['version'])
PY
)"
OUT_DIR="${1:-$ROOT/dist}"
mkdir -p "$OUT_DIR"
ARCHIVE="$OUT_DIR/RustBox-$VERSION.zip"
CHECKSUM="$ARCHIVE.sha256"
python3 make_zip.py "$ARCHIVE"
(
cd "$(dirname "$ARCHIVE")"
sha256sum "$(basename "$ARCHIVE")" > "$(basename "$CHECKSUM")"
)
printf 'Release archive: %s\n' "$ARCHIVE"
printf 'SHA256: '
cat "$CHECKSUM"
+90 -6
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env python3
"""Static/offline release checks for RustBox 0.2.1.
"""Static/offline release checks for RustBox 0.2.2.
These checks intentionally avoid pretending to replace `cargo test`: they validate
configuration, SQL migrations, Docker examples, query-catalog parity and release
@@ -49,19 +49,26 @@ def extract_section(text: str, begin: str, end: str) -> str:
# Release/toolchain metadata.
cargo = tomllib.loads(read("Cargo.toml"))
require(cargo["package"]["version"] == "0.2.1", "Cargo package version is 0.2.1")
require(cargo["package"]["version"] == "0.2.2", "Cargo package version is 0.2.2")
require(cargo["package"]["rust-version"] == "1.88", "Cargo rust-version is 1.88")
toolchain = tomllib.loads(read("rust-toolchain.toml"))
require(toolchain["toolchain"]["channel"] == "1.88.0", "rust-toolchain pins Rust 1.88.0")
require(not (ROOT / "Cargo.lock").exists(), "stale Cargo.lock is not shipped")
require("cargo generate-lockfile" in read("scripts/build-release.sh"), "release script regenerates Cargo.lock")
release_script = read("scripts/build-release.sh")
require("cargo test --locked" in release_script, "release script requires the full Rust test suite")
require("cargo clippy --locked --all-targets -- -D warnings" in release_script, "release script requires warning-free Clippy")
require("cargo build --release --locked" in release_script, "release script requires a locked release build")
package_script = read("scripts/package-release.sh")
require("./scripts/build-release.sh" in package_script, "release packaging runs the full build gate before creating artifacts")
require("make_zip.py" in package_script and "sha256sum" in package_script, "release packaging creates ZIP and SHA256 only after the build gate")
require("cargo generate-lockfile" in read(".github/workflows/ci.yml"), "CI regenerates Cargo.lock before --locked checks")
require("validate-release-static.py" in read("scripts/build-release.sh"), "release script runs offline/static release gate")
require("validate-release-static.py" in read(".github/workflows/ci.yml"), "CI runs offline/static release gate")
require(read("README.md").startswith("# RustBox 0.2.1"), "README identifies RustBox 0.2.1")
require(read("RELEASE-MANIFEST.txt").startswith("RustBox 0.2.1"), "release manifest identifies RustBox 0.2.1")
require(read("RELEASE-VALIDATION.txt").startswith("RustBox 0.2.1"), "release validation identifies RustBox 0.2.1")
require(read("docs/IMPLEMENTATION_STATUS.md").startswith("# RustBox 0.2.1"), "implementation status identifies RustBox 0.2.1")
require(read("README.md").startswith("# RustBox 0.2.2"), "README identifies RustBox 0.2.2")
require(read("RELEASE-MANIFEST.txt").startswith("RustBox 0.2.2"), "release manifest identifies RustBox 0.2.2")
require(read("RELEASE-VALIDATION.txt").startswith("RustBox 0.2.2"), "release validation identifies RustBox 0.2.2")
require(read("docs/IMPLEMENTATION_STATUS.md").startswith("# RustBox 0.2.2"), "implementation status identifies RustBox 0.2.2")
require((ROOT / "docs/STORAGE_SEARCH_OPERATIONS.md").is_file(), "storage/search operations guide exists")
require('imap_host = ""' not in read("config.dev.toml"), "development config does not override IMAP host with an invalid empty value")
require(not re.search(r"(?m)^IMAP_HOST=\s*$", read(".env.example")), "environment example does not set an invalid empty IMAP host")
@@ -70,6 +77,78 @@ require("setOpenPgpAvailability" in read("src/web/js/72-openpgp-identities.js"),
require("is_opensearch_backpressure_error" in read("src/search/engine.rs"), "OpenSearch worker classifies backpressure failures")
require("worker pass failed; backing off" in read("src/search/engine.rs"), "OpenSearch worker backs off after failures")
require("ENVIRONMENT: ${ENVIRONMENT:-development}" in read("docker/compose-postgres-s3-opensearch.yml"), "PostgreSQL Compose defaults direct HTTP to development cookie mode")
session_db = read("src/session_store/database.rs")
db_pool = read("src/db/pool.rs")
require("crate::db::pool::configure(config).await?" in session_db, "database session store delegates schema initialization to db::pool::configure")
require("global_schema_for_backend" not in session_db, "database session store does not run a second backend schema pass")
require("sqlx::raw_sql(&schema)" not in session_db and "execute_batch(&schema)" not in session_db, "session store avoids duplicate schema DDL for SQLite/MySQL/PostgreSQL")
require("SET LOCAL client_min_messages TO WARNING" in db_pool, "PostgreSQL idempotent startup DDL suppresses expected NOTICE noise locally")
require("Executor as _" in db_pool, "PostgreSQL startup imports SQLx Executor for direct raw DDL execution")
require(".execute(schema.as_str())" in db_pool, "PostgreSQL startup executes schema DDL through Executor instead of raw_sql")
require(not re.search(r"raw_sql\(&schema\)\s*\.execute\(&mut \*tx\)", db_pool), "PostgreSQL startup avoids raw_sql mutable-connection lifetime regression")
require("sqlite_test_configuration_does_not_pin_deleted_tempdir" in db_pool, "SQLite test isolation has a regression test for deleted TempDir reuse")
require("let config = AppConfig {" in db_pool and "let mut config = AppConfig::default();" not in db_pool, "SQLite isolation regression test is Clippy-clean with struct initialization")
require("each test's own data_dir" in db_pool and "#[cfg(test)]" in db_pool, "SQLite test configuration does not publish a temporary database path globally")
require("RUST_LOG: ${RUST_LOG:-info,sqlx::postgres::notice=warn}" not in read("docker/compose-postgres-s3-opensearch.yml"), "PostgreSQL Compose no longer depends on a notice-filter workaround")
pgsql_queries = read("src/queries/pgsql/queries.rs")
require("ON CONFLICT(key) DO UPDATE SET value = CAST(user_meta.value AS INTEGER) + 1" in pgsql_queries, "PostgreSQL header-sync UPSERT qualifies the existing target value")
require("tenant_rewriter_prefixes_header_sync_target_qualifier" in db_pool, "tenant SQL rewriter has a regression test for qualified header-sync value")
mysql_queries = read("src/queries/mysql/queries.rs")
require("ON DUPLICATE KEY UPDATE value = CAST(value AS SIGNED) + 1" in mysql_queries, "MySQL header-sync uses its non-EXCLUDED duplicate-key form")
# PostgreSQL UPSERT audit: any read of an INSERT target column inside
# ON CONFLICT ... DO UPDATE must be qualified as either excluded.<column>
# or <target_table>.<column>. PostgreSQL otherwise treats names such as
# `value` as ambiguous between the existing target row and EXCLUDED.
def postgres_upsert_ambiguities(source: str) -> list[str]:
const_re = re.compile(
r'pub const\s+(\w+)\s*:\s*&str\s*=\s*(r#".*?"#|"(?:\\.|[^"\\])*")\s*;',
re.S,
)
problems: list[str] = []
for const_name, literal in const_re.findall(source):
if literal.startswith('r#"'):
sql = literal[3:-2]
else:
sql = bytes(literal[1:-1], 'utf-8').decode('unicode_escape')
if not re.search(r'ON\s+CONFLICT.*DO\s+UPDATE\s+SET', sql, re.I | re.S):
continue
insert = re.search(
r'INSERT\s+INTO\s+([A-Za-z_][\w]*)\s*\((.*?)\)\s*VALUES',
sql,
re.I | re.S,
)
if insert is None:
problems.append(f'{const_name}: unable to parse INSERT target')
continue
table = insert.group(1)
columns = [column.strip().strip('"') for column in insert.group(2).split(',')]
update = re.split(r'DO\s+UPDATE\s+SET', sql, maxsplit=1, flags=re.I)[1]
scan = re.sub(r"'(?:''|[^'])*'", '', update)
for column in columns:
candidate = re.sub(
rf'\b(?:excluded|{re.escape(table)})\s*\.\s*{re.escape(column)}\b',
'',
scan,
flags=re.I,
)
candidate = re.sub(
rf'(?:(?<=^)|(?<=,))\s*{re.escape(column)}\s*=',
'=',
candidate,
flags=re.I,
)
if re.search(rf'(?<!\.)\b{re.escape(column)}\b', candidate, re.I):
problems.append(f'{const_name}: unqualified target read `{column}` in {table} UPSERT')
return problems
pgsql_upsert_ambiguities = postgres_upsert_ambiguities(pgsql_queries)
require(
not pgsql_upsert_ambiguities,
'PostgreSQL ON CONFLICT target-column reads are fully qualified'
+ (f" ({'; '.join(pgsql_upsert_ambiguities)})" if pgsql_upsert_ambiguities else ''),
)
# Parse all maintained TOML examples.
toml_files = [
@@ -152,6 +231,11 @@ try:
require(outbox is not None, "fresh SQLite schema has search_outbox")
outbox_columns = {row[1] for row in conn.execute("PRAGMA table_info(u_test__search_outbox)")}
require("claimed_until" in outbox_columns, "fresh SQLite search_outbox separates worker lease from retry availability")
sqlite_header_sync = "INSERT INTO u_test__user_meta (key, value) VALUES ('header_sync_generation', '1') ON CONFLICT(key) DO UPDATE SET value = CAST(value AS INTEGER) + 1"
conn.execute(sqlite_header_sync)
conn.execute(sqlite_header_sync)
sqlite_generation = conn.execute("SELECT value FROM u_test__user_meta WHERE key='header_sync_generation'").fetchone()
require(sqlite_generation == ("2",), "SQLite header-sync UPSERT remains unambiguous and increments generation")
account_default = conn.execute("PRAGMA table_info(rustbox_accounts)").fetchall()
version_default = next((row[4] for row in account_default if row[1] == "schema_version"), None)
require(str(version_default).strip("'\"") == "46", "fresh global compatibility baseline remains V046 before tenant registration")
+74 -11
View File
@@ -9,7 +9,7 @@ use rusqlite::OptionalExtension as _;
use rusqlite::types::{ToSql, ToSqlOutput, Value};
use sha2::{Digest, Sha256};
use sqlx::pool::PoolConnection;
use sqlx::{MySql, MySqlPool, PgPool, Postgres, Row as SqlxRow};
use sqlx::{Executor as _, MySql, MySqlPool, PgPool, Postgres, Row as SqlxRow};
use tokio::runtime::{Builder, Handle};
use tokio::sync::Mutex as AsyncMutex;
@@ -180,14 +180,6 @@ fn global_schema(schema: &str) -> Result<String, String> {
schema_section(schema, "-- RUSTBOX:GLOBAL-BEGIN", "-- RUSTBOX:GLOBAL-END")
}
pub(crate) fn global_schema_for_backend(backend: DatabaseBackend) -> Result<String, String> {
match backend {
DatabaseBackend::Sqlite => global_schema(SQLITE_SCHEMA),
DatabaseBackend::Mysql => global_schema(MYSQL_SCHEMA),
DatabaseBackend::Pgsql => global_schema(PGSQL_SCHEMA),
}
}
fn tenant_schema(schema: &str, prefix: &str) -> Result<String, String> {
Ok(
schema_section(schema, "-- RUSTBOX:TENANT-BEGIN", "-- RUSTBOX:TENANT-END")?
@@ -236,8 +228,23 @@ pub async fn configure(config: &AppConfig) -> Result<(), String> {
.map_err(|e| format!("failed to configure SQLite database: {e}"))?;
conn.execute_batch(&global_schema(SQLITE_SCHEMA)?)
.map_err(|e| format!("SQLite global schema initialization failed: {e}"))?;
// Unit tests create many independent temporary data directories in one
// process. Pinning the first SQLite path in the process-wide OnceLock
// makes later tests reuse a TempDir that may already have been deleted,
// causing a cascade of "unable to open database file" failures.
// Initialize the requested test database, but let open_user_db() resolve
// each test's own data_dir instead of publishing this path globally.
#[cfg(test)]
{
return Ok(());
}
#[cfg(not(test))]
{
ConfiguredDatabase::Sqlite { path }
}
}
DatabaseBackend::Mysql => {
let url = config
.database_url
@@ -280,10 +287,29 @@ pub async fn configure(config: &AppConfig) -> Result<(), String> {
let schema = global_schema(PGSQL_SCHEMA)?;
let init_pool = pool.clone();
runtime_wait(&runtime, async move {
sqlx::raw_sql(&schema)
.execute(&init_pool)
// PostgreSQL emits NOTICE for every CREATE ... IF NOT EXISTS that
// already exists. Keep expected idempotent startup DDL quiet without
// changing application-wide logging or server settings.
let mut tx = init_pool
.begin()
.await
.map_err(|e| format!("PostgreSQL global schema transaction failed: {e}"))?;
sqlx::query("SET LOCAL client_min_messages TO WARNING")
.execute(&mut *tx)
.await
.map_err(|e| format!("failed to quiet PostgreSQL schema notices: {e}"))?;
// `raw_sql(...).execute(&mut PgConnection)` can trigger SQLx's
// "Executor is not general enough" lifetime error when this
// future is spawned onto the dedicated database runtime. Execute
// the unprepared multi-statement DDL directly through `Executor`
// instead; this is SQLx's supported path for raw DDL batches.
(&mut *tx)
.execute(schema.as_str())
.await
.map_err(|e| format!("PostgreSQL global schema initialization failed: {e}"))?;
tx.commit()
.await
.map_err(|e| format!("PostgreSQL global schema commit failed: {e}"))?;
Ok(())
})?;
ConfiguredDatabase::Pgsql { pool, runtime }
@@ -2287,6 +2313,31 @@ pub fn open_test_db() -> Connection {
mod tests {
use super::*;
#[tokio::test]
async fn sqlite_test_configuration_does_not_pin_deleted_tempdir() {
let first = tempfile::tempdir().expect("failed to create first temp dir");
let config = AppConfig {
database_backend: "sqlite".to_string(),
database_url: None,
data_dir: first.path().to_string_lossy().into_owned(),
..Default::default()
};
configure(&config)
.await
.expect("failed to initialize first SQLite test database");
assert!(first.path().join("rustbox.sqlite").is_file());
drop(first);
let second = tempfile::tempdir().expect("failed to create second temp dir");
let second_dir = second.path().to_string_lossy().into_owned();
let connection = open_user_db(&second_dir, "second-test-account")
.expect("second test must not reuse the deleted first SQLite path");
assert_eq!(connection.backend(), DatabaseBackend::Sqlite);
assert!(second.path().join("rustbox.sqlite").is_file());
}
#[test]
fn table_prefix_is_short_and_deterministic() {
let hash = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef";
@@ -2322,6 +2373,18 @@ mod tests {
);
}
#[test]
fn tenant_rewriter_prefixes_header_sync_target_qualifier() {
let sql = prefix_tenant_tables(
"INSERT INTO user_meta (key, value) VALUES ('header_sync_generation', '1') ON CONFLICT(key) DO UPDATE SET value = CAST(user_meta.value AS INTEGER) + 1",
"u_test__",
);
assert_eq!(
sql,
"INSERT INTO u_test__user_meta (key, value) VALUES ('header_sync_generation', '1') ON CONFLICT(key) DO UPDATE SET value = CAST(u_test__user_meta.value AS INTEGER) + 1"
);
}
#[test]
fn test_schema_contains_sessions_and_user_tables() {
let conn = open_test_db();
+1 -1
View File
@@ -243,7 +243,7 @@ pub mod db_header_sync {
pub const Q_001: &str = "SELECT value FROM user_meta WHERE key = 'header_sync_generation'";
pub const Q_002: &str =
"INSERT INTO user_meta (key, value) VALUES ('header_sync_generation', '1')
ON CONFLICT(key) DO UPDATE SET value = CAST(value AS INTEGER) + 1";
ON CONFLICT(key) DO UPDATE SET value = CAST(user_meta.value AS INTEGER) + 1";
}
pub mod db_identities {
+1 -4
View File
@@ -1114,10 +1114,7 @@ async fn process_search_outbox(
Err(error) => {
if is_opensearch_backpressure_error(&error) {
crate::db::search_outbox::defer_transient_failure(
&conn,
&event,
&error,
30,
&conn, &event, &error, 30,
)?;
return Err(format!(
"OpenSearch temporarily rejected indexing for tenant {user_hash}: {error}"
+8 -23
View File
@@ -6,7 +6,6 @@ use sqlx::{FromRow, MySqlPool, PgPool};
use super::{PersistedSession, SessionPersistence};
use crate::config::AppConfig;
use crate::db::pool::{DatabaseBackend, global_schema_for_backend};
use crate::queries;
#[derive(Clone)]
@@ -149,20 +148,15 @@ pub struct DatabaseSessionStore {
impl DatabaseSessionStore {
pub async fn connect(config: &AppConfig) -> Result<Self, String> {
// Schema ownership is centralized in db::pool::configure(). Calling it here
// keeps direct/test construction safe, while the normal application startup
// hits the OnceLock fast path. Do not run schema.sql again from this store:
// that used to duplicate DDL for SQLite/MySQL/PostgreSQL and produced repeated
// PostgreSQL "relation already exists, skipping" NOTICE messages.
crate::db::pool::configure(config).await?;
let pool = match config.database_backend.as_str() {
"sqlite" => {
let path = sqlite_path(config)?;
let migration_path = path.clone();
let schema = global_schema_for_backend(DatabaseBackend::Sqlite)?;
sqlite_task(migration_path, move |conn| {
conn.execute_batch(&schema).map_err(|e| {
format!("SQLite database schema initialization failed: {e}")
})?;
Ok(())
})
.await?;
Pool::Sqlite(path)
}
"sqlite" => Pool::Sqlite(sqlite_path(config)?),
"mysql" => {
let url = config
.database_url
@@ -171,11 +165,6 @@ impl DatabaseSessionStore {
let pool = MySqlPool::connect(url)
.await
.map_err(|e| format!("failed to connect MySQL database: {e}"))?;
let schema = global_schema_for_backend(DatabaseBackend::Mysql)?;
sqlx::raw_sql(&schema)
.execute(&pool)
.await
.map_err(|e| format!("MySQL database schema initialization failed: {e}"))?;
Pool::Mysql(pool)
}
"pgsql" => {
@@ -186,10 +175,6 @@ impl DatabaseSessionStore {
let pool = PgPool::connect(url)
.await
.map_err(|e| format!("failed to connect PostgreSQL database: {e}"))?;
let schema = global_schema_for_backend(DatabaseBackend::Pgsql)?;
sqlx::raw_sql(&schema).execute(&pool).await.map_err(|e| {
format!("PostgreSQL database schema initialization failed: {e}")
})?;
Pool::Pgsql(pool)
}
other => return Err(format!("unsupported database backend: {other}")),