v0.31.7

docs/API.md

@@ -599,3 +599,7 @@ As of 0.24.3 the preferred configuration is per listener: `listeners.endpoints[]
 The top-level `http2` object introduced in 0.24.0 and `PUT /api/config/http2` remain accepted for backward compatibility. They are a fallback only for listeners that do not yet have explicit `binding.http2_settings`; the structured 0.24.3 UI writes listener-scoped values.
 
 The runtime system diagnostics object includes `platform`, supplied by the shared system-statistics backend (`Linux`, `FreeBSD`, `OpenBSD`, `NetBSD`, or the generic/unsupported fallback). The browser only renders that value; it does not detect the operating system itself.
+
+## Per-vhost User-Agent rate limiting (0.31.7)
+
+The revisioned `/api/config` vhost rate-limit object accepts optional `user_agent_limits`, up to 64 entries with `pattern` (1..256 bytes, case-insensitive substring or `*` wildcard), `requests` (1..1000000), `window_seconds` (1..3600) and `burst` (0..1000000). The first matching rule replaces the standard and country rate for that request and uses a separate fixed-window counter for each (vhost, rule index, client IP). Requests with no matching User-Agent use the existing limiter. A User-Agent is trivially spoofable; these overrides are operational shaping, not access control. Global and vhost User-Agent blocklists continue to deny matching requests before rate limiting.

docs/IMPLEMENTATION_0.31.7.md

@@ -0,0 +1,29 @@
+# RustProxy 0.31.7 implementation and acceptance gate
+
+## Settings and vhost throttling
+
+- **Settings → Rules & headers** includes the Global User-Agent blocklist, Global custom headers, and Client response sections. Forms and request endpoints are unchanged, so there is no data migration. Settings → Security retains password, session and host-policy controls.
+- `vhosts[].rate_limit.user_agent_limits` is an optional list (default `[]`) of `{pattern, requests, window_seconds, burst}`. At most 64 rules. Patterns use case-insensitive substring matching when no `*` is present, otherwise wildcard matching. Values are normalized and validated before persistence. The first matching rule has priority over the country override and base limiter for that request. No matching UA uses existing country/base behavior; missing UA does not select a UA override. Every vhost/UA rule/client IP has an independent fixed-window bucket within the shared 65,536-bucket cap. Cache, ACL, and global/vhost UA deny policies run with unchanged precedence.
+- **Important:** UA is client-supplied. These rules provide traffic shaping, not trustworthy identity or bot protection. An absent/changed/oversized UA may avoid matching. Use ACL and base limits for security enforcement.
+
+## Revision history and layout
+
+- Revision-history UI polish: comparison control sits under Restore without extra dividers; long change lists scroll within the bounded desktop detail pane, while the stacked narrow-screen layout scrolls as one modal. Full JSON comparison uses the shared dialog border and borderless code blocks.
+- Diff preview displays only changed leaf paths, sorted by object key. Array entries with stable `id` or `name` keys are compared by identity, avoiding spurious changes on reorder. Output is limited to 200 changes; large scalar previews truncate after 320 characters. The separate full JSON modal is opened explicitly and uses `textContent`, never HTML injection. Restore remains CAS-protected.
+- Listener protocol buttons have a gap and may wrap; ACL profile attachments are in a closed-by-default native `<details>` element.
+
+## Installed release retention and Cargo safety
+
+- `update.sh` executes installer/update first, then prunes under the existing `.deploy.lock`. It retains `current` and one other latest release; three private pre-update backups remain (separate retention). Old updater incorrectly passed `find` sort keys into `rm` paths, so it printed removals but removed nothing; fixed by splitting the mtime key before path validation/removal.
+- Cargo `target` is not cleared. Only directories under `target/{debug,release}/incremental` older than 30 days are eligible, and only if the corresponding Cargo profile lock is available and no symlinked target path is used. `deps`, `.fingerprint`, `build`, `.cargo-lock`, binaries and Cargo.lock are preserved. This cleanup does not promise meaningful disk recovery for release builds (incremental compilation is usually disabled for release mode).
+- Managed updates create `.managed-update` before `systemctl stop`, and the service checks it during graceful shutdown. With an enabled cold tier, it forces a **best-effort** hot-to-cold direct persistence independent of the ordinary `cold_on_shutdown` checkbox. systemd grants 120 seconds, and runtime allows up to 90 seconds for the transfer after draining. The reported moved count reflects successful COLD insertions rather than merely attempted writes. Entries may still fail to persist (I/O/disk limits/timeouts); verify logs and COLD objects in an actual update. Disabling COLD does not allow automatic persistence to disk.
+
+## Upgrade and live acceptance (open)
+
+1. Compile and run `./dev.sh check`, `./dev.sh test` and `./dev.sh release` on Rust 1.99+ and verify the lockfile resolves. Do not call this release production-tested without those results.
+2. Test Config revision CAS, UA matching and rate 429/Retry-After across two vhosts, clients, country policies and ordinary rate limits. Check `/settings?tab=policies` in EN/PL, including unsaved forms and the new full JSON modal.
+3. On a staging installation with HOT+COLD enabled and `cold_on_shutdown=false`, cache a cacheable GET, run `bash update.sh`, confirm managed-update stop log and cold recovery after restart. Repeat rollback on a failed staged binary; backup retention must not remove the rollback candidate.
+4. Verify `releases/` retains current and one predecessor, backups still retain three; verify `target/release/deps`, `.fingerprint`, `build` and the compiled binary remain intact and next `bash dev.sh release` is incremental.
+5. Re-test H3 logo navigation on iPhone and log regressions; prior H3/iOS acceptance is still open.
+
+SQLite schema 9 and COLD format 3 unchanged. No persisted setting is removed; old JSON without the new `user_agent_limits` field still deserializes.

docs/README.md

@@ -31,6 +31,7 @@ This is the canonical index for the current project documentation. The authentic
 - [0.26.3 implementation note](V0.26.3_IMPLEMENTATION.md) - global interface scaling, chart themes, square draft dock and Settings navigation polish.
 - [0.26.2 implementation note](V0.26.2_IMPLEMENTATION.md) - unified Logs navigation, field-query search, dynamic JSON columns and RAW parsing.
 - [0.26.1 implementation note](V0.26.1_IMPLEMENTATION.md) - historical vhost-aware protocol logging and Stored Logs UX milestone.
+- [0.31.7 UA throttling, Settings, revision history and safe update retention](IMPLEMENTATION_0.31.7.md) - rollout, storage and acceptance.
 - [0.31.6 H3 iOS navigation hardening and diagnostic events](IMPLEMENTATION_0.31.6.md) - bounded empty GET/HEAD handling, H3 error draining and mobile reproduction.
 - [0.31.5 User-Agent blocking and consolidated log analysis](IMPLEMENTATION_0.31.5.md) - global/per-vhost UA blocklists, Traffic copy actions, repeated-value aggregation and per-vhost gzip consolidation.
 - [0.31.4 administration public Origin and settings](IMPLEMENTATION_0.31.4.md) - opt-in HTTPS origin, one-field DNS allowlist, additional hosts in advanced settings, legacy deduplication, CSRF and remaining deployment checks.
@@ -85,7 +86,7 @@ This is the canonical index for the current project documentation. The authentic
 
 ## Historical or version-labelled context
 
-Version-labelled `V*.md` files are retained as historical implementation/audit records. Their version number describes the audited milestone. The current release implementation note is [IMPLEMENTATION_0.31.6.md](IMPLEMENTATION_0.31.6.md); [IMPLEMENTATION_0.31.5.md](IMPLEMENTATION_0.31.5.md) is retained as the previous log/cache milestone; [IMPLEMENTATION_0.31.4.md](IMPLEMENTATION_0.31.4.md) is retained as the previous administration-Origin milestone; [IMPLEMENTATION_0.31.3.md](IMPLEMENTATION_0.31.3.md) is retained as the previous mobile recovery milestone; [IMPLEMENTATION_0.31.1.md](IMPLEMENTATION_0.31.1.md) is retained as the prior mobile milestone; [IMPLEMENTATION_0.31.0.md](IMPLEMENTATION_0.31.0.md) is retained as the previous milestone; [V0.29.1_IMPLEMENTATION.md](V0.29.1_IMPLEMENTATION.md) is retained as historical context, and [V0.29.0_IMPLEMENTATION.md](V0.29.0_IMPLEMENTATION.md) is retained as the prior redirect/health/cache/log milestone; [V0.28.2_IMPLEMENTATION.md](V0.28.2_IMPLEMENTATION.md) is retained as the prior certificate-hardening milestone; [V0.28.1_IMPLEMENTATION.md](V0.28.1_IMPLEMENTATION.md) is retained as the prior external-path milestone; [V0.28.0_IMPLEMENTATION.md](V0.28.0_IMPLEMENTATION.md) is retained as the prior GeoIP/listener/cache milestone, and [V0.27.3_IMPLEMENTATION.md](V0.27.3_IMPLEMENTATION.md) is retained as the prior HTTP/3/cache-maintenance milestone, [V0.27.2_IMPLEMENTATION.md](V0.27.2_IMPLEMENTATION.md) is retained as the prior managed-ACME repair, [V0.27.1_IMPLEMENTATION.md](V0.27.1_IMPLEMENTATION.md) as the prior managed-runtime/list-sorting milestone, and older version notes remain historical evidence. The latest retained audit is [V0.25.5_AUDIT.md](V0.25.5_AUDIT.md); older audit files remain available for traceability.
+Version-labelled `V*.md` files are retained as historical implementation/audit records. Their version number describes the audited milestone. The current release implementation note is [IMPLEMENTATION_0.31.7.md](IMPLEMENTATION_0.31.7.md); [IMPLEMENTATION_0.31.6.md](IMPLEMENTATION_0.31.6.md) is retained as the prior H3 milestone; [IMPLEMENTATION_0.31.5.md](IMPLEMENTATION_0.31.5.md) is retained as the previous log/cache milestone; [IMPLEMENTATION_0.31.4.md](IMPLEMENTATION_0.31.4.md) is retained as the previous administration-Origin milestone; [IMPLEMENTATION_0.31.3.md](IMPLEMENTATION_0.31.3.md) is retained as the previous mobile recovery milestone; [IMPLEMENTATION_0.31.1.md](IMPLEMENTATION_0.31.1.md) is retained as the prior mobile milestone; [IMPLEMENTATION_0.31.0.md](IMPLEMENTATION_0.31.0.md) is retained as the previous milestone; [V0.29.1_IMPLEMENTATION.md](V0.29.1_IMPLEMENTATION.md) is retained as historical context, and [V0.29.0_IMPLEMENTATION.md](V0.29.0_IMPLEMENTATION.md) is retained as the prior redirect/health/cache/log milestone; [V0.28.2_IMPLEMENTATION.md](V0.28.2_IMPLEMENTATION.md) is retained as the prior certificate-hardening milestone; [V0.28.1_IMPLEMENTATION.md](V0.28.1_IMPLEMENTATION.md) is retained as the prior external-path milestone; [V0.28.0_IMPLEMENTATION.md](V0.28.0_IMPLEMENTATION.md) is retained as the prior GeoIP/listener/cache milestone, and [V0.27.3_IMPLEMENTATION.md](V0.27.3_IMPLEMENTATION.md) is retained as the prior HTTP/3/cache-maintenance milestone, [V0.27.2_IMPLEMENTATION.md](V0.27.2_IMPLEMENTATION.md) is retained as the prior managed-ACME repair, [V0.27.1_IMPLEMENTATION.md](V0.27.1_IMPLEMENTATION.md) as the prior managed-runtime/list-sorting milestone, and older version notes remain historical evidence. The latest retained audit is [V0.25.5_AUDIT.md](V0.25.5_AUDIT.md); older audit files remain available for traceability.
 
 Older context documents such as [INTEGRATIONS_040](INTEGRATIONS_040.md), [MANAGEMENT](MANAGEMENT.md), [SECURITY_AUDIT](SECURITY_AUDIT.md), [ACME_RFC2136](ACME_RFC2136.md) and older version audits remain available for traceability but should not override the current functional/technical documents above.
 

docs/SETTINGS.md

@@ -34,6 +34,28 @@ The main `/api/events?interval=N` connection (allowlisted 1/2/5/10, default 2 se
 
 ACME diagnostics formerly shown in About and under Administration are now on this tab: mode/state, certificate environment path, and the active Python/pip/Certbot versions. Versions are probed by the ACME runtime setup with bounded subprocess output/timeouts and published from stored runtime status. They are not queried at every page refresh or status API request. Failed/unavailable probes leave the version empty. No new persistent settings or permissions are introduced.
 
+## Rules and headers (0.31.7)
+
+**Settings -> Rules & headers** (`/settings?tab=policies`) now groups the existing **Global User-Agent blocklist**, **Global custom headers** and **Client response** forms as persistent sub-tabs, following the Cache settings layout. There is no migration: each form continues to update its previous configuration section. **Settings -> Security** retains administrative hostname, Origin, session and password policies.
+
+**Vhosts -> Security -> Traffic policy** also supports optional `rate_limit.user_agent_limits` in each vhost, for example:
+
+```json
+{
+  "rate_limit": {
+    "enabled": true, "requests": 120, "window_seconds": 60, "burst": 30,
+    "user_agent_limits": [
+      {"pattern": "*scanner*", "requests": 10, "window_seconds": 60, "burst": 0},
+      {"pattern": "BadBot", "requests": 2, "window_seconds": 60, "burst": 0}
+    ]
+  }
+}
+```
+
+The first case-insensitive match wins. A plain pattern searches for a substring and `*` matches any sequence. A matching User-Agent uses its **own bucket for the vhost, rule and client IP** and overrides that request's standard/country throttle; nonmatching requests continue using existing rate rules. Rules also work with `enabled=false` for the base limiter. The User-Agent header is untrusted, so avoid treating these limits as authentication or identity enforcement.
+
+The **Revision history** detail shows only changed leaf fields, scrolling locally when there are many. **Full JSON comparison** is directly below Restore in the action group and opens a separate standard dialog. ACL profile *Attached to* details are collapsed by default. Listener HTTP/2 and HTTP/3 controls have additional spacing.
+
 ## Configuration and update semantics
 
 GET `/api/config` returns a revisioned envelope. PUT the entire document with
@@ -69,7 +91,7 @@ This is a settings fragment, not a replacement for the existing routing/vhost do
 A complete opt-in example is in `examples/cache-dashboard.json`.
 
 The dashboard exposes the complete raw JSON editor under **Settings -> Manual config**.
-The same card exposes **Revision history**, which reads the newest 512 retained configuration revisions from the configuration-history table introduced in SQLite schema 8 (current database schema 9) and groups them by month/day. Selecting a revision shows timestamp, actor and a bounded path-level before/after diff against the previous retained revision. An older retained revision can be restored from the same dialog; RustProxy validates it through the normal configuration update path and saves it as a new revision, preserving the prior history.
+The same card exposes **Revision history**, which reads the newest 512 retained configuration revisions from the configuration-history table introduced in SQLite schema 8 (current database schema 9) and groups them by month/day. Selecting a revision shows timestamp, actor and only the changed fields against the previous retained revision; the separate Full JSON comparison dialog contains complete JSON documents. An older retained revision can be restored from the same dialog; RustProxy validates it through the normal configuration update path and saves it as a new revision, preserving the prior history.
 The historical `/configuration` browser route is retained only as a compatibility alias and
 opens that Settings block; it is no longer a primary-navigation item. Prefer the structured
 section editors for routine changes so unrelated sections are not republished.

docs/TEST_REPORT.md

@@ -1,4 +1,21 @@
-# Verification report - RustProxy 0.31.6
+# Verification report - RustProxy 0.31.7
+
+## 0.31.7 — revision history UI follow-up
+
+- Move **Full JSON comparison** directly below **Restore revision** inside the existing action group, without additional separators. Preserve existing revision selection, restore handlers, diff logic and JSON display behavior.
+- Bound the revision history dialog to the viewport. Large change lists and day-revision buttons scroll inside their desktop panes; the calendar and the stacked mobile layout scroll without forcing page-level overflow. Make the full JSON dialog use the same base dialog border/background/padding as other windows, with a right-aligned close button and borderless JSON blocks.
+- **PASS — browser layout fixture:** headless Chromium with the actual HTML/CSS dialog fragments, 200 changed fields, 50 day revisions and six month calendars at viewport sizes 1280×800, 1024×600, 800×600, 390×760 and 320×568. Desktop change list has independently scrollable content (including the final change); narrow viewports have a scrollable modal body and do not expand the document width. Full JSON dialog shares the generic border rule. This is synthetic browser rendering, not a running Rust server or real Safari.
+- **OPEN:** end-to-end panel render with authenticated session, actual revision content and iPhone browser. Rust/Cargo compilation remains unverified in this environment.
+
+## 0.31.7 — source, updater and UI regression verification (2026-10-11)
+
+- **PASS:** `./dev.sh test` executed all bundled Python security/source/version/SQLite-9 migration/upgrade/memory checks, Node.js UI regressions, documentation validation and language-pack checks. It exits **127** only at the missing Rust/Cargo toolchain gate. Rust tests and release compilation did not run.
+- **PASS:** new `tools/checks/release_0317.py` exercised the actual Bash `update.sh` retention code using a sandboxed fake installer: successful, failed and `--plan` updates; release directory names with spaces; active symlink protection; exactly two installed releases and three backups; age-based Cargo incremental cleanup while preserving recent state, `deps`, `.fingerprint`, `build`, binaries and locks; and skip on symlinked Cargo targets. Systemd was not called.
+- **PASS:** new `tools/checks/release_0317_ui_test.mjs` executed the real configuration diff function against changed leaves, reordered ID-bearing arrays, created/deleted fields and the 200-entry bound. Validated Settings form relocation, listener spacing, collapsed ACL attachments, user-agent editor/OpenAPI and separate JSON dialog at the source level. This was not a live browser render.
+- **PASS:** `GOPROXY=off GOSUMDB=off go test ./...` in `docker-agent`, `node --check` for all browser JavaScript modules, OpenAPI JSON parsing, `bash -n dev.sh update.sh`, `docs_check.py` and `langs_check.py`.
+- **CHECKED IN SOURCE, NOT LIVE:** `scripts/deploy/deploy_runtime.py` creates `.managed-update` before `systemctl stop`; `src/main/admin_runtime.rs` checks the marker and calls forced HOT→COLD even with `cold_on_shutdown=false`; shutdown time budget is 90 seconds with a 120-second systemd stop budget. The cache persistence worker now returns a success boolean so `moved` counts only successfully inserted COLD objects. Existing Rust cold-tier unit tests cover enabled COLD with ordinary shutdown disabled, but could **not** run without Cargo.
+- **OPEN:** Rust type checking/formatting, cargo --locked consistency, compiled vhost 429/Retry-After behavior, actual update/rollback and persistence after restart, browser UI on a live server, H2/H3 performance and iPhone reload acceptance. No production-ready claim.
+
 
 ## 0.31.6 - iPhone H3 full navigation and observability (2026-10-11)