v0.25.0

docs/TEST_REPORT.md

@@ -1,58 +1,72 @@
-# Verification report - RustProxy 0.24.10
+# Verification report - RustProxy 0.25.0
 
 ## Scope
 
-RustProxy 0.24.10 implements the Secret Store + password-encrypted full backup/restore and optional ERROR-only file logging specifications. Secret Store adds schema version 7, encrypts ACME DNS-provider credentials with a local master key, migrates legacy provider `.ini` files, and recreates plaintext credential files only around the external Certbot operation. Full backup/restore uses Argon2id plus authenticated AES-256-GCM encryption, an explicit manifest, Secret Store key wrapping, staged validation and atomic data-directory activation. Error-file logging is a separate, disabled-by-default asynchronous bounded sink with JSON/RAW formats, central secret redaction and external `logrotate` rotation.
-
-The release also retains the 0.24.8 overview cache diagnostics, 0.24.7 transform resource controls and 0.24.6 cache/background-worker memory hardening described by the earlier audit material below.
+RustProxy 0.25.0 adds a reusable Web Server source while preserving the existing reverse-proxy pipeline. Web Services are independent configuration resources; vhosts and custom locations only choose a source type and reference a Web Service ID. The current development tree provides static GET/HEAD streaming, directory/index behavior, validators, single ranges, document-root confinement, ordered `try_files`, Web Service rewrites, a bounded migration-oriented `.htaccess` compatibility layer, expanded MIME policy, per-extension/MIME Cache-Control rules, fixed source headers, precompressed sidecars, forced-download extensions and optional bounded FastCGI/PHP-FPM. The Web Service editor is organized into General, Routing, .htaccess, MIME, Cache & headers, Delivery, PHP-FPM and Security tabs, with one-shot presets for Static/SPA/PHP/WordPress/Drupal/Laravel/Symfony/Joomla/PrestaShop/Downloads. Existing 0.25 work also includes repeatable directory listing, Web Server vhost-wide metrics, Logging sub-tabs, corrected structured-log source validation, bounded per-vhost Traffic analytics with effective-client-IP rankings/donut charts, User-Agent/optional-country logging, stable Web Service rendering after fast refreshes, generic/customizable 403 handling and automatic RSS relief for reclaimable allocator/HOT-cache memory. The `/docs` surface separates the Markdown Reader from API/OpenAPI and documents the implemented PHP-FPM runtime.
 
 ## Source contracts reviewed
 
-- SQLite schema version 7 includes the generic encrypted `secret_store` table; stored secret values are AES-256-GCM ciphertext with per-record nonces and the application master key lives outside SQLite in a private file.
-- Cloudflare, OVH and RFC2136 ACME credentials are validated, imported into Secret Store, decrypted only when required, and legacy provider `.ini` files are removed after successful migration. Certbot credential files are temporary and removed in `finally` paths.
-- `rust-proxy backup create` writes a new `.rpb` file outside the data directory, derives an independent backup key with Argon2id, encrypts authenticated chunks with AES-256-GCM, excludes cache/log/runtime data and wraps the Secret Store master key inside the encrypted backup.
-- `rust-proxy backup restore` requires an interactive password, decrypts into private staging, validates manifest/checksums/schema/configuration/Secret Store decryptability before activation, then switches the data directory by rename with rollback on activation failure.
-- `error_log` is independent from normal access logging, defaults disabled, accepts only `ERROR`, supports JSON/RAW, uses a bounded queue and a single asynchronous writer, handles filesystem failures without terminating RustProxy, and relies on system `logrotate` rather than application rollover.
-- Error File stays in the existing revisioned `logging` section: its separate dashboard form and the access-log form both save a composite `{ logging, error_log }` value through `/api/config/logging`. A database-reopen regression verifies that `enabled`, path and format survive restart, the standalone `/api/config/error_log` route is intentionally absent, and historical access-only `/api/config/logging` payloads remain accepted for compatibility. The redundant Error Log help sentence is absent from the panel.
-- The stderr/journald path, access-log sinks and the optional error-file sink share secret-redaction helpers for authorization/provider/password/private-key material.
-- The KV secret-redaction expression uses a hash-delimited Rust raw string, avoiding the `E0762` unterminated-character-literal parser failure while preserving the same redaction character class.
-- The release checker preserves the existing dashboard contract while using bundled roots plus common system CA bundles (`SSL_CERT_FILE`, `CURL_CA_BUNDLE`, FreeBSD/OPNsense and Linux paths), follows at most five credential-free HTTP(S) redirects without HTTPS-to-HTTP downgrade, and reports network/TLS/HTTP/parser failures at `ERROR` level so the optional error-file sink receives them.
-- The overview Cache RAM block keeps separate histories for HOT allocated bytes, transient RAM, fill bytes, COLD queue bytes, COLD I/O bytes and COLD disk bytes; the disk series is never included in process RAM accounting.
-- HOT and transient histories are rendered against their runtime limits, while COLD storage is rendered against the configured COLD byte budget.
-- The long `memoryAccountingHint` remains translated and available on demand through the Process information control instead of consuming permanent dashboard height. Response time history is no longer collapsible and has the same compact information control explaining that it charts successive samples of the cumulative mean duration of completed HTTP exchanges.
-- Transform `preset`, `memory_budget_bytes` and `max_inflight` are part of the normal revisioned Settings document and therefore persist in SQLite; the reopen test covers them.
-- Existing configurations default to 128 MiB transform temporary memory and 32 concurrent attempts.
-- New transform attempts use atomic runtime byte/concurrency reservations and fail open with fixed `memory_pressure` / `concurrency_pressure` reasons rather than queueing full-body work.
-- Presets modify only resource limits; transform type switches and extension allow-lists remain independent.
-- Global transform representation/eligibility settings participate in cache compatibility identity. Resource-only preset/budget/concurrency fields do not.
-- OpenAPI and both language packs expose the new controls and preset states.
-- `GET /api/status` exposes `transform_resources` and `memory_accounting`; the latter keeps process RSS separate from HOT/transient cache accounting and treats transform bytes as an overlapping conservative reservation.
-- Cache runtime telemetry exposes COLD byte/object limits so the overview can show disk occupancy against configured capacity without mixing disk and RAM.
-- `GET /api/transform` exposes current process-wide reserved transform bytes and active slots alongside the existing counters.
-- 0.24.6 aggregate/decode bounds, allocation reductions and bounded cache/background workers remain in place.
-- The bounded transform-body collection path maps its boxed body error explicitly into `anyhow::Error`; source/release regressions reject the incompatible `.context(...)` form that was reported by an external compiler run.
-- A subsequent external Rust compile reached warning-only status and reported four lint warnings. The source now keeps the COLD transient read lease explicitly alive via `_read_lease`, names the RAII-only `TransientLease` field `_inner`, and removes the two dead CSS `last_char` assignments without lint suppression. Offline source/release regressions pin those shapes; a compiler-enabled rerun is still required to confirm a warning-clean build.
+- `Config.web_services` is a separate persisted collection; vhost/location `source` defaults to Reverse Proxy for old documents and Web Server stores only `web_service_id`.
+- Full configuration compilation validates Web Service IDs, absolute/existing document roots, index names, duplicate IDs and every active Web Service reference. A pure static vhost does not require a backend pool; mixed Web Server and Reverse Proxy locations do.
+- Web Server dispatch occurs after normal frontend routing, ACL and Basic/Tinyauth authorization and before backend selection/DNS/cache/upstream work. Static responses continue through the existing custom error-page, response-header/security, compression and transport pipeline.
+- Static files use streamed Tokio file bodies with explicit lengths rather than whole-file buffering.
+- Files support metadata-based ETag/Last-Modified, If-None-Match/If-Modified-Since, one Range/If-Range and 304/206/416 behavior.
+- Directory requests redirect to a slash path, then search the optional ordered index list (which may be empty), then optionally render an HTML-escaped and URL-encoded RustProxy-styled listing without exposing filesystem paths.
+- Root directory listing opens a fresh directory file description per request rather than a `dup` sharing the persistent root descriptor offset; a regression test requests the same listing twice. Nested directories independently apply index lookup first and listing fallback only when no index exists. Generated listings send `Cache-Control: no-store`.
+- A directory with no matching index and directory listing disabled returns the built-in generic 403 page (or the configured vhost/global 403 replacement in the outer response pass), not a Web Server implementation-detail string. HEAD keeps an empty body.
+- Web Services support up to 64 normalized MIME overrides, a validated fallback MIME header and an optional validated default `Cache-Control` header. Static files remain backed by the filesystem/OS page cache; existing vhost/location response-header rules run later and can override the source-level cache policy.
+- Web Services additionally support ordered `ext:`/`mime:` Cache-Control rules, validated fixed response headers, ordered `try_files`, regex internal/redirect rewrites, precompressed `.br`/`.gz` sidecars and attachment extensions. Protected framing/transport headers cannot be overridden by the source header map.
+- Optional `.htaccess` compatibility compiles a bounded root/subdirectory set at Web Service Apply time rather than parsing per request. It supports common RewriteRule/RewriteCond front controllers, DirectoryIndex/Indexes, AddType, Header, Expires, PHP AddHandler and simple Require all granted/denied. Unsupported access/handler scopes fail closed, `.ht*` control files are never served, and PHP AddHandler requires PHP-FPM.
+- Web Service presets are UI-only generators: they populate ordinary configuration fields and do not persist a hidden CMS mode. ID/name/document root and PHP-FPM endpoint/tuning remain operator-owned. Optional hardening is persisted explicitly and defaults off for compatibility.
+- `block_common_attacks` rejects TRACE/CONNECT/TRACK, sensitive dot/control/backup paths and a bounded set of high-confidence exploit probes; it also suppresses FastCGI `HTTP_PROXY` derivation from the inbound `Proxy` header. It does not claim generic WAF/SQLi/XSS filtering.
+- `strip_unnecessary_headers` removes common source/PHP fingerprint headers after Web Service and `.htaccess` response handling while leaving the later vhost/location response-header pipeline authoritative.
+- Optional PHP handling executes configured extensions through a bounded FastCGI connector over Unix socket or loopback TCP. PHP concurrency uses a fixed semaphore without an unbounded wait queue, request/response bodies use bounded streaming, and connection failure cannot fall back to serving PHP source. Advanced configuration adds Authorization forwarding control, a streaming request-body ceiling, bounded CGI response headers/stderr, trusted custom FastCGI parameters and validated `PHP_VALUE` overrides that cannot replace core CGI/HTTP fields or `PHP_ADMIN_VALUE`.
+- Web Service symlink policy defaults to `deny`/`O_NOFOLLOW`; Linux-only `inside_root` verifies opened descriptors through `/proc/self/fd` against the canonical root. PHP-FPM is intentionally limited to `deny` because the external worker reopens `SCRIPT_FILENAME`.
+- Routing rejects malformed/ambiguous encoded paths before filesystem lookup. Unix filesystem lookup walks from an opened root descriptor; `deny` uses `openat` + `O_NOFOLLOW`, while Linux-only `inside_root` verifies every opened descriptor against the canonical root. PHP-FPM requires `deny`. The secure Web Server compile path remains Unix-only.
+- No second URL router, custom-error subsystem or object cache was introduced. Reverse Proxy remains the default source for existing configuration.
+- Dashboard CRUD, source selectors, English/Polish strings and OpenAPI schemas expose the new model.
+- Web Service index files are optional (0..16). An empty list skips index lookup, allowing directory-listing-only services without synthetic index names.
+- Generated directory listings use a self-contained RustProxy panel-style layout (matching palette/typography, responsive light/dark treatment and RustProxy footer) while preserving HTML escaping, URL encoding and filesystem-path secrecy.
+- Pure Web Server vhosts no longer present backend diagnostics as though an origin pool existed; their Backends summary identifies `Web Server`, while mixed static/proxy vhosts retain backend diagnostics.
+- Status payloads expose fixed-cardinality per-vhost metrics independently of backend pools. The Vhosts dashboard uses those counters for Web Server-only request/active/error values, EWMA latency and byte-rate traffic while retaining backend health details for reverse-proxy sources.
+- Logging settings are split into Outputs, JSON format and Error log sub-tabs inside one card, matching the Cache subsection layout. `route`, `cache_effective_enabled` and `transform_effective_enabled` are accepted by the same backend validator that saves custom JSON mappings, and the regression test validates then renders those fields.
+- Structured logging now exposes `user_agent` and `country`. User-Agent capture defaults on; country capture defaults off and GeoIP resolution is skipped unless capture or an existing country ACL/rate-limit needs it.
+- The authenticated Traffic workspace uses a bounded 2048-sample non-blocking queue, at most 64 vhost buckets and 32-entry approximate top tables for countries, User-Agents, effective client IPs, paths, routes, methods, status classes and cache outcomes. It never writes per-request analytics to SQLite. Countries, User-Agents, client IPs, methods, status classes and cache outcomes also render bounded donut summaries from the same in-memory TOP state.
+- Sustained-traffic memory maintenance calls `malloc_trim(0)` every 60 seconds on Linux/glibc to release already-free allocator pages and trims HOT objects only when above the configured waterline and idle past the configured age. It does not periodically reload vhosts, purge active objects or reset bounded upstream keep-alive pools.
+- Cache compatibility currently includes vhost configuration, so a cache-affecting vhost save can purge cache during snapshot publication; replacing the compiled snapshot also retires its upstream clients after outstanding references drain. These mechanisms explain why a manual vhost save can visibly reduce RSS without implying that reloads are required for steady-state operation.
+- The Web Server list now uses the same vhost card/detail-cell visual structure as the Vhosts list. Reverse Proxy vhost creation reports a dedicated “at least one backend” validation error instead of the generic paths/ACL/headers JSON error.
+- Web Service cards use only the shared `el(...)` DOM helper; a regression contract rejects the previous undefined `text(...)` call that aborted list rendering. Authenticated-shell startup now verifies `/api/auth/session` first and never reveals the legacy embedded sign-in form for non-authentication JavaScript/API failures, preventing F5 from falsely presenting the user as logged out.
+- The Web Server UI module performs a catch-up render when it registers after `app.js`; this closes the race where a very fast `/api/config` response could finish before `web-server.js` existed and leave the Web Service list blank until a later render.
+- The Web Service editor now builds its configuration tabs with the same `vh-tabs` + `quiet` button pattern and active/ARIA state handling as the Vhost editor; the source contract also checks this shared modal-tab pattern and invalid-field navigation switches to the owning tab.
+- Static dispatch clones only `Method`, `HeaderMap` and `Uri` before awaiting the filesystem source; it does not hold `&Request<Body>` across an await, keeping the Axum 0.8 fallback handler future `Send`. This addresses the reported `Handler<_, _>` / E0277 failure at `.fallback(forward)`.
+- `/docs` is an authenticated documentation surface with a Markdown Reader, searchable embedded docs bundle and a separate API/OpenAPI tab. Reader Markdown files are direct `include_str!` compiler inputs and no longer depend on a generated `OUT_DIR/docs-content.json`, preventing clean/incremental builds from failing on a missing documentation artifact.
+- Documentation cleanup keeps README and the active ROADMAP current-only; operator release history, developer-level changelog history and completed roadmap milestones are separated into CHANGELOG/audit/archive documents.
+
+## Implemented automated coverage
+
+Rust unit tests in `src/web_server.rs` cover GET/HEAD/index/slash redirect, MIME/length, validators, valid/invalid range handling, If-Range, escaped/repeated directory listings, per-directory index-before-listing fallback, generic 403/HEAD behavior, MIME overrides/default MIME, cache rules, source headers, `try_files`, rewrite redirect/internal behavior, `.htaccess` WordPress-style front-controller routing, nested DirectoryIndex/MIME/Header/Expires behavior, strict-mode rejection and PHP-source protection, hardening probe/method blocking and fingerprint-header stripping, precompressed representations/ranges, download headers, PHP FastCGI fixture execution/failure-source-disclosure behavior, request-body 413 enforcement, custom FastCGI/PHP_VALUE validation and reserved-parameter rejection, direct traversal rejection and Unix/Linux symlink policy. `src/traffic.rs` includes bounded-table and effective-client-IP view coverage. Configuration tests cover pure static vhosts without a pool, mixed Web Server/Reverse Proxy routing, reuse of one Web Service by multiple vhosts, unknown service references and backward-compatible default source behavior. ACL tests include encoded traversal/separator/malformed-encoding cases.
+
+Rust-independent release checks additionally validate UI/JSON/language/source contracts and reproducible packaging.
 
 ## Verification status in this workspace
 
 | Check | Status |
 |---|---|
-| `python3 scripts/preflight.py` | PASS |
 | `python3 scripts/langs_check.py` | PASS |
+| `python3 scripts/docs_check.py` | PASS |
 | `python3 scripts/memory_contracts.py` | PASS |
-| Browser JavaScript `node --check` | PASS for all administration JS bundles |
+| `python3 scripts/web_server_contracts.py` | PASS |
+| Browser JavaScript `node --check` | PASS for app, Settings, Docs Reader, Web Server, Traffic and vhost administration bundles |
 | OpenAPI/language JSON parse | PASS |
-| Shell `bash -n` | PASS |
-| Python syntax / bytecode compile, including ACME provider worker | PASS |
-| Secret Store / encrypted backup / error-log / portable update-check source-contract regression checks | PASS |
-| SQLite schema 7 migration smoke (`secret_store`) | PASS |
-| Docker agent `go test ./...` / `go vet ./...` / `go build ./...` | PASS |
+| Shell `bash -n` | PASS in source/release checks |
+| `go test ./...` in `docker-agent` | PASS |
+| Python source parsing/contract checks | PASS |
 | `python3 scripts/check_source.py` | PASS |
 | `python3 scripts/release_smoke.py` | PASS |
-| FreeBSD/OPNsense live release-check execution | NOT RUN in this workspace; source contract and platform CA paths are covered, target runtime confirmation is still required |
-| `./dev.sh fmt` / `cargo fmt --all` | BLOCKED: Rust/Cargo toolchain unavailable in this workspace |
-| `./dev.sh check` / `cargo check --locked --all-targets` | BLOCKED: Rust/Cargo toolchain unavailable in this workspace |
-| `./dev.sh test` Rust phase | BLOCKED: Rust/Cargo toolchain unavailable in this workspace |
-| Sustained per-preset transform RSS/heap profiling | release-host acceptance gate |
+| `./dev.sh test` Rust-independent phases | PASS; Rust phase BLOCKED because Cargo is unavailable |
+| `cargo fmt --all` / `./dev.sh fmt` | BLOCKED: Rust/Cargo toolchain unavailable in this workspace |
+| `cargo check --locked --all-targets` / `./dev.sh check` | BLOCKED: Rust/Cargo toolchain unavailable in this workspace |
+| `cargo test --locked --all-targets` / Rust phase of `./dev.sh test` | BLOCKED: Rust/Cargo toolchain unavailable in this workspace |
+| Live H1/H2/H3 Web Server integration and large-file acceptance | release-host acceptance gate |
 
-See [V0.24.8_AUDIT.md](V0.24.8_AUDIT.md) for the dashboard change and [V0.24.7_AUDIT.md](V0.24.7_AUDIT.md) for the retained transform resource-control design and runtime acceptance gate.
+The source package therefore has static/source-contract verification here, but compiler-enabled and live transport acceptance must still be run on the release host before production release.