v0.25.6

docs/API.md

@@ -226,8 +226,8 @@ endpoint or separate unsynchronized frontend/backend write API.
 includes `id`, `algorithm`, `ready`, `health_mode`, `sticky_mode`, `keepalive`,
 `members`. Members expose ID/origin/weight, status, healthy flag, active exchanges,
 request/error/byte totals, EWMA byte rate/header latency, last check/result, failure
-and success counters, transition count and last upstream status. It does not expose
-affinity tokens. Member status is up/down/checking/draining/disabled. In health-off
+and success counters, transition count, last upstream status and upstream-client
+generation/request/age/rotation counters. It does not expose affinity tokens. Member status is up/down/checking/draining/disabled. In health-off
 mode, eligibility is not proof that the origin is live. Draining describes operator
 intent; inspect `healthy` as well. `ready` reflects new-assignment availability;
 a pool containing only draining members may still serve existing cookie sessions.

docs/API_ACCESS.md

@@ -1,4 +1,4 @@
-# Administration API access - 0.25.5
+# Administration API access - 0.25.6
 
 ## Sign-in boundary
 

docs/CACHE_DIAGNOSTICS.md

@@ -167,7 +167,7 @@ The 0.25.0 development runtime adds two maintenance mechanisms that do not reloa
 - every 60 seconds on Linux/glibc, RustProxy calls `malloc_trim(0)` from a blocking maintenance task; this can return allocator pages that are already free, but cannot and does not evict live Rust objects;
 - HOT cache maintenance trims only least-recently-used entries that have been idle for the configured delay and only while HOT allocation is above the configured target waterline. The pass is bounded by `hot_eviction_max_objects`; recent/active entries are not purged.
 
-Upstream HTTP keep-alive is not reset on a timer. It is already bounded per backend by `max_idle_per_host` and `idle_timeout_seconds`, so forcing periodic client/snapshot replacement would trade memory symptoms for connection churn and lower proxy efficiency. If RSS remains high while HOT/transient usage and connection-pool bounds are stable, the remaining resident pages may still be active runtime/TLS/QUIC/Hyper/Tokio working set rather than reclaimable cache memory.
+Upstream HTTP clients now also have bounded generations. Each backend Hyper client is recycled after 25,000 requests or 3,600 seconds by default (both limits are configurable under vhost keepalive and either can be disabled with `0`). Recycling replaces only the backend client/pool and leaves the vhost snapshot, HOT cache and configuration untouched. HTTP/1 upstream read buffers are capped at 128 KiB by default. This bounds long-lived Hyper/TLS/pool high-water allocations while preserving normal keep-alive reuse inside each generation. The existing 60-second allocator trim can then return already-free glibc pages to the OS.
 
 ## Hot and cold tiers
 

docs/INSTALL.md

@@ -1,4 +1,4 @@
-# Installing and updating RustProxy 0.25.5
+# Installing and updating RustProxy 0.25.6
 
 The scripts prepare a Linux/systemd installation. This source archive is **not a
 verified production build**: Rust/Cargo, real CA operations, live systemd startup

docs/MIGRATION.md

@@ -1,5 +1,9 @@
 # Deployment and configuration migration history
 
+## 0.25.5 -> 0.25.6
+
+There is no SQLite schema or saved-configuration migration. Version 0.25.6 adds the already-exposed upstream client generation/rotation diagnostics to the live Backends table and keeps the 0.25.5 backend client recycling defaults unchanged.
+
 ## 0.25.4 -> 0.25.5
 
 There is no SQLite schema or saved-configuration migration. Existing `cloudflare_client_ip` vhost settings remain opt-in. When enabled and the direct peer is a verified Cloudflare edge, RustProxy now prefers a valid `CF-IPCountry` value for country analytics/logging and country policy evaluation, falling back to the installed GeoIP database when that header is unavailable or invalid.

docs/TEST_REPORT.md

@@ -1,8 +1,8 @@
-# Verification report - RustProxy 0.25.5
+# Verification report - RustProxy 0.25.6
 
 ## Scope
 
-RustProxy 0.25.5 is a patch release for dashboard consistency and observability: Revision History used a separate rounded modal style, Cloudflare-backed vhosts resolved Traffic country from local GeoIP without consulting Cloudflare's verified country header, Prometheus did not expose the bounded Traffic dimensions or an explicit built-in Web Server serving-source view, the mobile dashboard rendered the full section list permanently across the top of the page, and the dashboard lacked a compact About surface. The top bar now exposes project About information and uses the full Documentation label. The patch keeps SQLite schema 8 and the existing configuration format.
+RustProxy 0.25.6 is a patch release that keeps the bounded upstream-client memory controls from 0.25.5 and makes their runtime state visible in the main Backends diagnostics table. Each row now shows client generation, requests in the current generation, generation age and total rotations. Those values come from the same backend-pool status object delivered through `/api/events` and `/api/status`, so the dashboard refreshes them with the existing live diagnostics path. The defaults remain 25,000 requests, 3,600 seconds and a 131,072-byte upstream HTTP/1 adaptive read-buffer ceiling. Recycling remains local to the backend client and does not reload the vhost or purge cache. SQLite remains schema 8 and no saved-configuration migration is introduced.
 
 ## Source contracts reviewed
 
@@ -46,7 +46,8 @@ RustProxy 0.25.5 is a patch release for dashboard consistency and observability:
 - Logging settings are split into Outputs, JSON format and Error log sub-tabs inside one card, matching the Cache subsection layout. `route`, `cache_effective_enabled` and `transform_effective_enabled` are accepted by the same backend validator that saves custom JSON mappings, and the regression test validates then renders those fields.
 - Structured logging exposes `user_agent` and `country`. User-Agent capture defaults on; country capture defaults off. Country resolution is skipped unless capture or an existing country ACL/rate-limit needs it; verified Cloudflare mode prefers trusted `CF-IPCountry`, otherwise local GeoIP is used.
 - The authenticated Traffic workspace uses a bounded 2048-sample non-blocking queue, at most 64 vhost buckets and 32-entry approximate top tables for countries, User-Agents, effective client IPs, paths, routes, methods, status classes and cache outcomes. It never writes per-request analytics to SQLite. The Traffic UI is split into Overview, Clients, Requests and Delivery sub-tabs. Countries/methods/status/cache use donut summaries; User-Agent, effective client IP, path, route and request/response byte volume use bounded horizontal bar charts from the same in-memory TOP state.
-- Sustained-traffic memory maintenance calls `malloc_trim(0)` every 60 seconds on Linux/glibc to release already-free allocator pages and trims HOT objects only when above the configured waterline and idle past the configured age. It does not periodically reload vhosts, purge active objects or reset bounded upstream keep-alive pools.
+- Sustained-traffic memory maintenance calls `malloc_trim(0)` every 60 seconds on Linux/glibc to release already-free allocator pages and trims HOT objects only when above the configured waterline and idle past the configured age. Backend Hyper clients additionally recycle their connection-pool generation after 25,000 requests or 3,600 seconds by default; both triggers are configurable/disableable per vhost, and HTTP/1 upstream adaptive read buffers default to a 128 KiB ceiling. Recycling does not publish a new vhost snapshot or purge cache.
+- The live Backends table renders `upstream_client_generation`, `upstream_client_generation_requests`, `upstream_client_generation_age_seconds` and `upstream_client_rotations_total` in the same row as Active/Requests/Errors/Latency/Traffic. `app/core.js` feeds the table from the existing `/api/events` `snapshot.status` payload, so the new fields do not introduce another polling loop or control-plane query.
 - Cache compatibility currently includes vhost configuration, so a cache-affecting vhost save can purge cache during snapshot publication; replacing the compiled snapshot also retires its upstream clients after outstanding references drain. These mechanisms explain why a manual vhost save can visibly reduce RSS without implying that reloads are required for steady-state operation.
 - The Web Server list now uses the same vhost card/detail-cell visual structure as the Vhosts list. Reverse Proxy vhost creation reports a dedicated “at least one backend” validation error instead of the generic paths/ACL/headers JSON error.
 - Web Service cards use only the shared `el(...)` DOM helper; a regression contract rejects the previous undefined `text(...)` call that aborted list rendering. Authenticated-shell startup now verifies `/api/auth/session` first and never reveals the legacy embedded sign-in form for non-authentication JavaScript/API failures, preventing F5 from falsely presenting the user as logged out.
@@ -58,41 +59,47 @@ RustProxy 0.25.5 is a patch release for dashboard consistency and observability:
 
 ## Implemented automated coverage
 
-Rust unit tests in `src/web_server/tests.rs` cover GET/HEAD/index/slash redirect, MIME/length, validators, valid/invalid range handling, If-Range, escaped/repeated directory listings, per-directory index-before-listing fallback, generic 403/HEAD behavior, MIME overrides/default MIME, cache rules, source headers, `try_files`, rewrite redirect/internal behavior, `.htaccess` WordPress-style front-controller routing, nested DirectoryIndex/MIME/Header/Expires behavior, strict-mode rejection and PHP-source protection, hardening probe/method blocking and fingerprint-header stripping, precompressed representations/ranges, download headers, PHP FastCGI fixture execution/failure-source-disclosure behavior, request-body 413 enforcement, custom FastCGI/PHP_VALUE validation and reserved-parameter rejection, direct traversal rejection and Unix/Linux symlink policy. `src/traffic/tests.rs` includes bounded-table, effective-client-IP view and fixed Prometheus User-Agent/method family coverage. Configuration tests cover pure static vhosts without a pool, mixed Web Server/Reverse Proxy routing, reuse of one Web Service by multiple vhosts, unknown service references and backward-compatible default source behavior. Vhost tests cover method-policy normalization/allow/block behavior, database tests cover retained configuration revision documents, actor metadata and previous-revision lookup, and admin tests cover restoring revision 1 as revision 3 while preserving the previous document and emitting `config.restore`. ACL tests include encoded traversal/separator/malformed-encoding cases.
+Rust unit tests in `src/web_server/tests.rs` cover GET/HEAD/index/slash redirect, MIME/length, validators, valid/invalid range handling, If-Range, escaped/repeated directory listings, per-directory index-before-listing fallback, generic 403/HEAD behavior, MIME overrides/default MIME, cache rules, source headers, `try_files`, rewrite redirect/internal behavior, `.htaccess` WordPress-style front-controller routing, nested DirectoryIndex/MIME/Header/Expires behavior, strict-mode rejection and PHP-source protection, hardening probe/method blocking and fingerprint-header stripping, precompressed representations/ranges, download headers, PHP FastCGI fixture execution/failure-source-disclosure behavior, request-body 413 enforcement, custom FastCGI/PHP_VALUE validation and reserved-parameter rejection, direct traversal rejection and Unix/Linux symlink policy. Upstream tests cover request-budget client generation recycling and disabled recycling; balancing tests cover the new keepalive defaults and validation bounds. `src/traffic/tests.rs` includes bounded-table, effective-client-IP view and fixed Prometheus User-Agent/method family coverage. Configuration tests cover pure static vhosts without a pool, mixed Web Server/Reverse Proxy routing, reuse of one Web Service by multiple vhosts, unknown service references and backward-compatible default source behavior. Vhost tests cover method-policy normalization/allow/block behavior, database tests cover retained configuration revision documents, actor metadata and previous-revision lookup, and admin tests cover restoring revision 1 as revision 3 while preserving the previous document and emitting `config.restore`. ACL tests include encoded traversal/separator/malformed-encoding cases.
 
 Rust-independent release checks additionally validate UI/JSON/language/source contracts and reproducible packaging.
 
-During the prior 0.25.4 compiler gate, an external build exposed one module-privacy error in the history split (`CONFIG_HISTORY_CAPACITY` was private to `db::history`) and one unused `country_resolved` assignment in the location GeoIP branch. Both were corrected before the 0.25.5 baseline: the capacity is exposed to its parent module with `pub(super)`, referenced through `history::CONFIG_HISTORY_CAPACITY`, and the dead assignment is removed. This workspace still has no Rust/Cargo toolchain, so compiler confirmation of the 0.25.5 tree remains a release-host gate.
+During the prior 0.25.4 compiler gate, an external build exposed one module-privacy error in the history split (`CONFIG_HISTORY_CAPACITY` was private to `db::history`) and one unused `country_resolved` assignment in the location GeoIP branch. Both were corrected before the 0.25.5 baseline: the capacity is exposed to its parent module with `pub(super)`, referenced through `history::CONFIG_HISTORY_CAPACITY`, and the dead assignment is removed. This workspace still has no Rust/Cargo toolchain, so compiler confirmation of the 0.25.6 tree remains a release-host gate.
 
 ### Modular source-tree verification
 
-- `python3 scripts/rust_module_contracts.py`: PASS. Every repository `.rs` file is at most 500 lines.
-- `python3 scripts/source_module_contracts.py`: PASS. Dashboard JavaScript, Docker Agent Go implementation and operational deploy/NPM-migration Python modules are at most 500 lines; Docker tests remain in `_test.go`, and browser split boundaries/public bundles parse with Node when available.
-- Oversized `app.js`/`vhosts.js` source is split while the public `/assets/app.js` and `/assets/vhosts.js` URLs remain stable; the Docker Agent is split into model/Docker/network/certificate/server/entry-point files.
-- Rust unit-test bodies are isolated under `tests.rs` / `tests/`; implementation files contain only guarded test includes/declarations.
-- All `include!("*.rs")` targets resolve, split fragments have balanced delimiters, and no fragment ends in a dangling Rust attribute or doc comment.
-- Regression guards cover the Web Server split errors previously observed around `#[derive]`, `htaccess`, `server_protocol` and `anyhow::Context`.
-- This is a compiler-independent structural check; Cargo parsing/type checking remains a release-host requirement when the Rust toolchain is unavailable in the packaging environment.
+- The new upstream client implementation is split into `src/upstream/client.rs` with test bodies in `src/upstream/client_tests.rs`; both stay below the 500-line module limit, and `src/upstream.rs` remains below that limit after moving client construction out.
+- `python3 scripts/source_module_contracts.py`: PASS. Dashboard JavaScript, Docker Agent Go implementation and operational deploy/NPM-migration Python modules remain within their source-size contracts.
+- `python3 scripts/rust_module_contracts.py` is BLOCKED by a pre-existing source-tree issue in the supplied 0.25.5 baseline archive: `src/admin/documentation.rs` is 554 lines while the contract limit is 500. The same failure reproduces on the unmodified 0.25.5 baseline archive.
+- `python3 scripts/docs_check.py` is also BLOCKED by a pre-existing supplied-archive issue: the Reader does not embed `docs/ACME_PROVIDERS.md`. This also reproduces on the unmodified input archive.
+- Rust unit-test bodies added for this change are isolated in a test include file; the implementation module contains only the guarded include declaration.
+- Cargo parsing/type checking remains a release-host requirement because this workspace has no Rust toolchain.
 
 ## Verification status in this workspace
 
 | Check | Status |
 |---|---|
 | `python3 scripts/langs_check.py` | PASS |
-| `python3 scripts/docs_check.py` | PASS |
-| `python3 scripts/memory_contracts.py` | PASS |
+| `node --check src/admin/web/vhosts/render.js` | PASS |
+| Backend live-table source contract | PASS: four client-generation diagnostics are rendered from backend member status fields |
+| `python3 scripts/memory_contracts.py` | PASS, including upstream generation-recycle/buffer-cap source contracts |
 | `python3 scripts/web_server_contracts.py` | PASS |
-| Browser JavaScript `node --check` | PASS for split app source, vhost split boundaries/public bundle, Settings, Docs Reader, Web Server and Traffic |
+| `python3 scripts/security_contracts.py` | PASS |
+| `python3 scripts/security_dependency_contracts.py` | PASS |
+| `python3 scripts/source_module_contracts.py` | PASS |
+| `node --check src/admin/web/vhosts/editor.js` | PASS |
+| `node scripts/web_service_ui_test.mjs` | PASS |
 | OpenAPI/language JSON parse | PASS |
-| Shell `bash -n` | PASS in source/release checks |
 | `go test ./...` in `docker-agent` | PASS |
-| Python source parsing/contract checks | PASS |
-| `python3 scripts/check_source.py` | PASS |
-| `python3 scripts/release_smoke.py` | PASS |
-| `./dev.sh test` Rust-independent phases | PASS; Rust phase BLOCKED because Cargo is unavailable |
+| SQL migration/backup smoke (`release_smoke.sql_and_backup_smoke`) | PASS |
+| Stopped-source upgrade smoke (`release_smoke.upgrade_smoke`) | PASS |
+| `python3 scripts/rust_module_contracts.py` | BLOCKED by pre-existing `src/admin/documentation.rs` 554-line baseline failure |
+| `python3 scripts/docs_check.py` | BLOCKED by pre-existing missing `docs/ACME_PROVIDERS.md` Reader embed |
+| `python3 scripts/check_source.py` | BLOCKED because it invokes the same pre-existing Rust module-size gate |
+| `python3 scripts/release_smoke.py` | BLOCKED because it invokes the same pre-existing Rust module-size gate; its SQL/upgrade smoke functions pass when run directly |
+| `./dev.sh test` | BLOCKED at the same pre-existing Rust module-size gate before reaching Cargo |
 | `cargo fmt --all` / `./dev.sh fmt` | BLOCKED: Rust/Cargo toolchain unavailable in this workspace |
 | `cargo check --locked --all-targets` / `./dev.sh check` | BLOCKED: Rust/Cargo toolchain unavailable in this workspace |
-| `cargo test --locked --all-targets` / Rust phase of `./dev.sh test` | BLOCKED: Rust/Cargo toolchain unavailable in this workspace |
-| Live H1/H2/H3 Web Server integration and large-file acceptance | release-host acceptance gate |
+| `cargo test --locked --all-targets` | BLOCKED: Rust/Cargo toolchain unavailable in this workspace |
+| Live sustained HTTPS/HTTP/1.1 RSS acceptance | release-host acceptance gate; verify RSS plateau and `upstream_client_rotations_total` under representative traffic |
 
-The source package therefore has static/source-contract verification here, but compiler-enabled and live transport acceptance must still be run on the release host before production release.
+The new source-level memory bounds and UI/config contracts pass the available targeted checks. Compiler-enabled and sustained-traffic runtime acceptance remain required before production release; two unrelated static gates are already failing in the supplied 0.25.5 baseline as described above.

docs/TRANSPORT.md

@@ -204,8 +204,12 @@ requires `HTTP2-Settings`, and becomes HTTP/2 stream 1 after `101`. Optional dow
 remains a separate QUIC listener capability (see HTTP3.md).
 
 Pool keepalive settings configure idle connection reuse, idle timeout/limit, TCP
-keepalive and HTTP/2 active-connection pings. Health probes are separate no-reuse
-HTTP/1.1 or raw TCP operations; their concurrency cap is 16. Existing active exchanges
-are not killed by disabling reuse or draining a member. Config publication rebuilds
-client pools while preserving compatible observations and existing exchange leases.
-The full contract and bounds are in VHOSTS.md. Keepalived/VRRP is not involved.
+keepalive and HTTP/2 active-connection pings. They also bound each backend Hyper client
+generation by request count/age and cap the upstream HTTP/1 adaptive read buffer. Client
+generation recycling is local to the backend pool and does not publish a new vhost
+snapshot or purge cache. Health probes are separate no-reuse HTTP/1.1 or raw TCP
+operations; their concurrency cap is 16. Existing active exchanges are not intentionally
+killed by disabling reuse, recycling a generation or draining a member. Config
+publication still rebuilds client pools while preserving compatible observations and
+existing exchange leases. The full contract and bounds are in VHOSTS.md. Keepalived/VRRP
+is not involved.

docs/VHOSTS.md

@@ -194,13 +194,21 @@ origin do not expose a selected backend or stale origin timing.
 | `idle_timeout_seconds` | 30 | 1..3600 |
 | `max_idle_per_host` | 4 | 1..64 |
 | `tcp_keepalive_seconds` | 30 | 1..3600 |
+| `pool_recycle_requests` | 25000 | 0 (off) or 1000..10000000 |
+| `pool_recycle_seconds` | 3600 | 0 (off) or 60..604800 |
+| `http1_max_buffer_bytes` | 131072 | 8192..1048576 |
 
 These configure HTTP idle reuse and TCP keepalive; HTTP/2 also uses pings for active
 connections, with a 10-second ping timeout and no pinging solely to keep idle
-connections alive. Disabling reuse sets the idle pool limit to zero and disables
-those keepalive probes. It does not abort already active requests/tunnels. The idle
-limit is not a separate per-backend active-request cap; global connection/exchange
-bounds still apply. HTTP health probes use separate no-reuse HTTP/1.1 connections.
+connections alive. `pool_recycle_requests` and `pool_recycle_seconds` bound the lifetime
+of each backend Hyper client generation. Recycling swaps only that backend client/pool;
+it does not reload the vhost, purge cache or intentionally abort active exchanges. The
+request counter is per backend client generation, not per TCP connection. Setting either
+recycle value to `0` disables that trigger independently. `http1_max_buffer_bytes` bounds
+Hyper's adaptive upstream HTTP/1 read buffer per connection; it does not apply to HTTP/2.
+Disabling reuse sets the idle pool limit to zero and disables keepalive probes. The idle
+limit is not a separate per-backend active-request cap; global connection/exchange bounds
+still apply. HTTP health probes use separate no-reuse HTTP/1.1 connections.
 
 `http2` over HTTPS prefers h2 but keeps HTTP/1.1 ALPN compatibility. Over plaintext,
 `http2` remains explicit prior-knowledge h2c with no retry-based downgrade. `auto`