v0.34.0_2

docs/PROXY_1KB_HOTPATH_DEV_0.34.0.md

@@ -0,0 +1,137 @@
+# RustProxy 0.34.0 DEV: 1 KiB reverse-proxy throughput audit
+
+## Goal and evidence boundaries
+
+Priority 1 is short (~1 KiB) HTTP reverse-proxy responses: aim for approximately
+100,000 requests/s, and eventually at least 80% of nginx under the **same**
+controlled benchmark, without disabling authentication, ACLs, caching semantics,
+rate limits, traffic accounting, request IDs, logs, body streaming or protocol
+support. Priority 2 is local WWW serving, to be tuned and measured separately.
+This is a development revision; Cargo/package version remains **0.34.0**.
+
+**This source is not compiled in the current environment.** Neither `cargo`
+nor `rustc` is available. Only the user's pre-existing executable (SHA-256
+`453757748455d7b9b50a6140e322ee4d4cf4a030eea29318686f2e28c594b644`)
+was run. Live configuration A/B tests therefore substantiate a **configuration
+bottleneck**, not the throughput of this edited source.
+
+## Real baseline experiment (existing executable only)
+
+Disposable loopback nginx upstream, 1,040-byte body, HTTP/1.1 persistent
+frontend connections, caching disabled, identity response encoding, four Tokio
+workers, ApacheBench (`ab -k -n 12000`), 16/64 concurrent frontend clients.
+Checks outside the timed intervals verified HTTP 200, exact 1,040 bytes and
+SHA-256 `62a08d75cdde0416c883d72458385b5e56ae8c14466934ad784587ddb93a548b`;
+0 failed requests in the shown samples. The host has shared CPU resources;
+results vary and are not a pinned-CPU or production acceptance benchmark.
+
+| Existing executable configuration | 16 clients, RPS | 64 clients, RPS |
+|---|---:|---:|
+| max_idle_per_host = 4 | 18,226 | 17,935 |
+| max_idle_per_host = 64 | 30,906 | 24,872 |
+| Relative change (configuration only) | +69.6% | +38.7% |
+
+A separate repeat experiment at 64 clients (two runs per variant, in order)
+produced: `idle4` 14,028 / 14,834; `idle16` 22,205 / 24,711;
+`idle32` 25,580 / 29,088; `idle64` 36,521 / 33,847;
+`idle64 + recycle 250,000` 31,168 / 29,753;
+`idle32_again` 34,798 / 29,194; `idle64_again` 34,677 / 29,965;
+`idle4_again` 23,503 / 23,514 RPS. The order drift warns against
+reporting a single percentage as a robust speedup. Raising generation
+recycling from 25,000 to 250,000 did not establish any improvement; its
+original limit is retained. All shown variants reported zero HTTP failures.
+
+A preliminary nginx comparison gave ~75k/117k RPS at 16/64 clients, but later
+nginx samples were ~50k/93k on the shared host. Do not compute a definitive
+"% nginx" from these non-isolated runs. The 100k/80% goals remain **OPEN**.
+
+## Source modifications
+
+| Location | Change | Preserved behavior / caveat |
+|---|---|---|
+| `src/balance/model.rs` | New vhost default `max_idle_per_host`: **4 -> 64** | Configured values **1..64** remain validated; existing explicitly saved `4` stays `4`. Disabled keepalive, 30 s idle timeout, 25k-request and 3,600 s recycling rules are unchanged. Increased idle sockets use more file descriptors and memory. |
+| `src/admin/web/vhosts/editor.js` | Match the new `64` default when the field is absent | Existing values, limits and all editor settings remain unchanged. |
+| `src/proxy/body.rs` | At `Exchange::drop`, use exclusive `Mutex::get_mut()` rather than lock/unlock; move owned `route`, `client_ip`, `country`, `upstream_error` into final observation where possible | Exactly the same log fields, analytics sample, counters, permit lifetime, errors and websocket/H3 fields. Analytics still gets a clone when a vhost sample is queued; history still receives the original. |
+| `src/proxy/headers.rs` | Skip reserved cookie sanitization immediately if the corresponding `Cookie`/`Set-Cookie` header is absent | Full sanitization, admin-cookie separation and restoration restrictions still apply when the header is present. |
+| `src/balance/tests.rs`, `src/proxy/body/tests.rs`, `src/proxy/headers/tests.rs` | Add compatibility, completion and cookie regression assertions | These Rust tests **require** a compiler before they count as run. |
+| `scripts/tests/*hotpath*` | Update source-only guards for the new **bounded** default | Static assertions do not prove functional or runtime performance equivalence. |
+
+**Persistence note:** old vhosts may store `keepalive.max_idle_per_host=4`
+explicitly; the new code default does not rewrite them. To apply the measured
+configuration improvement before rebuilding, change that option to `64` in the
+relevant vhost and verify actual idle socket/FD usage. Where hundreds of origins
+share a host, test `16` or `32` if the FD/memory trade-off is preferable.
+
+## Bottlenecks to profile next, not silently disable
+
+1. `src/cache/dashboard.rs::Cache::record`: obtains a **shared cache-store
+   mutex on every outcome**, including `BYPASS` when cache is disabled through
+   `src/proxy/forward/cache_lookup.rs`. This may compete with HOT cache
+   operations and admin dashboard reads; profile `pthread_mutex`/futex and
+   lock hold times. Candidate: separate bounded, sharded domain/reason
+   counters from the HOT object store while preserving exact dashboard totals,
+   reason labels and reset semantics.
+2. `src/logging/format.rs::Logs::push`: per-request bounded history
+   `try_lock`, potentially causing recorded log drops under high concurrency.
+   Preserve *existing* drops/accounting, live subscribers and external sinks.
+   Candidate: worker-local bounded batches or sharded history with deterministic
+   bounded merge; validate log order, recent history, retention and drops.
+3. `src/proxy/body.rs::Exchange::drop` + `src/traffic.rs::Analytics::push`:
+   request-sized log strings, sampling queue sends, atomics and lifecycle
+   observations. Profile allocations and CPU per completed request. Candidate:
+   compact shared immutable identity fields or conditional cloning solely
+   where consumers require ownership, without removing logs or analytics.
+4. `src/proxy/forward/route_cache.rs`: request-target path copies,
+   header policy and several route/cache eligibility steps. Measure with
+   sampling `perf record -g` before extracting additional fast paths; preserve
+   query handling, canonical host, ACL, authentication, retries and WebSockets.
+5. `src/upstream/*`: profile H1 pool misses, TCP connect rate, Hyper
+   connection reuse and generation swaps. Increasing recycle thresholds or
+   changing HTTP/2/H2C modes without an isolated performance case is not
+   recommended.
+6. **Separate WWW priority:** benchmark short static objects independently;
+   existing `sendfile: auto` applies to eligible cleartext H1 objects >=64 KiB
+   and cannot accelerate 1 KiB bodies.
+
+No lock removal, tracing/log disabling, caching-policy weakening, bypass of
+security checks, or transformation of complete bodies into buffered copies is
+proposed without a test proving semantic equivalence.
+
+## Acceptance gate on a machine with Rust toolchain and isolated load generator
+
+```bash
+./dev.sh doctor
+./dev.sh check
+CARGO_INCREMENTAL=0 ./dev.sh test
+CARGO_INCREMENTAL=0 cargo build --release --locked
+python3 scripts/performance/regression.py \
+  --binary after="$(realpath target/release/rust-proxy)" \
+  --out /tmp/rustproxy034-1kb-regression
+python3 scripts/performance/lb_regression.py \
+  --binary before=/absolute/path/to/old-rust-proxy \
+  --binary after="$(realpath target/release/rust-proxy)" \
+  --out /tmp/rustproxy034-1kb-lb
+python3 scripts/performance/transport_regression.py \
+  --binary before=/absolute/path/to/old-rust-proxy \
+  --binary after="$(realpath target/release/rust-proxy)" \
+  --out /tmp/rustproxy034-1kb-transport
+python3 scripts/performance/benchmark.py \
+  --binary before=/absolute/path/to/old-rust-proxy \
+  --binary after="$(realpath target/release/rust-proxy)" \
+  --nginx /usr/sbin/nginx --wrk /absolute/path/to/wrk \
+  --out /tmp/rustproxy034-1kb-benchmark \
+  --modes proxy --sizes small --concurrency 16,64,128 \
+  --duration 30 --warmup 5 --repeats 7
+```
+
+Use the **same explicit** `max_idle_per_host=64` and all other vhost settings
+for the *source A/B* test. Run 4 vs 64 as a separate *configuration A/B* test;
+never attribute a settings difference to code. Verify full content checksums,
+HTTP 2xx/4xx/5xx, p50/p95/p99, drop counters, backend connections,
+FD/RSS trends, CPU/request and packet/TCP retransmit rates. Test H1/H2/H3,
+TLS, WebSockets, sticky/rate/ACL, cache MISS/HIT/BYPASS, admin sessions, active
+failover, cancellation, idle timeouts and a sustained memory/connection soak.
+Profile the freshly **built** executable separately from timed RPS runs.
+
+Until compilation and those gates pass, this is a **development source-only
+candidate**, not a claim of reaching 100k RPS or 80% nginx.

docs/PROXY_RPS_0.34.0.md

@@ -18,11 +18,12 @@ passes compile, unit tests, live regressions and paired performance comparisons.
 | Log history | Move completed `LogEntry` directly into bounded history when there are no live viewers or enabled external sinks, instead of deeply cloning its strings. Continue cloning and delivering to live viewers/sinks when enabled. | Same JSON fields, history size, drop counters, file/Loki/Logstash forwarding. |
 | Route/HTTP instrumentation | Remove redundant route-ID mutex/write in forwarding and collapse H1/H2 version tallying into one branch. | Route is recorded once before forwarding; H1/H2 counters remain. |
 
-These changes build on the previous 0.34.0 route-Arc, single-backend LB,
+These historical changes build on the previous 0.34.0 route-Arc, single-backend LB,
 conditional retry/fallback-header cloning, atomic exchange flags and cache HOT
-LRU changes. There are no new UI toggles. Pool defaults remain `max_idle_per_host`
-4 and `pool_recycle_requests` 25,000; tune these only in separately labelled
-trials. `sendfile: auto` remains opt-in and applies only to eligible Linux
+LRU changes. There are no new UI toggles. This **earlier** revision retained `max_idle_per_host`
+4 and `pool_recycle_requests` 25,000. The **subsequent DEV 1 KiB revision**
+changes only the default idle limit to 64, based on live configuration A/B
+with the old binary (not source A/B); see `docs/PROXY_1KB_HOTPATH_DEV_0.34.0.md`. `sendfile: auto` remains opt-in and applies only to eligible Linux
 plaintext H1 static WWW responses, not reverse proxy or TLS/H2/H3.
 
 ## Cache transform/query policy

docs/TEST_REPORT_0.34.0.md

@@ -9,6 +9,34 @@ compiled**, the new Rust tests were **not executed**, and no 0.34.0 RustProxy
 binary or live throughput figures are supplied. Source inspection and simulated
 UI/cache tests do not close the runtime acceptance gate.
 
+## DEV 1 KiB reverse-proxy follow-up (October 2026; source only)
+
+The supplied **old executable** passed loopback HTTP/1.1 1,040-byte content
+checks and ApacheBench 12,000-request samples. The configuration trial
+`max_idle_per_host=4 -> 64` increased short-run RPS, but shared-CPU noise
+precludes a stable nginx-parity percentage. The edited sources now use 64 by
+default for absent/new keepalive fields and reduce exclusive-Drop mutex/clone
+and absent-cookie overhead. See `docs/PROXY_1KB_HOTPATH_DEV_0.34.0.md`.
+**No newly modified binary was built or run.** Rust unit tests, runtime
+regressions, long soaks and post-edit code A/B are still required.
+
+Current DEV source checks performed on the modified tree:
+
+| Check | Result | Scope |
+|---|---|---|
+| `./dev.sh doctor` | PASS | Source preflight; reports `cargo` and `rustc` absent |
+| `scripts/tests/rps_hotpath_034_static_test.py` | PASS | Guardrails, defaults, retained routing/security and source-size limits only |
+| `scripts/tests/proxy_hotpath_034_static_test.py` | PASS | Routing and telemetry source guards only |
+| `scripts/tests/cache_scope_034_static_test.py`, `cache_hot_034_static_test.py`, `sendfile_034_static_test.py` | PASS | Cache and WWW source guards only |
+| `scripts/tests/dashboard_presets_sql_test.py` | PASS | SQLite migrations and preset constraints |
+| `scripts/tests/upgrade_performance_test.py` | PASS | Disposable upgrade/preservation/locking test |
+| `scripts/tests/sendfile_034_panel_browser_test.py` | PASS, 2 widths | Simulated Chromium admin API, not live Rust |
+| `scripts/tests/vhost_backend_popup_0337_browser_test.py` | PASS, 4 widths | Simulated Chromium admin API, not live Rust |
+| Node `--check` (vhost editor), Python `compileall`, JSON parser (OpenAPI) | PASS | Syntax only |
+| `./dev.sh check`, `./dev.sh test` | BLOCKED, exit 127 | Rust toolchain missing; no Rust unit/build acceptance |
+| Old supplied binary, `ab`/nginx configuration A/B | COMPLETED, DIAGNOSTIC | 1,040-byte H1 payload; shared-CPU and no new code exercised |
+
+
 ## Later 0.34.0 reverse-proxy optimization revision
 
 The refreshed source archive additionally reuses immutable route handles,

docs/TRANSPORT.md

@@ -182,7 +182,7 @@ Downstream H2 idle retirement is activity-aware: RustProxy tracks transport I/O
 | Backend response-header deadline | 120 seconds, including upload until headers |
 | HTTP body idle timeout | 60 seconds |
 | WebSocket idle timeout / upgrade completion | 300 seconds / 10 seconds |
-| Idle pooled upstream connections | 4 per backend origin, 30-second idle timeout |
+| Idle pooled upstream connections | up to 64 per backend origin by default for new vhosts (1..64 configurable), 30-second idle timeout; existing explicit values preserved |
 | Configuration writers / log-viewer streams | 8 / 8 |
 
 CLI serve flags expose connection/in-flight counts and HTTP body, WebSocket idle,

docs/VHOSTS.md

@@ -228,12 +228,14 @@ origin do not expose a selected backend or stale origin timing.
 |---|---|---|
 | `enabled` | true | Boolean |
 | `idle_timeout_seconds` | 30 | 1..3600 |
-| `max_idle_per_host` | 4 | 1..64 |
+| `max_idle_per_host` | 64 | 1..64 |
 | `tcp_keepalive_seconds` | 30 | 1..3600 |
 | `pool_recycle_requests` | 25000 | 0 (off) or 1000..10000000 |
 | `pool_recycle_seconds` | 3600 | 0 (off) or 60..604800 |
 | `http1_max_buffer_bytes` | 131072 | 8192..1048576 |
 
+The new default of 64 applies when this field is absent in a newly created or deserialized vhost; existing saved values (including 4) remain unchanged. Each idle upstream connection consumes a file descriptor and memory, so use a smaller value for resource-constrained deployments with many origins. See `docs/PROXY_1KB_HOTPATH_DEV_0.34.0.md` for the live configuration A/B evidence.
+
 These configure HTTP idle reuse and TCP keepalive; HTTP/2 also uses pings for active
 connections, with a 10-second ping timeout and no pinging solely to keep idle
 connections alive. `pool_recycle_requests` and `pool_recycle_seconds` bound the lifetime