RustProxy/docs/PERFORMANCE_0.33.8.md

# RustProxy 0.33.8: performance and compatibility results

RustProxy 0.33.8 substantially improves local static-file throughput in this
experiment: **5.23 times the rebuilt original for 64 KiB and 10.13 times for
1 MiB**. Small proxy and WWW responses improve by 15.71% and 10.41%, respectively.
Results are mixed for larger proxy responses: 64 KiB improves by 2.41%, while
1 MiB decreases by 2.77%. This is a first optimization package, with remaining
distance to nginx and visible memory and observability costs. It is not a
universal speedup, nginx-equivalence or production-capacity claim. [B1, B2]

The changes preserve the LB panel, schema version 10, existing feature controls,
TLS/cache defaults and bounded pool recycling. Actual process tests verify the
release separately from throughput measurements. The complete package is the
unit of attribution; no experiment isolates the contribution of each edit. [F1,
F2, S1]

## Executable and evidence identity

The tested final executable is `rust-proxy 0.33.8`, SHA256:

```text
24cc77a084ee1db4cd007f86c4b18b319bdc2e3f19afae7804739abff5e62a12
```

The rebuilt original 0.33.7 uses the same Rust 1.99.1 compiler, locked dependency
versions and release profile. Its SHA256 is
`85a91318b41a405d1322cbfb84979e05490a7c2ad429ee7cb0c0836acca63899`.
The independently supplied 0.33.7 executable has SHA256
`c68d63326ddf1e25911a583d994a213132dfb1b094425fc6cf358470299f0a92`.
Differences from the supplied executable therefore remain a separate comparison;
they cannot be assigned solely to source edits or solely to compiler provenance.
[B1, F1]

The primary run finished at `2026-10-17T09:01:41Z`. All **2,077** files in its
sealed manifest were independently rehashed. The SHA256 of that `SHA256SUMS`
file is `5036d1f27a53dd6434b4be5a9be99a6e300f10821cffcc15b79f4da06ff147ee`;
`runs.json` is
`ebd9cc06d52496f776deefc48615598a64f018a10be5811ac89401b06e821e5e`.
This report was created after the release build; raw results identify the tested
binary independently of subsequent documentation. Evidence identifiers below
refer to workspace-relative paths in the accompanying verification package. [B2]

## What changed, and what remains deferred

The supplied PERF-00..08 plan is broader than this release. Its boundaries are:

| Plan item | Implemented here | Remaining work |
|---|---|---|
| PERF-00: reproducible measurement | Real HTTP runners, interleaved comparison, resource/telemetry capture, exact payload checks, separate syscall tracing | Full concurrency sweep, remote generator/origin, allocator/cycle profiles and per-edit experiments |
| PERF-01: keep-alive tuning | Existing controls and default limits retained | Controlled idle-pool/recycle sweep; no claim from the old configuration-only experiment |
| PERF-02: compiled feature plan | Request-local cache eligibility plan, credited below | Broader immutable `CompiledRequestPlan` for all optional features |
| PERF-03: cache OFF path | Skip inapplicable cache-policy/key/header/revalidation preparation; retain bypass reasons and cache ON semantics | Further changes only with profiles and equivalent regression coverage |
| PERF-04: body/telemetry cost | Narrow compression-header copies, direct ID hex formatting, lazy idle timers, shorter history mutex section | Wider `Exchange`/analytics redesign and other locking changes |
| PERF-05: routing context | Existing validated snapshots retained | Shared validated route context and rewrite invalidation redesign |
| PERF-06: static I/O | Narrow filesystem job context, borrowed pinned root, reads capped at 64 KiB, precise local EOF | Buffer sweep, secure sendfile/io_uring and extra FD/content caches |
| PERF-07: pools/connections | Existing semantics tested with real peers | Pool/selection optimization; no LB UI additions |
| PERF-08: release gate | Listed laboratory checks and 30-minute soak completed; documented rollback | Memory plateau, deployment-specific 24-hour test and staged rollout |

`CachePlan` avoids force-cache/cookie matching and request-policy work when
ineligible, while retaining diagnostic reasons, origin conditionals, privacy,
range/transform interactions and publication tickets. WWW requests move headers
into local serving when cache conditionals do not need a retained copy.
Compression copies only relevant header values when needed. Request IDs retain
fresh OS randomness, SHA-256 and their 64-character lowercase representation.
Idle deadlines and cancellation semantics remain even when timer allocation is
deferred until a pending read. [S1, F1]

Filesystem jobs receive a narrow root context. Nonempty paths borrow the pinned
root descriptor for the first secure open, while root-directory operations keep
independent directory offsets. Local file bodies report EOF with their final
bounded DATA frame; short reads and cancellation do not publish complete cache
entries. Arbitrary upstream bodies are not declared complete from Content-Length
alone, preserving trailers and framing checks. [S1, F3]

History entry cloning and destruction of evicted entries occur outside its
mutex. The 4,096-entry history, chronological retention, `try_lock` admission,
drop counter, live broadcast and configured sinks retain their contract.
Compatibility repairs also cover explicit Host ports, long-form `.htaccess`
Expires, expired cache inventory, reserved diagnostic headers, escaped-JSON
secret redaction and certificate authorization ordering. These are included in
package-level results, without separately assigned RPS gains. [F1, F4, S1]

## Measurement method

The primary matrix contains **120 measured samples**: proxy/WWW, three object
sizes, four executables and five interleaved repetitions. Each run has 15 seconds
of measurement after three seconds of excluded warm-up, using 64 actual
HTTP/1.1 keep-alive connections. Payloads are 1,040, 65,536 and 1,048,576 bytes;
cache is OFF and requests use identity encoding. [B1]

Four frontend workers are pinned to CPUs 0–3, two origin workers to 4–5, and
four wrk threads share CPUs 6–8. All processes share one Linux host/kernel and
an eight-CPU cgroup quota. nginx 1.26.3 uses sendfile, access logging OFF,
upstream keep-alive 64 and `proxy_buffering off`. RustProxy retains its existing
history/analytics and request-policy work; nginx is a differently featured
reference. This is a warm local-file-cache experiment. [B1]

Before measurement, the actual ACME/Python startup must finish: authenticated
status must be terminal, with no active jobs/configured issuers and one stable
RustProxy process for five seconds. The harness validates process identities,
nginx worker counts and affinity throughout sampling. A discarded early attempt
had incomplete nginx descendant accounting and is retained only as diagnostic
evidence. Earlier syscall traces without this quiescence gate are also excluded.
[B1, S1]

Exact HTTP 200 and payload hashes are checked before and after every measured
run; aggregate transfer bytes receive a gross sanity check. wrk does not hash
every timed response and counts status errors above 399. All 120 runs passed
these checks, with **77,713,019 completed requests and zero reported errors**.
No completed sample was removed to improve the result. [B1, B2]

## Throughput and repeatability

The table reports median requests/second across all five runs.

| Scenario | Supplied 0.33.7 | Rebuilt 0.33.7 | Final 0.33.8 | nginx |
|---|---:|---:|---:|---:|
| Proxy, 1,040 B | 27,341.75 | 25,350.63 | 29,334.12 | 170,610.16 |
| Proxy, 64 KiB | 20,485.54 | 20,246.28 | 20,735.07 | 35,393.29 |
| Proxy, 1 MiB | 4,428.31 | 4,588.06 | 4,461.19 | 2,730.23 |
| WWW, 1,040 B | 22,535.57 | 22,558.35 | 24,906.20 | 351,605.13 |
| WWW, 64 KiB | 3,960.03 | 3,990.22 | 20,856.67 | 203,709.91 |
| WWW, 1 MiB | 272.40 | 279.80 | 2,833.71 | 12,611.49 |

Gains use `100 × (median final / median reference − 1)`. nginx attainment uses
`100 × median final / median nginx`. [B1, B2]

| Scenario | Gain vs rebuilt | Gain vs supplied | nginx attainment | Paired gain range vs rebuilt |
|---|---:|---:|---:|---:|
| Proxy, 1,040 B | +15.71% | +7.29% | 17.19% | +6.40% to +17.52% |
| Proxy, 64 KiB | +2.41% | +1.22% | 58.58% | −4.54% to +14.40% |
| Proxy, 1 MiB | **−2.77%** | +0.74% | 163.40% | −2.77% to +5.48% |
| WWW, 1,040 B | +10.41% | +10.52% | 7.08% | +1.73% to +23.63% |
| WWW, 64 KiB | +422.70% | +426.68% | 10.24% | +385.45% to +477.68% |
| WWW, 1 MiB | +912.75% | +940.29% | 22.47% | +818.88% to +991.33% |

The five final/rebuilt ratios, in scheduled repetition order, are:

| Scenario | Rep 1 | Rep 2 | Rep 3 | Rep 4 | Rep 5 |
|---|---:|---:|---:|---:|---:|
| Proxy, 1,040 B | 1.0679 | 1.0767 | 1.0640 | 1.1752 | 1.0747 |
| Proxy, 64 KiB | 1.0160 | 0.9546 | 1.0789 | 1.0707 | 1.1440 |
| Proxy, 1 MiB | 0.9723 | 0.9770 | 0.9868 | 1.0548 | 0.9883 |
| WWW, 1,040 B | 1.0173 | 1.0210 | 1.0893 | 1.1092 | 1.2363 |
| WWW, 64 KiB | 5.3555 | 4.8545 | 5.7768 | 5.3480 | 5.2270 |
| WWW, 1 MiB | 10.2897 | 10.5518 | 9.1888 | 10.9133 | 9.2866 |

Five pairs support the large WWW effect's direction and magnitude, but do not
provide a production confidence interval. Proxy 64 KiB has a small headline
improvement with one negative pair. Proxy 1 MiB is slower in four pairs, so the
modest regression remains explicit. For proxy-small, the paired median gain is
7.47%, different from the 15.71% ratio of variant medians; both statistics are
available. Its supplied comparison also contains one negative pair. Do not
average these scenario percentages into a universal speedup. [B2]

## Latency, CPU and resources

Values below are medians of per-run measurements; latency percentiles are not
pooled. Frontend CPU includes its threads and background work. [B2]

| Scenario | p95 ms, rebuilt → final | p99 ms, rebuilt → final | CPU µs/request, rebuilt → final |
|---|---:|---:|---:|
| Proxy, 1,040 B | 4.093 → 3.591 | 5.066 → 4.707 | 147.05 → 127.41 |
| Proxy, 64 KiB | 6.188 → 5.921 | 36.594 → 37.245 | 182.66 → 178.44 |
| Proxy, 1 MiB | 29.206 → 30.222 | 49.315 → 50.747 | 861.28 → 885.37 |
| WWW, 1,040 B | 4.956 → 4.432 | 6.038 → 5.432 | 155.68 → 140.05 |
| WWW, 64 KiB | 27.929 → 5.217 | 35.301 → 6.393 | 657.26 → 171.85 |
| WWW, 1 MiB | 291.222 → 31.234 | 318.346 → 35.295 | 8,388.17 → 1,219.50 |

WWW 64-KiB/1-MiB CPU cost per request falls 73.85%/85.46%, while p95 falls
81.32%/89.27%. Proxy 1-MiB CPU rises 2.80%, p95 rises 3.48% and p99 rises
2.90%. Proxy 64-KiB p99 also rises 1.78%; improvement is not uniform. [B2]

| Scenario | Sampled peak RSS MiB, rebuilt → final | Peak FDs, rebuilt → final |
|---|---:|---:|
| Proxy, 1,040 B | 56.965 → 59.137 | 162 → 162 |
| Proxy, 64 KiB | 68.543 → 69.105 | 160 → 157 |
| Proxy, 1 MiB | 81.359 → 83.684 | 153 → 151 |
| WWW, 1,040 B | 77.441 → 77.727 | 124 → 123 |
| WWW, 64 KiB | 79.043 → 102.156 | 145 → 124 |
| WWW, 1 MiB | 78.473 → 118.664 | 145 → 145 |

Large WWW responses cost **23.113 MiB and 40.191 MiB** additional sampled peak
RSS. Processes persist across scenarios, so this includes allocator/workload
history. The source-derived 7.5-MiB extra reader-capacity example is not total
measured memory cost. Candidate 1-MiB end RSS increases from 114.508 to 118.664
MiB across five repetitions; these short runs do not prove a plateau. End FDs
remain 17–19 for large WWW, without an accumulating signal here. Peaks are
sampled, not absolute bounds; nginx process RSS can double-count shared pages.
[B2, B3]

Only one cgroup throttle event occurs: 7,839 µs during nginx proxy-small; Rust
intervals record none. The nginx proxy-small origin consumes approximately
1.982 of its two CPUs, plausibly limiting that reference. nginx static 64-KiB
uses 2.573 of three generator CPUs. Rust proxy uses roughly 3.70–3.95 frontend
CPUs, with generator/origin aggregate use below their allocations. These bounds
and the shared kernel limit capacity interpretation. nginx static 64-KiB p95
is 34.361 ms despite p50 0.215 ms; its corrected closed-loop histogram is not
a generic nginx latency verdict. [B2, B3]

## Observability under saturation

History still uses best-effort admission. Median rejected-history entries as a
percentage of wrk completions are shown separately from HTTP failures. [B2]

| Scenario | Supplied | Rebuilt | Final | Final minus rebuilt, percentage points |
|---|---:|---:|---:|---:|
| Proxy, 1,040 B | 3.350984% | 3.325181% | 1.290544% | −2.034637 |
| Proxy, 64 KiB | 2.231502% | 2.203964% | 0.780042% | −1.423922 |
| Proxy, 1 MiB | 0.763138% | 0.765001% | 0.221602% | −0.543399 |
| WWW, 1,040 B | 3.908337% | 3.851348% | 1.504784% | −2.346564 |
| WWW, 64 KiB | 0.707405% | 0.700592% | 1.230427% | **+0.529835** |
| WWW, 1 MiB | 0.114600% | 0.120948% | 0.165351% | **+0.044403** |

For WWW 64 KiB, every final fraction exceeds its paired baseline at 5.23-times
throughput. For 1 MiB, four fractions increase at 10.13-times throughput. These
are saturation comparisons, not fixed-arrival-rate experiments. The increases
must remain visible; this release does not guarantee zero history rejection.
History rejection does not itself reject live broadcast or external sinks. [B2,
F1]

Analytics queue drops are zero in all Rust samples. Sink, error-file and
cold-cache queue drops are also zero, with those paths disabled in this primary
configuration; this does not prove their active throughput. Asynchronous
recorded analytics counts differ from wrk completions. Status/traffic snapshots
bracket load, postflight and possible in-flight work, making rate denominators
approximate. Raw counters, every repetition and aggregate weighted fractions are
retained; they are not presented as exact loss probabilities. [B1, B2]

## Functional verification and inherited defects

| Verification layer | Final result | Evidence |
|---|---|---|
| Rust suites | **706 passed**, zero failed, one optional ID timing microbenchmark ignored | F1 |
| Actual HTTP process regression | **152/152 cases passed** | F2 |
| Actual TLS/H2/H3/QUIC and H1 WebSocket tests | **111/111 assertions passed** | F3 |
| Two-origin LB behaviour | **17/17 cases passed** | F2 |
| Finite concurrency | **18 cases, 1,920 exact responses**, zero failures | F2 |
| Separate syscall experiment | **1,042 exact response checks** | P1 |

The suites protect cache OFF/ON, privacy, Vary/scheme separation, purge barriers,
range/conditional/HEAD, compression/transforms, ACL/SSRF policy, administration
session/CSRF, streaming uploads, cancellation, non-replayed writes and leases
through EOF. History tests check exact fields, eviction accounting and real
file/live delivery during forced contention. SQL and disposable stopped-source
upgrade checks pass; UI checks remain simulated API tests. PHP coverage uses a
deterministic FastCGI peer, not a deployed PHP-FPM application. [F1–F4]

Both original executables reproduced three live WWW failures:
`observability.static_complete_after_content_length`, `cache.static_hit` and
`cache.static_empty_hit`. All three pass on the final binary; the final expanded
original matrix retained 149 passes and those three failures. Precise local EOF
and initially empty-body handling restore complete-response accounting and cache
publication. Separately, the 22 original Rust failures were audited as six real
production defects, one missing embedded document and fifteen stale/incomplete
fixtures. Corrections retain privacy and invalidation guarantees. [F2, F4]

The syscall experiment traced 500 measured static 64-KiB requests per binary,
plus warm-up/post-detach checks. Selected calls decreased from 60,374 to 7,502
(87.57%); `read` decreased from 8,135 to 501. All 1,042 responses were exact,
and both traced processes detached and stopped cleanly. This supports a reduction
in observed system-call work; it is not an 87.57% throughput gain, allocation
profile or per-edit attribution. [P1]

## Completed 30-minute mixed-traffic soak

The final executable completed **1,800.000810 seconds** of paced HTTP/1.1
proxy/static traffic at an offered 100 requests/s. All **180,000 workload
operations passed**, with zero response errors and zero scheduler-lateness
events. This includes **1,782 intentional client closes after 128 bytes**;
the remaining **178,218 operations** checked the complete expected response
body and status, including HEAD and ranges. A controlled close does not prove
that the server had not already sent the rest of the body. Every operation used
a new frontend TCP connection; this is not a long-lived keep-alive soak. [F5]

All **14 cache toggles** passed **32 source/mode checks comprising 64 additional
GETs**, covering real HIT behaviour after enabling and bypass after disabling.
The fixture restored its base configuration, completed a 15-second cooldown,
and verified **backend active exchanges = 0**. All four recorded server/origin
processes and the disposable fixture directory were absent after teardown.
The final binary hash remained unchanged. [F5]

| Observation | Final soak result |
|---|---|
| Analytics queue rejects | **0**; recorded-request delta **180,060**, including post-baseline cache probes and cooldown |
| History admission rejects | **31 observed**, from counter 0 at 0.001 s to 31 at 1,798.842 s; this counter was not resampled after cooldown |
| Upstream pool recycling | **1 observed rotation**; the normal bounded recycling policy remained active |
| Sampled RSS peak | **60.641 MiB** |
| RSS median, first minute after warm-up to final sampled minute | **52.273 → 60.016 MiB**, a **7.742 MiB increase** |
| Open file descriptors | **20 at start, 23 sampled peak, 18 after cooldown** |

The final sampled minute includes the cooldown. RSS growth and the descriptive
linear fit of approximately 16.165 MiB/hour are observations over this short,
cache-toggling run, not an extrapolation or a diagnosis of a leak. This test
**does not establish a memory plateau**. There was no accumulating FD signal,
but low-rate correctness and resource observations do not establish stability
at the much higher saturation rates in the primary benchmark. There is no
matched baseline soak from which to attribute this RSS change to the patch.
The RSS/FD samples, both telemetry series and the independent interpretation
are retained. No 24-hour, TLS/H2/H3 or production-traffic soak was executed. [F5]

## Reproduction and deployment boundary

Use project entry points with the private umask they establish:

```bash
./dev.sh check
CARGO_INCREMENTAL=0 ./dev.sh test
./dev.sh release
```

Set the executable/tool variables to reviewed absolute paths and use a fresh
results directory. Build the untouched baseline with the same toolchain/lock
before comparing it with the candidate:

```bash
python3 -B scripts/performance/benchmark.py \
  --binary "supplied=$SUPPLIED_BINARY" --binary "before=$REBUILT_BASELINE" \
  --binary "after=$CANDIDATE_BINARY" --nginx "$NGINX_BINARY" --wrk "$WRK_BINARY" \
  --out "$RESULTS_DIR" --workers 4 --origin-workers 2 \
  --server-cpus 0-3 --origin-cpus 4-5 --client-cpus 6-8 --client-threads 4 \
  --concurrency 64 --modes proxy,static --sizes small,large,huge \
  --repeats 5 --duration 15 --warmup 3
```

See `scripts/performance/README.md` for dependencies and separate regression,
transport, profile and soak commands. The offline analyzer refuses incomplete
or unsealed data. Its outputs include `comparisons.csv`, `paired-ratios.csv`,
`variants.csv`, `observability.csv` and `observability-paired.csv`. [B2]

Remaining acceptance includes remote-network latency/loss, actual PHP-FPM,
Tinyauth and DNS infrastructure, H2/H3 WebSocket tunnels, deployment-specific
24-hour testing and staged rollout. No production deployment or service-manager
rollback was executed. The detailed script-specific upgrade/rollback review is
`verification/release-scope-and-rollback.txt`: current `update.sh` retains two
installed releases and three pre-update backups, conflicting with the older
installation document's no-pruning statement. Keep an independent recovery copy
outside automatic retention and restore matching code/state when needed. [S1]

## Evidence index

| ID | Workspace-relative evidence |
|---|---|
| B1 | `verification/benchmark-primary/`: `runs.json`, `schedule.json`, `environment.json`, `completion.json`, fixture configurations, raw responses/resources/telemetry and `SHA256SUMS` |
| B2 | `verification/analysis/benchmark-primary/analysis.json`, companion CSV/text files, and `verification/analysis/primary-review.txt` |
| B3 | `verification/static-primary-resource-review.txt` and `verification/primary-teardown-audit.json` |
| F1 | `verification/build/final-tests-summary.json`, `dev-test-final.log`; source `docs/TEST_REPORT.md` |
| F2 | `verification/finalrelease-summary.json`; `http-release/`, `lb-release/`, `concurrency-release/` reports beneath `verification/` |
| F3 | `verification/transport-release/release/report.json`; `verification/static-compatibility-audit.txt` |
| F4 | `verification/baseline-failure-audit.txt`; `verification/source_checks/results.json` |
| F5 | `verification/soak-release/`; `verification/soak-release-review.json` and `.txt`; `verification/soak-process-teardown.json` |
| P1 | `verification/syscalls-release/` and `verification/syscalls-release-analysis.txt` |
| S1 | `verification/release-scope-and-rollback.txt`; supplied `AGENT_PERFORMANCE_EXECUTION_PLAN.md`; implementation references listed in that review |