21 KiB
RustProxy 0.33.8: performance and compatibility results
RustProxy 0.33.8 substantially improves local static-file throughput in this experiment: 5.23 times the rebuilt original for 64 KiB and 10.13 times for 1 MiB. Small proxy and WWW responses improve by 15.71% and 10.41%, respectively. Results are mixed for larger proxy responses: 64 KiB improves by 2.41%, while 1 MiB decreases by 2.77%. This is a first optimization package, with remaining distance to nginx and visible memory and observability costs. It is not a universal speedup, nginx-equivalence or production-capacity claim. [B1, B2]
The changes preserve the LB panel, schema version 10, existing feature controls, TLS/cache defaults and bounded pool recycling. Actual process tests verify the release separately from throughput measurements. The complete package is the unit of attribution; no experiment isolates the contribution of each edit. [F1, F2, S1]
Executable and evidence identity
The tested final executable is rust-proxy 0.33.8, SHA256:
24cc77a084ee1db4cd007f86c4b18b319bdc2e3f19afae7804739abff5e62a12
The rebuilt original 0.33.7 uses the same Rust 1.99.1 compiler, locked dependency
versions and release profile. Its SHA256 is
85a91318b41a405d1322cbfb84979e05490a7c2ad429ee7cb0c0836acca63899.
The independently supplied 0.33.7 executable has SHA256
c68d63326ddf1e25911a583d994a213132dfb1b094425fc6cf358470299f0a92.
Differences from the supplied executable therefore remain a separate comparison;
they cannot be assigned solely to source edits or solely to compiler provenance.
[B1, F1]
The primary run finished at 2026-10-17T09:01:41Z. All 2,077 files in its
sealed manifest were independently rehashed. The SHA256 of that SHA256SUMS
file is 5036d1f27a53dd6434b4be5a9be99a6e300f10821cffcc15b79f4da06ff147ee;
runs.json is
ebd9cc06d52496f776deefc48615598a64f018a10be5811ac89401b06e821e5e.
This report was created after the release build; raw results identify the tested
binary independently of subsequent documentation. Evidence identifiers below
refer to workspace-relative paths in the accompanying verification package. [B2]
What changed, and what remains deferred
The supplied PERF-00..08 plan is broader than this release. Its boundaries are:
| Plan item | Implemented here | Remaining work |
|---|---|---|
| PERF-00: reproducible measurement | Real HTTP runners, interleaved comparison, resource/telemetry capture, exact payload checks, separate syscall tracing | Full concurrency sweep, remote generator/origin, allocator/cycle profiles and per-edit experiments |
| PERF-01: keep-alive tuning | Existing controls and default limits retained | Controlled idle-pool/recycle sweep; no claim from the old configuration-only experiment |
| PERF-02: compiled feature plan | Request-local cache eligibility plan, credited below | Broader immutable CompiledRequestPlan for all optional features |
| PERF-03: cache OFF path | Skip inapplicable cache-policy/key/header/revalidation preparation; retain bypass reasons and cache ON semantics | Further changes only with profiles and equivalent regression coverage |
| PERF-04: body/telemetry cost | Narrow compression-header copies, direct ID hex formatting, lazy idle timers, shorter history mutex section | Wider Exchange/analytics redesign and other locking changes |
| PERF-05: routing context | Existing validated snapshots retained | Shared validated route context and rewrite invalidation redesign |
| PERF-06: static I/O | Narrow filesystem job context, borrowed pinned root, reads capped at 64 KiB, precise local EOF | Buffer sweep, secure sendfile/io_uring and extra FD/content caches |
| PERF-07: pools/connections | Existing semantics tested with real peers | Pool/selection optimization; no LB UI additions |
| PERF-08: release gate | Listed laboratory checks and 30-minute soak completed; documented rollback | Memory plateau, deployment-specific 24-hour test and staged rollout |
CachePlan avoids force-cache/cookie matching and request-policy work when
ineligible, while retaining diagnostic reasons, origin conditionals, privacy,
range/transform interactions and publication tickets. WWW requests move headers
into local serving when cache conditionals do not need a retained copy.
Compression copies only relevant header values when needed. Request IDs retain
fresh OS randomness, SHA-256 and their 64-character lowercase representation.
Idle deadlines and cancellation semantics remain even when timer allocation is
deferred until a pending read. [S1, F1]
Filesystem jobs receive a narrow root context. Nonempty paths borrow the pinned root descriptor for the first secure open, while root-directory operations keep independent directory offsets. Local file bodies report EOF with their final bounded DATA frame; short reads and cancellation do not publish complete cache entries. Arbitrary upstream bodies are not declared complete from Content-Length alone, preserving trailers and framing checks. [S1, F3]
History entry cloning and destruction of evicted entries occur outside its
mutex. The 4,096-entry history, chronological retention, try_lock admission,
drop counter, live broadcast and configured sinks retain their contract.
Compatibility repairs also cover explicit Host ports, long-form .htaccess
Expires, expired cache inventory, reserved diagnostic headers, escaped-JSON
secret redaction and certificate authorization ordering. These are included in
package-level results, without separately assigned RPS gains. [F1, F4, S1]
Measurement method
The primary matrix contains 120 measured samples: proxy/WWW, three object sizes, four executables and five interleaved repetitions. Each run has 15 seconds of measurement after three seconds of excluded warm-up, using 64 actual HTTP/1.1 keep-alive connections. Payloads are 1,040, 65,536 and 1,048,576 bytes; cache is OFF and requests use identity encoding. [B1]
Four frontend workers are pinned to CPUs 0–3, two origin workers to 4–5, and
four wrk threads share CPUs 6–8. All processes share one Linux host/kernel and
an eight-CPU cgroup quota. nginx 1.26.3 uses sendfile, access logging OFF,
upstream keep-alive 64 and proxy_buffering off. RustProxy retains its existing
history/analytics and request-policy work; nginx is a differently featured
reference. This is a warm local-file-cache experiment. [B1]
Before measurement, the actual ACME/Python startup must finish: authenticated status must be terminal, with no active jobs/configured issuers and one stable RustProxy process for five seconds. The harness validates process identities, nginx worker counts and affinity throughout sampling. A discarded early attempt had incomplete nginx descendant accounting and is retained only as diagnostic evidence. Earlier syscall traces without this quiescence gate are also excluded. [B1, S1]
Exact HTTP 200 and payload hashes are checked before and after every measured run; aggregate transfer bytes receive a gross sanity check. wrk does not hash every timed response and counts status errors above 399. All 120 runs passed these checks, with 77,713,019 completed requests and zero reported errors. No completed sample was removed to improve the result. [B1, B2]
Throughput and repeatability
The table reports median requests/second across all five runs.
| Scenario | Supplied 0.33.7 | Rebuilt 0.33.7 | Final 0.33.8 | nginx |
|---|---|---|---|---|
| Proxy, 1,040 B | 27,341.75 | 25,350.63 | 29,334.12 | 170,610.16 |
| Proxy, 64 KiB | 20,485.54 | 20,246.28 | 20,735.07 | 35,393.29 |
| Proxy, 1 MiB | 4,428.31 | 4,588.06 | 4,461.19 | 2,730.23 |
| WWW, 1,040 B | 22,535.57 | 22,558.35 | 24,906.20 | 351,605.13 |
| WWW, 64 KiB | 3,960.03 | 3,990.22 | 20,856.67 | 203,709.91 |
| WWW, 1 MiB | 272.40 | 279.80 | 2,833.71 | 12,611.49 |
Gains use 100 × (median final / median reference − 1). nginx attainment uses
100 × median final / median nginx. [B1, B2]
| Scenario | Gain vs rebuilt | Gain vs supplied | nginx attainment | Paired gain range vs rebuilt |
|---|---|---|---|---|
| Proxy, 1,040 B | +15.71% | +7.29% | 17.19% | +6.40% to +17.52% |
| Proxy, 64 KiB | +2.41% | +1.22% | 58.58% | −4.54% to +14.40% |
| Proxy, 1 MiB | −2.77% | +0.74% | 163.40% | −2.77% to +5.48% |
| WWW, 1,040 B | +10.41% | +10.52% | 7.08% | +1.73% to +23.63% |
| WWW, 64 KiB | +422.70% | +426.68% | 10.24% | +385.45% to +477.68% |
| WWW, 1 MiB | +912.75% | +940.29% | 22.47% | +818.88% to +991.33% |
The five final/rebuilt ratios, in scheduled repetition order, are:
| Scenario | Rep 1 | Rep 2 | Rep 3 | Rep 4 | Rep 5 |
|---|---|---|---|---|---|
| Proxy, 1,040 B | 1.0679 | 1.0767 | 1.0640 | 1.1752 | 1.0747 |
| Proxy, 64 KiB | 1.0160 | 0.9546 | 1.0789 | 1.0707 | 1.1440 |
| Proxy, 1 MiB | 0.9723 | 0.9770 | 0.9868 | 1.0548 | 0.9883 |
| WWW, 1,040 B | 1.0173 | 1.0210 | 1.0893 | 1.1092 | 1.2363 |
| WWW, 64 KiB | 5.3555 | 4.8545 | 5.7768 | 5.3480 | 5.2270 |
| WWW, 1 MiB | 10.2897 | 10.5518 | 9.1888 | 10.9133 | 9.2866 |
Five pairs support the large WWW effect's direction and magnitude, but do not provide a production confidence interval. Proxy 64 KiB has a small headline improvement with one negative pair. Proxy 1 MiB is slower in four pairs, so the modest regression remains explicit. For proxy-small, the paired median gain is 7.47%, different from the 15.71% ratio of variant medians; both statistics are available. Its supplied comparison also contains one negative pair. Do not average these scenario percentages into a universal speedup. [B2]
Latency, CPU and resources
Values below are medians of per-run measurements; latency percentiles are not pooled. Frontend CPU includes its threads and background work. [B2]
| Scenario | p95 ms, rebuilt → final | p99 ms, rebuilt → final | CPU µs/request, rebuilt → final |
|---|---|---|---|
| Proxy, 1,040 B | 4.093 → 3.591 | 5.066 → 4.707 | 147.05 → 127.41 |
| Proxy, 64 KiB | 6.188 → 5.921 | 36.594 → 37.245 | 182.66 → 178.44 |
| Proxy, 1 MiB | 29.206 → 30.222 | 49.315 → 50.747 | 861.28 → 885.37 |
| WWW, 1,040 B | 4.956 → 4.432 | 6.038 → 5.432 | 155.68 → 140.05 |
| WWW, 64 KiB | 27.929 → 5.217 | 35.301 → 6.393 | 657.26 → 171.85 |
| WWW, 1 MiB | 291.222 → 31.234 | 318.346 → 35.295 | 8,388.17 → 1,219.50 |
WWW 64-KiB/1-MiB CPU cost per request falls 73.85%/85.46%, while p95 falls 81.32%/89.27%. Proxy 1-MiB CPU rises 2.80%, p95 rises 3.48% and p99 rises 2.90%. Proxy 64-KiB p99 also rises 1.78%; improvement is not uniform. [B2]
| Scenario | Sampled peak RSS MiB, rebuilt → final | Peak FDs, rebuilt → final |
|---|---|---|
| Proxy, 1,040 B | 56.965 → 59.137 | 162 → 162 |
| Proxy, 64 KiB | 68.543 → 69.105 | 160 → 157 |
| Proxy, 1 MiB | 81.359 → 83.684 | 153 → 151 |
| WWW, 1,040 B | 77.441 → 77.727 | 124 → 123 |
| WWW, 64 KiB | 79.043 → 102.156 | 145 → 124 |
| WWW, 1 MiB | 78.473 → 118.664 | 145 → 145 |
Large WWW responses cost 23.113 MiB and 40.191 MiB additional sampled peak RSS. Processes persist across scenarios, so this includes allocator/workload history. The source-derived 7.5-MiB extra reader-capacity example is not total measured memory cost. Candidate 1-MiB end RSS increases from 114.508 to 118.664 MiB across five repetitions; these short runs do not prove a plateau. End FDs remain 17–19 for large WWW, without an accumulating signal here. Peaks are sampled, not absolute bounds; nginx process RSS can double-count shared pages. [B2, B3]
Only one cgroup throttle event occurs: 7,839 µs during nginx proxy-small; Rust intervals record none. The nginx proxy-small origin consumes approximately 1.982 of its two CPUs, plausibly limiting that reference. nginx static 64-KiB uses 2.573 of three generator CPUs. Rust proxy uses roughly 3.70–3.95 frontend CPUs, with generator/origin aggregate use below their allocations. These bounds and the shared kernel limit capacity interpretation. nginx static 64-KiB p95 is 34.361 ms despite p50 0.215 ms; its corrected closed-loop histogram is not a generic nginx latency verdict. [B2, B3]
Observability under saturation
History still uses best-effort admission. Median rejected-history entries as a percentage of wrk completions are shown separately from HTTP failures. [B2]
| Scenario | Supplied | Rebuilt | Final | Final minus rebuilt, percentage points |
|---|---|---|---|---|
| Proxy, 1,040 B | 3.350984% | 3.325181% | 1.290544% | −2.034637 |
| Proxy, 64 KiB | 2.231502% | 2.203964% | 0.780042% | −1.423922 |
| Proxy, 1 MiB | 0.763138% | 0.765001% | 0.221602% | −0.543399 |
| WWW, 1,040 B | 3.908337% | 3.851348% | 1.504784% | −2.346564 |
| WWW, 64 KiB | 0.707405% | 0.700592% | 1.230427% | +0.529835 |
| WWW, 1 MiB | 0.114600% | 0.120948% | 0.165351% | +0.044403 |
For WWW 64 KiB, every final fraction exceeds its paired baseline at 5.23-times throughput. For 1 MiB, four fractions increase at 10.13-times throughput. These are saturation comparisons, not fixed-arrival-rate experiments. The increases must remain visible; this release does not guarantee zero history rejection. History rejection does not itself reject live broadcast or external sinks. [B2, F1]
Analytics queue drops are zero in all Rust samples. Sink, error-file and cold-cache queue drops are also zero, with those paths disabled in this primary configuration; this does not prove their active throughput. Asynchronous recorded analytics counts differ from wrk completions. Status/traffic snapshots bracket load, postflight and possible in-flight work, making rate denominators approximate. Raw counters, every repetition and aggregate weighted fractions are retained; they are not presented as exact loss probabilities. [B1, B2]
Functional verification and inherited defects
| Verification layer | Final result | Evidence |
|---|---|---|
| Rust suites | 706 passed, zero failed, one optional ID timing microbenchmark ignored | F1 |
| Actual HTTP process regression | 152/152 cases passed | F2 |
| Actual TLS/H2/H3/QUIC and H1 WebSocket tests | 111/111 assertions passed | F3 |
| Two-origin LB behaviour | 17/17 cases passed | F2 |
| Finite concurrency | 18 cases, 1,920 exact responses, zero failures | F2 |
| Separate syscall experiment | 1,042 exact response checks | P1 |
The suites protect cache OFF/ON, privacy, Vary/scheme separation, purge barriers, range/conditional/HEAD, compression/transforms, ACL/SSRF policy, administration session/CSRF, streaming uploads, cancellation, non-replayed writes and leases through EOF. History tests check exact fields, eviction accounting and real file/live delivery during forced contention. SQL and disposable stopped-source upgrade checks pass; UI checks remain simulated API tests. PHP coverage uses a deterministic FastCGI peer, not a deployed PHP-FPM application. [F1–F4]
Both original executables reproduced three live WWW failures:
observability.static_complete_after_content_length, cache.static_hit and
cache.static_empty_hit. All three pass on the final binary; the final expanded
original matrix retained 149 passes and those three failures. Precise local EOF
and initially empty-body handling restore complete-response accounting and cache
publication. Separately, the 22 original Rust failures were audited as six real
production defects, one missing embedded document and fifteen stale/incomplete
fixtures. Corrections retain privacy and invalidation guarantees. [F2, F4]
The syscall experiment traced 500 measured static 64-KiB requests per binary,
plus warm-up/post-detach checks. Selected calls decreased from 60,374 to 7,502
(87.57%); read decreased from 8,135 to 501. All 1,042 responses were exact,
and both traced processes detached and stopped cleanly. This supports a reduction
in observed system-call work; it is not an 87.57% throughput gain, allocation
profile or per-edit attribution. [P1]
Completed 30-minute mixed-traffic soak
The final executable completed 1,800.000810 seconds of paced HTTP/1.1 proxy/static traffic at an offered 100 requests/s. All 180,000 workload operations passed, with zero response errors and zero scheduler-lateness events. This includes 1,782 intentional client closes after 128 bytes; the remaining 178,218 operations checked the complete expected response body and status, including HEAD and ranges. A controlled close does not prove that the server had not already sent the rest of the body. Every operation used a new frontend TCP connection; this is not a long-lived keep-alive soak. [F5]
All 14 cache toggles passed 32 source/mode checks comprising 64 additional GETs, covering real HIT behaviour after enabling and bypass after disabling. The fixture restored its base configuration, completed a 15-second cooldown, and verified backend active exchanges = 0. All four recorded server/origin processes and the disposable fixture directory were absent after teardown. The final binary hash remained unchanged. [F5]
| Observation | Final soak result |
|---|---|
| Analytics queue rejects | 0; recorded-request delta 180,060, including post-baseline cache probes and cooldown |
| History admission rejects | 31 observed, from counter 0 at 0.001 s to 31 at 1,798.842 s; this counter was not resampled after cooldown |
| Upstream pool recycling | 1 observed rotation; the normal bounded recycling policy remained active |
| Sampled RSS peak | 60.641 MiB |
| RSS median, first minute after warm-up to final sampled minute | 52.273 → 60.016 MiB, a 7.742 MiB increase |
| Open file descriptors | 20 at start, 23 sampled peak, 18 after cooldown |
The final sampled minute includes the cooldown. RSS growth and the descriptive linear fit of approximately 16.165 MiB/hour are observations over this short, cache-toggling run, not an extrapolation or a diagnosis of a leak. This test does not establish a memory plateau. There was no accumulating FD signal, but low-rate correctness and resource observations do not establish stability at the much higher saturation rates in the primary benchmark. There is no matched baseline soak from which to attribute this RSS change to the patch. The RSS/FD samples, both telemetry series and the independent interpretation are retained. No 24-hour, TLS/H2/H3 or production-traffic soak was executed. [F5]
Reproduction and deployment boundary
Use project entry points with the private umask they establish:
./dev.sh check
CARGO_INCREMENTAL=0 ./dev.sh test
./dev.sh release
Set the executable/tool variables to reviewed absolute paths and use a fresh results directory. Build the untouched baseline with the same toolchain/lock before comparing it with the candidate:
python3 -B scripts/performance/benchmark.py \
--binary "supplied=$SUPPLIED_BINARY" --binary "before=$REBUILT_BASELINE" \
--binary "after=$CANDIDATE_BINARY" --nginx "$NGINX_BINARY" --wrk "$WRK_BINARY" \
--out "$RESULTS_DIR" --workers 4 --origin-workers 2 \
--server-cpus 0-3 --origin-cpus 4-5 --client-cpus 6-8 --client-threads 4 \
--concurrency 64 --modes proxy,static --sizes small,large,huge \
--repeats 5 --duration 15 --warmup 3
See scripts/performance/README.md for dependencies and separate regression,
transport, profile and soak commands. The offline analyzer refuses incomplete
or unsealed data. Its outputs include comparisons.csv, paired-ratios.csv,
variants.csv, observability.csv and observability-paired.csv. [B2]
Remaining acceptance includes remote-network latency/loss, actual PHP-FPM,
Tinyauth and DNS infrastructure, H2/H3 WebSocket tunnels, deployment-specific
24-hour testing and staged rollout. No production deployment or service-manager
rollback was executed. The detailed script-specific upgrade/rollback review is
verification/release-scope-and-rollback.txt: current update.sh retains two
installed releases and three pre-update backups, conflicting with the older
installation document's no-pruning statement. Keep an independent recovery copy
outside automatic retention and restore matching code/state when needed. [S1]
Evidence index
| ID | Workspace-relative evidence |
|---|---|
| B1 | verification/benchmark-primary/: runs.json, schedule.json, environment.json, completion.json, fixture configurations, raw responses/resources/telemetry and SHA256SUMS |
| B2 | verification/analysis/benchmark-primary/analysis.json, companion CSV/text files, and verification/analysis/primary-review.txt |
| B3 | verification/static-primary-resource-review.txt and verification/primary-teardown-audit.json |
| F1 | verification/build/final-tests-summary.json, dev-test-final.log; source docs/TEST_REPORT.md |
| F2 | verification/finalrelease-summary.json; http-release/, lb-release/, concurrency-release/ reports beneath verification/ |
| F3 | verification/transport-release/release/report.json; verification/static-compatibility-audit.txt |
| F4 | verification/baseline-failure-audit.txt; verification/source_checks/results.json |
| F5 | verification/soak-release/; verification/soak-release-review.json and .txt; verification/soak-process-teardown.json |
| P1 | verification/syscalls-release/ and verification/syscalls-release-analysis.txt |
| S1 | verification/release-scope-and-rollback.txt; supplied AGENT_PERFORMANCE_EXECUTION_PLAN.md; implementation references listed in that review |