RustProxy/scripts/performance

Live performance and regression harness

DEV OPT3 / 1 KiB response: uncompiled source

docs/PROXY_1KB_OPT3_DEV_0.34.0.md describes 7 optimized hot-path sites and how to benchmark their newly rebuilt executable. Baseline rust-proxy(2) passed 89 real proxy/policy/cache checks. This runtime result is not evidence that uncompiled OPT3 changes work, nor a measured OPT3 RPS gain.

These scripts exercise actual RustProxy executables through loopback sockets. They do not simulate Rust request handling. The Python server in peer.py is a controllable test upstream, used to verify streaming, non-replayable writes, headers, cache policy and WebSocket frames. Performance measurements use nginx as the shared upstream instead of Python.

DEV 1 KiB histogram writes (source candidate)

VhostMetrics::complete now writes one exclusive histogram bin for each completed exchange instead of up to 11 cumulative atomics. Both Prometheus exporters reconstruct le and +Inf at scrape time. Unit tests cover boundary semantics. This change has not been compiled or benchmarked: see docs/PROXY_1KB_COUNTERS_DEV_0.34.0.md.

DEV 1 KiB cache counter follow-up

See docs/PROXY_1KB_COUNTERS_DEV_0.34.0.md. Cache domain metrics no longer lock the HOT store on BYPASS; analytics cloning is deferred until the queue has capacity. The new source remains uncompiled in the editing environment. For quick 1,040-byte loopback diagnostics with ApacheBench (not a production acceptance benchmark), use python3 scripts/performance/ab_1kb_dev.py --help. The authoritative long-run comparison remains benchmark.py with pinned CPU sets, an explicitly equal upstream idle pool and multiple wrk repeats.

DEV 1 KiB follow-up on 0.34.0 (newer source)

See docs/PROXY_1KB_HOTPATH_DEV_0.34.0.md. The previous executable's 4-vs-64 idle-connection tuning experiment is real loopback evidence, but the edited source has not been compiled. Its new default is 64 for new vhosts, whereas explicitly saved 4 remains 4. Keep the idle limit equal between before/after code benchmarks; measure configuration tuning in a separate labelled series. The source release is not eligible for a claim of 100k RPS or 80% nginx yet.

Reverse-proxy optimization revision 0.34.0

After compiling the refreshed candidate, measure the same unmodified 0.33.8 binary and nginx reference as before, without turning off logging, request IDs, ACL, cache-policy enforcement or LB health. Profile request-size 1,040 B first. Before RPS runs execute full proxy, LB, transport and concurrency regressions (including single-member capacity, recovery, sticky cookies, replays on safe methods, rejected non-replayable writes and H2 connection shutdown). Treat --max-idle tuning and recycles as separately labelled experiments rather than mixing them into the primary comparison. There are no new measured RPS figures in the source candidate.

Follow-up RPS candidate on 0.34.0

docs/PROXY_RPS_0.34.0.md tracks the code changes and their regressions. Compile before/after binaries from the two 0.34.0 source revisions and use the paired benchmark below; a live run against the older binary cannot establish a measured RPS improvement for the uncompiled candidate. Keep max_idle_per_host, recycling, HTTP1/H2 settings and all application security and logging features equal between runs. Treat any configuration tuning as a separate experiment. Profile the resulting binary with perf only after correctness checks pass.

Offline cache-retention model for 0.34.0

python3 scripts/performance/simulate_cache_034.py compares cache invalidation policies under a deterministic synthetic sequence of vhost saves and uniform cacheable GETs. The model contains no RustProxy server, no timing and no network traffic; its percentages are not measured RPS gains. See docs/TEST_REPORT_0.34.0.md for assumptions and the live acceptance gap.

Requirements

• Linux with readable /proc, Python 3.10+, executable RustProxy binaries.
• nginx and wrk for performance comparisons. Both paths are explicit; no system-wide installation is required.
• taskset from util-linux for the benchmark's explicit CPU affinity and for optional affinity settings in the other scripts.
• Optional Python brotli and zstandard decoders increase compression acceptance coverage. A missing optional decoder is reported as skipped.
• Permission to bind unprivileged loopback sockets and start local processes. In a sandbox without sockets or /proc, run in the approved local execution environment. Do not replace real runtime results with a peer self-test.

Each run creates a new private temporary data directory and initializes its own administrator with a random password supplied only over stdin. Configurations are applied through RustProxy's existing authenticated CLI/API and listener changes receive a restart. No historical SQLite database is copied or edited. TLS keys, databases and passwords are never written to the evidence directory. Temporary state is outside the evidence tree and is removed on normal exit. fixture-cleanup.json records the cleanup check. Interrupted runs can leave an unrelated /tmp/rustproxy-perf-* directory; do not include it in a release.

Before timed benchmark, soak or syscall-profile work, wait_quiescent() waits for the existing authenticated ACME runtime API to report ready or unavailable. A fresh RustProxy data directory can start a Python/Certbot environment bootstrap even when no certificates have an issuer. The gate requires no configured issuers, jobs or renewal observations, the same live Rust PID and start time, and no child processes for five continuous seconds. It polls every 0.5 seconds and fails after 180 seconds instead of measuring installation work. Selected status, process count and timing evidence is saved in <label>-quiescence.json. No PATH, pip settings or application features are changed. An unavailable result permits the HTTP fixture to proceed; these runs do not test ACME issuance or certificate-provider availability.

Regression tests

Run from the repository root, providing absolute binary and output paths:

python3 scripts/performance/regression.py \
  --binary before=/absolute/path/rust-proxy-before \
  --binary after=/absolute/path/rust-proxy-after \
  --out /absolute/path/results/regression

--groups static,proxy,policy,cache,transform selects groups; all are enabled by default. --workers 4 sets the same TOKIO_WORKER_THREADS for each binary. The report contains individual pass/fail/skip observations and a comparison matrix. An existing failure in the supplied binary remains a failure in its report; it is not silently converted to a successful baseline expectation.

Coverage includes:

• Static GET/HEAD and byte checksums at 1,040 B, 64 KiB and 1 MiB; empty files; ETag/Last-Modified conditional requests; prefix/suffix/open/multipart ranges, If-Range and 416; directory redirect/index/listing; decoded path rejection; deny/inside-root symlinks; precompressed gzip; MIME/download headers; try_files, redirects and the compiled .htaccess header subset.
• Proxy POST/PUT, 1 MiB upload checksums, chunked upload delivery before the last chunk is sent, early response delivery, client disconnect, original status/redirect preservation, failed-POST no replay, request/response header scopes, hop-by-hop stripping, request-ID shape/uniqueness and WebSocket echo.
• Global host/vhost/profile/location ACL composition, upstream suppression on denial, trusted versus spoofed forwarding, untrusted country hints, method and rate policies, hot changes and rejection of invalid configuration.
• Private administration routes, login, session/CSRF and Origin enforcement, Basic Auth, ACL-before-auth, private-origin rejection, retained exchange logging fields and byte accounting.
• Cache global/vhost/route OFF and ON, MISS/HIT, Vary isolation, cookie, Authorization/private/no-store bypass, PURGE, concurrent fills, validator revalidation, static cached ranges/conditionals; transforms OFF/ON with cache OFF/ON; compression negotiation, exclusions and scope/global codec ceilings.

The generic suite does not establish TLS/H2/H3, Tinyauth, real PHP-FPM, real DNS rebinding, concurrent filesystem rename-race safety or long-soak acceptance. Use the dedicated transport tests and existing Rust integration suites for their covered cases. The sequential symlink-target test is explicitly not a concurrent rename-race proof.

TLS, HTTP/2, HTTP/3 and WebSocket interoperability

python3 scripts/performance/transport_regression.py \
  --binary before=/absolute/path/rust-proxy-before \
  --binary after=/absolute/path/rust-proxy-after \
  --out /absolute/path/results/transport

This suite requires aioquic, httpx[http2] and websockets; use --python-deps /absolute/path/pydeps for a separate dependency directory. It verifies the temporary server certificate, negotiated protocol, response status and bytes for TLS H2 and QUIC H3, including HEAD, validators, ranges, cache ON/OFF, 1 MiB bodies and concurrent streams. Dedicated WebSocket checks include bidirectional echo and a normal close handshake. Client protocol self-checks are reported separately from RustProxy acceptance cases. This establishes covered interoperability, not TLS/H2/H3 throughput or production-certificate compatibility.

Two-origin load-balancer regression

python3 scripts/performance/lb_regression.py \
  --binary before=/absolute/path/rust-proxy-before \
  --binary after=/absolute/path/rust-proxy-after \
  --out /absolute/path/results/lb

This separate, small suite sends real requests through RustProxy to two local TCP origins. It verifies equal round-robin allocation, GET failover after a 503 and after a stopped origin refuses connections, and the absence of POST replay. A controlled response pauses after its first body fragment while configuration changes drain or disable its member. New requests must use the other member, the admitted response must finish intact, and its active lease must remain counted until completion and then return to zero. The main HTTP case matrix is unchanged. This suite does not cover sticky sessions, active health probes, DNS rotation or TLS upstream failures.

Finite concurrency correctness stress

python3 scripts/performance/concurrency_regression.py \
  --binary before=/absolute/path/rust-proxy-before \
  --binary after=/absolute/path/rust-proxy-after \
  --nginx /absolute/path/nginx --out /absolute/path/results/concurrency

This separate functional check uses 1, 16 and 128 concurrent client tasks for proxy and static responses at 1,040 B, 64 KiB and 1 MiB. Each finite batch sends max(32, 2 * clients) requests and verifies every status, length and body SHA-256. After the clients finish, backend active-exchange counters must settle to zero. RSS and file descriptors are recorded before the batch and after its drain. These are not peak resource measurements or throughput results. Client tasks start behind a barrier; configured concurrency does not establish that the same number of server exchanges were active simultaneously.

Before/after/nginx benchmark

First verify starts, loopback routing and exact payload hashes without invoking the load generator:

python3 scripts/performance/benchmark.py \
  --binary before=/absolute/path/rust-proxy-before \
  --binary after=/absolute/path/rust-proxy-after \
  --nginx /absolute/path/nginx --wrk /absolute/path/wrk \
  --server-cpus 0-3 --origin-cpus 4-5 --client-cpus 6-8 \
  --out /absolute/path/results/preflight --prepare-only

For a host with CPUs 0–8 available, a reproducible primary run is:

python3 scripts/performance/benchmark.py \
  --binary before=/absolute/path/rust-proxy-before \
  --binary after=/absolute/path/rust-proxy-after \
  --nginx /absolute/path/nginx --wrk /absolute/path/wrk \
  --out /absolute/path/results/primary \
  --workers 4 --origin-workers 2 --client-threads 4 \
  --server-cpus 0-3 --origin-cpus 4-5 --client-cpus 6-8 \
  --concurrency 64 --sizes small,large,huge --modes proxy,static \
  --duration 15 --warmup 3 --repeats 5

Choose explicit, nonempty and disjoint CPU sets allowed in the run environment; adapt all three sets on a smaller host. Frontends use the same CPU set and worker count, the shared origin uses another set, and the generator uses a third. The harness checks each process/thread's actual affinity and refuses unavailable resource counters. Four wrk threads share the three client CPUs in this example. The structured result records thread count and effective connection count; at 64 connections, four threads keep exactly 64 connections, avoiding wrk's integer division of connections between threads. Only one frontend is loaded at a time. Stop builds, other tests and profilers during throughput measurement.

The primary run takes approximately 27 minutes: six scenarios, three variants, five samples, and 18 seconds per sample including excluded warm-up. To explore concurrency, repeat in a fresh output directory with --concurrency 1,16,64,128. More scenarios intentionally take more time. Short sanity checks require the explicit --exploratory option and cannot satisfy the >=5 samples / >=15-second measurement gate.

Cache is explicitly disabled for the primary benchmark. Transforms retain their disabled defaults. Identity encoding is requested. Logging history, analytics, request IDs, origin policy, body accounting, health/lease plumbing and normal application functionality remain active. nginx uses sendfile on, HTTP/1.1, keep-alive and streaming proxy delivery. The generated exact configurations are saved, so this intentional difference in nginx/application features is visible. No cache-disabled claim is inferred from a cache-on speedup.

Optional keep-alive tuning uses --max-idle label=64. Keep this as a separately labelled experiment. The harness does not turn off bounded pool recycling.

Recorded evidence and interpretation

• schedule.json: complete deterministic crossover order, rotating and reversing variants between repetitions. No best-run selection.
• preflight.json, fixture/payloads.json, binary/version/config metadata: exact status, body SHA-256, payload sizes and inputs.
• Per-sample exact status/body checks before and after the measured interval, plus a wire-bytes-per-request sanity check, detect wrong content and gross transfer-size anomalies. These do not hash every response inside wrk.
• raw/*.txt and raw/*.json: unmodified wrk output and structured requests, duration, bytes, RPS, p50/p95/p99, effective concurrency, and connect/read/write/timeout/status errors.
• raw/*-resources.json: timestamped /proc samples for each server process and its children, plus the separate origin process group.
• raw/*-affinity.json: observed process/thread CPU masks for the front end, origin and generator.
• runs.csv/json, summary.csv/json: all samples, medians, RPS range and coefficient of variation, latency medians, server CPU time per completed request, peak/end RSS and file descriptors.
• status/*.json: numeric drop and pool-generation/rotation counters read outside each measured interval. Warm-up output is kept separately.
• completion.json and SHA256SUMS: completion state and hashes of evidence. Any generator/network/status error stops the run with failure.json; partial data remains available and is not presented as completed acceptance.

Use (after_median / before_median - 1) * 100 for throughput improvement and after_median / nginx_median * 100 for relative nginx throughput. Rebuilding both revisions with one compiler, target and resolved lockfile is required for source-change attribution. Compare the supplied executable separately: build flags and dependency versions can change its performance.

These are same-host, closed-loop HTTP/1.1 diagnostics. They do not establish production capacity, TLS/H2/H3 performance, an open-loop latency SLA, allocator rates, CPU cycles or 24-hour resource stability. wrk latency follows wrk's own histogram/correction behavior. RSS and descriptor peaks are sampled observations, so shorter spikes may be missed. Summed nginx RSS can double-count shared pages. Origin or generator CPU saturation can limit observed throughput; inspect their resource data and move the generator/origin to separate machines for release measurements. Run profilers separately so profiling overhead does not enter the headline A/B throughput numbers.

Mixed-traffic resource soak

Run this in its own phase, after correctness and throughput checks:

python3 scripts/performance/soak.py \
  --binary /absolute/path/rust-proxy-after --label after \
  --nginx /absolute/path/nginx --out /absolute/path/results/soak \
  --duration 1800 --rate 100 --clients 8 \
  --workers 4 --origin-workers 2 --server-cpus 0-3 --origin-cpus 4-5 \
  --toggle-seconds 120 --sample-seconds 5 --abort-every 101

The bounded client executor alternates proxy/static and 1,040 B/64 KiB/1 MiB payloads, verifies full bytes and occasional HEAD/range semantics, closes some clients after an initial body prefix, and applies live cache OFF/ON changes. Every workload request opens a new frontend HTTP/1.1 connection; upstream keepalive retains its configured defaults. This exercises connection churn. The primary wrk benchmark separately uses persistent frontend connections. The nominal request rate is paced; scheduler delays and actual completion rate are reported instead of hiding an under-driven client. Pending jobs are bounded to twice the client count. The script restores base configuration, drains and rechecks both listeners before shutdown. After each cache apply, two separate GETs per source verify exact contents and require a HIT on the second GET when enabled; globally disabled cache must expose no X-Cache header. These probe requests are recorded separately from paced traffic. Backend active-exchange counters must settle to zero after the final drain and cooldown.

Five-second resource/status records, body/status failures, observed cache-status counts, cache transitions/probes, RSS/FD trend and completed traffic totals are retained. Controlled client closes mean that the client closes after reading a prefix. The server may already have sent the complete response, so those counts are not presented as proven server-side cancellations. Memory trend is descriptive: switching caching and allocator retention affect RSS. This is a 30-minute resource gate, not a 24-hour leak proof. A short --exploratory --duration ... sanity run stays explicitly outside that gate.

0.34.0 hypothesis-only sendfile/nginx gap simulator

python3 scripts/performance/simulate_nginx_gap_034.py --output /tmp/nginx-gap.json

This is not a benchmark of RustProxy 0.34.0. It reproduces measured 0.33.8/nginx medians with cache OFF and applies explicitly hypothetical Amdahl service-time reductions (WWW and reverse proxy separately). A real RustProxy binary is required to validate the code path, detect regressions and measure latency/RPS against nginx under identical conditions.