RustProxy/scripts/performance

Live performance and regression harness

These scripts exercise actual RustProxy executables through loopback sockets. They do not simulate Rust request handling. The Python server in peer.py is a controllable test upstream, used to verify streaming, non-replayable writes, headers, cache policy and WebSocket frames. Performance measurements use nginx as the shared upstream instead of Python.

Requirements

• Linux with readable /proc, Python 3.10+, executable RustProxy binaries.
• nginx and wrk for performance comparisons. Both paths are explicit; no system-wide installation is required.
• taskset from util-linux for the benchmark's explicit CPU affinity and for optional affinity settings in the other scripts.
• Optional Python brotli and zstandard decoders increase compression acceptance coverage. A missing optional decoder is reported as skipped.
• Permission to bind unprivileged loopback sockets and start local processes. In a sandbox without sockets or /proc, run in the approved local execution environment. Do not replace real runtime results with a peer self-test.

Each run creates a new private temporary data directory and initializes its own administrator with a random password supplied only over stdin. Configurations are applied through RustProxy's existing authenticated CLI/API and listener changes receive a restart. No historical SQLite database is copied or edited. TLS keys, databases and passwords are never written to the evidence directory. Temporary state is outside the evidence tree and is removed on normal exit. fixture-cleanup.json records the cleanup check. Interrupted runs can leave an unrelated /tmp/rustproxy-perf-* directory; do not include it in a release.

Before timed benchmark, soak or syscall-profile work, wait_quiescent() waits for the existing authenticated ACME runtime API to report ready or unavailable. A fresh RustProxy data directory can start a Python/Certbot environment bootstrap even when no certificates have an issuer. The gate requires no configured issuers, jobs or renewal observations, the same live Rust PID and start time, and no child processes for five continuous seconds. It polls every 0.5 seconds and fails after 180 seconds instead of measuring installation work. Selected status, process count and timing evidence is saved in <label>-quiescence.json. No PATH, pip settings or application features are changed. An unavailable result permits the HTTP fixture to proceed; these runs do not test ACME issuance or certificate-provider availability.

Regression tests

Run from the repository root, providing absolute binary and output paths:

python3 scripts/performance/regression.py \
  --binary before=/absolute/path/rust-proxy-before \
  --binary after=/absolute/path/rust-proxy-after \
  --out /absolute/path/results/regression

--groups static,proxy,policy,cache,transform selects groups; all are enabled by default. --workers 4 sets the same TOKIO_WORKER_THREADS for each binary. The report contains individual pass/fail/skip observations and a comparison matrix. An existing failure in the supplied binary remains a failure in its report; it is not silently converted to a successful baseline expectation.

Coverage includes:

• Static GET/HEAD and byte checksums at 1,040 B, 64 KiB and 1 MiB; empty files; ETag/Last-Modified conditional requests; prefix/suffix/open/multipart ranges, If-Range and 416; directory redirect/index/listing; decoded path rejection; deny/inside-root symlinks; precompressed gzip; MIME/download headers; try_files, redirects and the compiled .htaccess header subset.
• Proxy POST/PUT, 1 MiB upload checksums, chunked upload delivery before the last chunk is sent, early response delivery, client disconnect, original status/redirect preservation, failed-POST no replay, request/response header scopes, hop-by-hop stripping, request-ID shape/uniqueness and WebSocket echo.
• Global host/vhost/profile/location ACL composition, upstream suppression on denial, trusted versus spoofed forwarding, untrusted country hints, method and rate policies, hot changes and rejection of invalid configuration.
• Private administration routes, login, session/CSRF and Origin enforcement, Basic Auth, ACL-before-auth, private-origin rejection, retained exchange logging fields and byte accounting.
• Cache global/vhost/route OFF and ON, MISS/HIT, Vary isolation, cookie, Authorization/private/no-store bypass, PURGE, concurrent fills, validator revalidation, static cached ranges/conditionals; transforms OFF/ON with cache OFF/ON; compression negotiation, exclusions and scope/global codec ceilings.

The generic suite does not establish TLS/H2/H3, Tinyauth, real PHP-FPM, real DNS rebinding, concurrent filesystem rename-race safety or long-soak acceptance. Use the dedicated transport tests and existing Rust integration suites for their covered cases. The sequential symlink-target test is explicitly not a concurrent rename-race proof.

TLS, HTTP/2, HTTP/3 and WebSocket interoperability

python3 scripts/performance/transport_regression.py \
  --binary before=/absolute/path/rust-proxy-before \
  --binary after=/absolute/path/rust-proxy-after \
  --out /absolute/path/results/transport

This suite requires aioquic, httpx[http2] and websockets; use --python-deps /absolute/path/pydeps for a separate dependency directory. It verifies the temporary server certificate, negotiated protocol, response status and bytes for TLS H2 and QUIC H3, including HEAD, validators, ranges, cache ON/OFF, 1 MiB bodies and concurrent streams. Dedicated WebSocket checks include bidirectional echo and a normal close handshake. Client protocol self-checks are reported separately from RustProxy acceptance cases. This establishes covered interoperability, not TLS/H2/H3 throughput or production-certificate compatibility.

Two-origin load-balancer regression

python3 scripts/performance/lb_regression.py \
  --binary before=/absolute/path/rust-proxy-before \
  --binary after=/absolute/path/rust-proxy-after \
  --out /absolute/path/results/lb

This separate, small suite sends real requests through RustProxy to two local TCP origins. It verifies equal round-robin allocation, GET failover after a 503 and after a stopped origin refuses connections, and the absence of POST replay. A controlled response pauses after its first body fragment while configuration changes drain or disable its member. New requests must use the other member, the admitted response must finish intact, and its active lease must remain counted until completion and then return to zero. The main HTTP case matrix is unchanged. This suite does not cover sticky sessions, active health probes, DNS rotation or TLS upstream failures.

Finite concurrency correctness stress

python3 scripts/performance/concurrency_regression.py \
  --binary before=/absolute/path/rust-proxy-before \
  --binary after=/absolute/path/rust-proxy-after \
  --nginx /absolute/path/nginx --out /absolute/path/results/concurrency

This separate functional check uses 1, 16 and 128 concurrent client tasks for proxy and static responses at 1,040 B, 64 KiB and 1 MiB. Each finite batch sends max(32, 2 * clients) requests and verifies every status, length and body SHA-256. After the clients finish, backend active-exchange counters must settle to zero. RSS and file descriptors are recorded before the batch and after its drain. These are not peak resource measurements or throughput results. Client tasks start behind a barrier; configured concurrency does not establish that the same number of server exchanges were active simultaneously.

Before/after/nginx benchmark

First verify starts, loopback routing and exact payload hashes without invoking the load generator:

python3 scripts/performance/benchmark.py \
  --binary before=/absolute/path/rust-proxy-before \
  --binary after=/absolute/path/rust-proxy-after \
  --nginx /absolute/path/nginx --wrk /absolute/path/wrk \
  --server-cpus 0-3 --origin-cpus 4-5 --client-cpus 6-8 \
  --out /absolute/path/results/preflight --prepare-only

For a host with CPUs 0–8 available, a reproducible primary run is:

python3 scripts/performance/benchmark.py \
  --binary before=/absolute/path/rust-proxy-before \
  --binary after=/absolute/path/rust-proxy-after \
  --nginx /absolute/path/nginx --wrk /absolute/path/wrk \
  --out /absolute/path/results/primary \
  --workers 4 --origin-workers 2 --client-threads 4 \
  --server-cpus 0-3 --origin-cpus 4-5 --client-cpus 6-8 \
  --concurrency 64 --sizes small,large,huge --modes proxy,static \
  --duration 15 --warmup 3 --repeats 5

Choose explicit, nonempty and disjoint CPU sets allowed in the run environment; adapt all three sets on a smaller host. Frontends use the same CPU set and worker count, the shared origin uses another set, and the generator uses a third. The harness checks each process/thread's actual affinity and refuses unavailable resource counters. Four wrk threads share the three client CPUs in this example. The structured result records thread count and effective connection count; at 64 connections, four threads keep exactly 64 connections, avoiding wrk's integer division of connections between threads. Only one frontend is loaded at a time. Stop builds, other tests and profilers during throughput measurement.

The primary run takes approximately 27 minutes: six scenarios, three variants, five samples, and 18 seconds per sample including excluded warm-up. To explore concurrency, repeat in a fresh output directory with --concurrency 1,16,64,128. More scenarios intentionally take more time. Short sanity checks require the explicit --exploratory option and cannot satisfy the >=5 samples / >=15-second measurement gate.

Cache is explicitly disabled for the primary benchmark. Transforms retain their disabled defaults. Identity encoding is requested. Logging history, analytics, request IDs, origin policy, body accounting, health/lease plumbing and normal application functionality remain active. nginx uses sendfile on, HTTP/1.1, keep-alive and streaming proxy delivery. The generated exact configurations are saved, so this intentional difference in nginx/application features is visible. No cache-disabled claim is inferred from a cache-on speedup.

Optional keep-alive tuning uses --max-idle label=64. Keep this as a separately labelled experiment. The harness does not turn off bounded pool recycling.

Recorded evidence and interpretation

• schedule.json: complete deterministic crossover order, rotating and reversing variants between repetitions. No best-run selection.
• preflight.json, fixture/payloads.json, binary/version/config metadata: exact status, body SHA-256, payload sizes and inputs.
• Per-sample exact status/body checks before and after the measured interval, plus a wire-bytes-per-request sanity check, detect wrong content and gross transfer-size anomalies. These do not hash every response inside wrk.
• raw/*.txt and raw/*.json: unmodified wrk output and structured requests, duration, bytes, RPS, p50/p95/p99, effective concurrency, and connect/read/write/timeout/status errors.
• raw/*-resources.json: timestamped /proc samples for each server process and its children, plus the separate origin process group.
• raw/*-affinity.json: observed process/thread CPU masks for the front end, origin and generator.
• runs.csv/json, summary.csv/json: all samples, medians, RPS range and coefficient of variation, latency medians, server CPU time per completed request, peak/end RSS and file descriptors.
• status/*.json: numeric drop and pool-generation/rotation counters read outside each measured interval. Warm-up output is kept separately.
• completion.json and SHA256SUMS: completion state and hashes of evidence. Any generator/network/status error stops the run with failure.json; partial data remains available and is not presented as completed acceptance.

Use (after_median / before_median - 1) * 100 for throughput improvement and after_median / nginx_median * 100 for relative nginx throughput. Rebuilding both revisions with one compiler, target and resolved lockfile is required for source-change attribution. Compare the supplied executable separately: build flags and dependency versions can change its performance.

These are same-host, closed-loop HTTP/1.1 diagnostics. They do not establish production capacity, TLS/H2/H3 performance, an open-loop latency SLA, allocator rates, CPU cycles or 24-hour resource stability. wrk latency follows wrk's own histogram/correction behavior. RSS and descriptor peaks are sampled observations, so shorter spikes may be missed. Summed nginx RSS can double-count shared pages. Origin or generator CPU saturation can limit observed throughput; inspect their resource data and move the generator/origin to separate machines for release measurements. Run profilers separately so profiling overhead does not enter the headline A/B throughput numbers.

Mixed-traffic resource soak

Run this in its own phase, after correctness and throughput checks:

python3 scripts/performance/soak.py \
  --binary /absolute/path/rust-proxy-after --label after \
  --nginx /absolute/path/nginx --out /absolute/path/results/soak \
  --duration 1800 --rate 100 --clients 8 \
  --workers 4 --origin-workers 2 --server-cpus 0-3 --origin-cpus 4-5 \
  --toggle-seconds 120 --sample-seconds 5 --abort-every 101

The bounded client executor alternates proxy/static and 1,040 B/64 KiB/1 MiB payloads, verifies full bytes and occasional HEAD/range semantics, closes some clients after an initial body prefix, and applies live cache OFF/ON changes. Every workload request opens a new frontend HTTP/1.1 connection; upstream keepalive retains its configured defaults. This exercises connection churn. The primary wrk benchmark separately uses persistent frontend connections. The nominal request rate is paced; scheduler delays and actual completion rate are reported instead of hiding an under-driven client. Pending jobs are bounded to twice the client count. The script restores base configuration, drains and rechecks both listeners before shutdown. After each cache apply, two separate GETs per source verify exact contents and require a HIT on the second GET when enabled; globally disabled cache must expose no X-Cache header. These probe requests are recorded separately from paced traffic. Backend active-exchange counters must settle to zero after the final drain and cooldown.

Five-second resource/status records, body/status failures, observed cache-status counts, cache transitions/probes, RSS/FD trend and completed traffic totals are retained. Controlled client closes mean that the client closes after reading a prefix. The server may already have sent the complete response, so those counts are not presented as proven server-side cancellations. Memory trend is descriptive: switching caching and allocator retention affect RSS. This is a 30-minute resource gate, not a 24-hour leak proof. A short --exploratory --duration ... sanity run stays explicitly outside that gate.