v0.33.8

scripts/performance/README.md

@@ -0,0 +1,276 @@
+# Live performance and regression harness
+
+These scripts exercise actual RustProxy executables through loopback sockets.
+They do not simulate Rust request handling. The Python server in `peer.py` is a
+controllable test **upstream**, used to verify streaming, non-replayable writes,
+headers, cache policy and WebSocket frames. Performance measurements use nginx
+as the shared upstream instead of Python.
+
+## Requirements
+
+- Linux with readable `/proc`, Python 3.10+, executable RustProxy binaries.
+- nginx and wrk for performance comparisons. Both paths are explicit; no
+  system-wide installation is required.
+- `taskset` from util-linux for the benchmark's explicit CPU affinity and for
+  optional affinity settings in the other scripts.
+- Optional Python `brotli` and `zstandard` decoders increase compression
+  acceptance coverage. A missing optional decoder is reported as **skipped**.
+- Permission to bind unprivileged loopback sockets and start local processes.
+  In a sandbox without sockets or `/proc`, run in the approved local execution
+  environment. Do not replace real runtime results with a peer self-test.
+
+Each run creates a new private temporary data directory and initializes its own
+administrator with a random password supplied only over stdin. Configurations
+are applied through RustProxy's existing authenticated CLI/API and listener
+changes receive a restart. No historical SQLite database is copied or edited.
+TLS keys, databases and passwords are never written to the evidence directory.
+Temporary state is outside the evidence tree and is removed on normal exit.
+`fixture-cleanup.json` records the cleanup check. Interrupted runs can leave an
+unrelated `/tmp/rustproxy-perf-*` directory; do not include it in a release.
+
+Before timed benchmark, soak or syscall-profile work, `wait_quiescent()` waits
+for the existing authenticated ACME runtime API to report `ready` or
+`unavailable`. A fresh RustProxy data directory can start a Python/Certbot
+environment bootstrap even when no certificates have an issuer. The gate
+requires no configured issuers, jobs or renewal observations, the same live Rust
+PID and start time, and no child processes for five continuous seconds. It polls
+every 0.5 seconds and fails after 180 seconds instead of measuring installation
+work. Selected status, process count and timing evidence is saved in
+`<label>-quiescence.json`. No PATH, pip settings or application features are
+changed. An `unavailable` result permits the HTTP fixture to proceed; these runs
+do not test ACME issuance or certificate-provider availability.
+
+## Regression tests
+
+Run from the repository root, providing absolute binary and output paths:
+
+```bash
+python3 scripts/performance/regression.py \
+  --binary before=/absolute/path/rust-proxy-before \
+  --binary after=/absolute/path/rust-proxy-after \
+  --out /absolute/path/results/regression
+```
+
+`--groups static,proxy,policy,cache,transform` selects groups; all are enabled by
+default. `--workers 4` sets the same `TOKIO_WORKER_THREADS` for each binary.
+The report contains individual pass/fail/skip observations and a comparison
+matrix. An existing failure in the supplied binary remains a failure in its
+report; it is not silently converted to a successful baseline expectation.
+
+Coverage includes:
+
+- Static GET/HEAD and byte checksums at 1,040 B, 64 KiB and 1 MiB; empty files;
+  ETag/Last-Modified conditional requests; prefix/suffix/open/multipart ranges,
+  If-Range and 416; directory redirect/index/listing; decoded path rejection;
+  deny/inside-root symlinks; precompressed gzip; MIME/download headers;
+  `try_files`, redirects and the compiled `.htaccess` header subset.
+- Proxy POST/PUT, 1 MiB upload checksums, chunked upload delivery **before the
+  last chunk is sent**, early response delivery, client disconnect, original
+  status/redirect preservation, failed-POST no replay, request/response header
+  scopes, hop-by-hop stripping, request-ID shape/uniqueness and WebSocket echo.
+- Global host/vhost/profile/location ACL composition, upstream suppression on
+  denial, trusted versus spoofed forwarding, untrusted country hints, method
+  and rate policies, hot changes and rejection of invalid configuration.
+- Private administration routes, login, session/CSRF and Origin enforcement,
+  Basic Auth, ACL-before-auth, private-origin rejection, retained exchange
+  logging fields and byte accounting.
+- Cache global/vhost/route OFF and ON, MISS/HIT, Vary isolation, cookie,
+  Authorization/private/no-store bypass, PURGE, concurrent fills, validator
+  revalidation, static cached ranges/conditionals; transforms OFF/ON with cache
+  OFF/ON; compression negotiation, exclusions and scope/global codec ceilings.
+
+The generic suite does not establish TLS/H2/H3, Tinyauth, real PHP-FPM, real DNS
+rebinding, concurrent filesystem rename-race safety or long-soak acceptance.
+Use the dedicated transport tests and existing Rust integration suites for
+their covered cases. The sequential symlink-target test is explicitly not a
+concurrent rename-race proof.
+
+## TLS, HTTP/2, HTTP/3 and WebSocket interoperability
+
+```bash
+python3 scripts/performance/transport_regression.py \
+  --binary before=/absolute/path/rust-proxy-before \
+  --binary after=/absolute/path/rust-proxy-after \
+  --out /absolute/path/results/transport
+```
+
+This suite requires `aioquic`, `httpx[http2]` and `websockets`; use
+`--python-deps /absolute/path/pydeps` for a separate dependency directory.
+It verifies the temporary server certificate, negotiated protocol, response
+status and bytes for TLS H2 and QUIC H3, including HEAD, validators, ranges,
+cache ON/OFF, 1 MiB bodies and concurrent streams. Dedicated WebSocket checks
+include bidirectional echo and a normal close handshake. Client protocol
+self-checks are reported separately from RustProxy acceptance cases.
+This establishes covered interoperability, not TLS/H2/H3 throughput or
+production-certificate compatibility.
+
+## Two-origin load-balancer regression
+
+```bash
+python3 scripts/performance/lb_regression.py \
+  --binary before=/absolute/path/rust-proxy-before \
+  --binary after=/absolute/path/rust-proxy-after \
+  --out /absolute/path/results/lb
+```
+
+This separate, small suite sends real requests through RustProxy to two local
+TCP origins. It verifies equal round-robin allocation, GET failover after a 503
+and after a stopped origin refuses connections, and the absence of POST replay.
+A controlled response pauses after its first body fragment while configuration
+changes drain or disable its member. New requests must use the other member,
+the admitted response must finish intact, and its active lease must remain
+counted until completion and then return to zero. The main HTTP case matrix is
+unchanged. This suite does not cover sticky sessions, active health probes,
+DNS rotation or TLS upstream failures.
+
+## Finite concurrency correctness stress
+
+```bash
+python3 scripts/performance/concurrency_regression.py \
+  --binary before=/absolute/path/rust-proxy-before \
+  --binary after=/absolute/path/rust-proxy-after \
+  --nginx /absolute/path/nginx --out /absolute/path/results/concurrency
+```
+
+This separate functional check uses 1, 16 and 128 concurrent client tasks for
+proxy and static responses at 1,040 B, 64 KiB and 1 MiB. Each finite batch sends
+`max(32, 2 * clients)` requests and verifies every status, length and body SHA-256.
+After the clients finish, backend active-exchange counters must settle to zero.
+RSS and file descriptors are recorded before the batch and after its drain.
+These are not peak resource measurements or throughput results. Client tasks
+start behind a barrier; configured concurrency does not establish that the same
+number of server exchanges were active simultaneously.
+
+## Before/after/nginx benchmark
+
+First verify starts, loopback routing and exact payload hashes without invoking
+the load generator:
+
+```bash
+python3 scripts/performance/benchmark.py \
+  --binary before=/absolute/path/rust-proxy-before \
+  --binary after=/absolute/path/rust-proxy-after \
+  --nginx /absolute/path/nginx --wrk /absolute/path/wrk \
+  --server-cpus 0-3 --origin-cpus 4-5 --client-cpus 6-8 \
+  --out /absolute/path/results/preflight --prepare-only
+```
+
+For a host with CPUs 0–8 available, a reproducible primary run is:
+
+```bash
+python3 scripts/performance/benchmark.py \
+  --binary before=/absolute/path/rust-proxy-before \
+  --binary after=/absolute/path/rust-proxy-after \
+  --nginx /absolute/path/nginx --wrk /absolute/path/wrk \
+  --out /absolute/path/results/primary \
+  --workers 4 --origin-workers 2 --client-threads 4 \
+  --server-cpus 0-3 --origin-cpus 4-5 --client-cpus 6-8 \
+  --concurrency 64 --sizes small,large,huge --modes proxy,static \
+  --duration 15 --warmup 3 --repeats 5
+```
+
+Choose explicit, nonempty and disjoint CPU sets allowed in the run environment;
+adapt all three sets on a smaller host. Frontends use the same CPU set and worker
+count, the shared origin uses another set, and the generator uses a third. The
+harness checks each process/thread's actual affinity and refuses unavailable
+resource counters. Four wrk threads share the three client CPUs in this example.
+The structured result records thread count and effective connection count; at
+64 connections, four threads keep exactly 64 connections, avoiding wrk's integer
+division of connections between threads. Only one frontend is loaded at a time.
+Stop builds, other tests and profilers during throughput measurement.
+
+The primary run takes approximately 27 minutes: six scenarios, three variants,
+five samples, and 18 seconds per sample including excluded warm-up. To explore
+concurrency, repeat in a fresh output directory with
+`--concurrency 1,16,64,128`. More scenarios intentionally take more time.
+Short sanity checks require the explicit `--exploratory` option and cannot
+satisfy the >=5 samples / >=15-second measurement gate.
+
+Cache is explicitly disabled for the primary benchmark. Transforms retain their
+disabled defaults. Identity encoding is requested. Logging history, analytics,
+request IDs, origin policy, body accounting, health/lease plumbing and normal
+application functionality remain active. nginx uses `sendfile on`, HTTP/1.1,
+keep-alive and streaming proxy delivery. The generated exact configurations are
+saved, so this intentional difference in nginx/application features is visible.
+No cache-disabled claim is inferred from a cache-on speedup.
+
+Optional keep-alive tuning uses `--max-idle label=64`. Keep this as a separately
+labelled experiment. The harness does not turn off bounded pool recycling.
+
+## Recorded evidence and interpretation
+
+- `schedule.json`: complete deterministic crossover order, rotating and
+  reversing variants between repetitions. No best-run selection.
+- `preflight.json`, `fixture/payloads.json`, binary/version/config metadata:
+  exact status, body SHA-256, payload sizes and inputs.
+- Per-sample exact status/body checks before and after the measured interval,
+  plus a wire-bytes-per-request sanity check, detect wrong content and gross
+  transfer-size anomalies. These do not hash every response inside wrk.
+- `raw/*.txt` and `raw/*.json`: unmodified wrk output and structured requests,
+  duration, bytes, RPS, p50/p95/p99, effective concurrency, and
+  connect/read/write/timeout/status errors.
+- `raw/*-resources.json`: timestamped `/proc` samples for each server process
+  and its children, plus the separate origin process group.
+- `raw/*-affinity.json`: observed process/thread CPU masks for the front end,
+  origin and generator.
+- `runs.csv/json`, `summary.csv/json`: all samples, medians, RPS range and
+  coefficient of variation, latency medians, server CPU time per completed
+  request, peak/end RSS and file descriptors.
+- `status/*.json`: numeric drop and pool-generation/rotation counters read
+  outside each measured interval. Warm-up output is kept separately.
+- `completion.json` and `SHA256SUMS`: completion state and hashes of evidence.
+  Any generator/network/status error stops the run with `failure.json`; partial
+  data remains available and is not presented as completed acceptance.
+
+Use `(after_median / before_median - 1) * 100` for throughput improvement and
+`after_median / nginx_median * 100` for relative nginx throughput. Rebuilding
+both revisions with one compiler, target and resolved lockfile is required for
+source-change attribution. Compare the supplied executable separately: build
+flags and dependency versions can change its performance.
+
+These are same-host, closed-loop HTTP/1.1 diagnostics. They do not establish
+production capacity, TLS/H2/H3 performance, an open-loop latency SLA, allocator
+rates, CPU cycles or 24-hour resource stability. wrk latency follows wrk's own
+histogram/correction behavior. RSS and descriptor peaks are sampled observations,
+so shorter spikes may be missed. Summed nginx RSS can double-count shared pages.
+Origin or generator CPU saturation can limit observed throughput; inspect their
+resource data and move the generator/origin to separate machines for release
+measurements. Run profilers separately so profiling overhead does not enter the
+headline A/B throughput numbers.
+
+## Mixed-traffic resource soak
+
+Run this in its own phase, after correctness and throughput checks:
+
+```bash
+python3 scripts/performance/soak.py \
+  --binary /absolute/path/rust-proxy-after --label after \
+  --nginx /absolute/path/nginx --out /absolute/path/results/soak \
+  --duration 1800 --rate 100 --clients 8 \
+  --workers 4 --origin-workers 2 --server-cpus 0-3 --origin-cpus 4-5 \
+  --toggle-seconds 120 --sample-seconds 5 --abort-every 101
+```
+
+The bounded client executor alternates proxy/static and 1,040 B/64 KiB/1 MiB
+payloads, verifies full bytes and occasional HEAD/range semantics, closes some
+clients after an initial body prefix, and applies live cache OFF/ON changes.
+Every workload request opens a new frontend HTTP/1.1 connection; upstream
+keepalive retains its configured defaults. This exercises connection churn.
+The primary wrk benchmark separately uses persistent frontend connections.
+The nominal request rate is paced; scheduler delays and actual completion rate
+are reported instead of hiding an under-driven client. Pending jobs are bounded
+to twice the client count. The script restores base configuration, drains and
+rechecks both listeners before shutdown. After each cache apply, two separate
+GETs per source verify exact contents and require a HIT on the second GET when
+enabled; globally disabled cache must expose no X-Cache header. These probe
+requests are recorded separately from paced traffic. Backend active-exchange
+counters must settle to zero after the final drain and cooldown.
+
+Five-second resource/status records, body/status failures, observed cache-status
+counts, cache transitions/probes, RSS/FD trend and completed traffic totals are
+retained. Controlled client closes mean that the client closes after reading a
+prefix. The server may already have sent the complete response, so those counts
+are not presented as proven server-side cancellations.
+Memory trend is descriptive: switching caching and allocator retention affect
+RSS. This is a 30-minute resource gate, not a 24-hour leak proof. A short
+`--exploratory --duration ...` sanity run stays explicitly outside that gate.