RustProxy/scripts/performance/README.md

# Live performance and regression harness

These scripts exercise actual RustProxy executables through loopback sockets.
They do not simulate Rust request handling. The Python server in `peer.py` is a
controllable test **upstream**, used to verify streaming, non-replayable writes,
headers, cache policy and WebSocket frames. Performance measurements use nginx
as the shared upstream instead of Python.

## DEV 1 KiB follow-up on 0.34.0 (newer source)

See `docs/PROXY_1KB_HOTPATH_DEV_0.34.0.md`. The previous executable's 4-vs-64
idle-connection tuning experiment is real loopback evidence, but the *edited*
source has not been compiled. Its new default is 64 for new vhosts, whereas
explicitly saved `4` remains `4`. Keep the idle limit equal between before/after
*code* benchmarks; measure configuration tuning in a separate labelled series.
The source release is not eligible for a claim of 100k RPS or 80% nginx yet.

## Reverse-proxy optimization revision 0.34.0

After compiling the refreshed candidate, measure the same unmodified 0.33.8
binary and nginx reference as before, without turning off logging, request
IDs, ACL, cache-policy enforcement or LB health. Profile request-size 1,040 B
first. Before RPS runs execute full proxy, LB, transport and concurrency
regressions (including single-member capacity, recovery, sticky cookies,
replays on safe methods, rejected non-replayable writes and H2 connection
shutdown). Treat `--max-idle` tuning and recycles as separately labelled
experiments rather than mixing them into the primary comparison. There are no
new measured RPS figures in the source candidate.

## Follow-up RPS candidate on 0.34.0

`docs/PROXY_RPS_0.34.0.md` tracks the code changes and their regressions.
Compile before/after binaries from the two 0.34.0 source revisions and use
the paired benchmark below; a live run against the older binary cannot
establish a measured RPS improvement for the uncompiled candidate. Keep
`max_idle_per_host`, recycling, HTTP1/H2 settings and all application security
and logging features equal between runs. Treat any configuration tuning as a
separate experiment. Profile the resulting binary with `perf` only after
correctness checks pass.

## Offline cache-retention model for 0.34.0

`python3 scripts/performance/simulate_cache_034.py` compares cache invalidation
policies under a deterministic synthetic sequence of vhost saves and uniform
cacheable GETs. The model contains **no RustProxy server**, no timing and no
network traffic; its percentages are **not measured RPS gains**. See
`docs/TEST_REPORT_0.34.0.md` for assumptions and the live acceptance gap.

## Requirements

- Linux with readable `/proc`, Python 3.10+, executable RustProxy binaries.
- nginx and wrk for performance comparisons. Both paths are explicit; no
  system-wide installation is required.
- `taskset` from util-linux for the benchmark's explicit CPU affinity and for
  optional affinity settings in the other scripts.
- Optional Python `brotli` and `zstandard` decoders increase compression
  acceptance coverage. A missing optional decoder is reported as **skipped**.
- Permission to bind unprivileged loopback sockets and start local processes.
  In a sandbox without sockets or `/proc`, run in the approved local execution
  environment. Do not replace real runtime results with a peer self-test.

Each run creates a new private temporary data directory and initializes its own
administrator with a random password supplied only over stdin. Configurations
are applied through RustProxy's existing authenticated CLI/API and listener
changes receive a restart. No historical SQLite database is copied or edited.
TLS keys, databases and passwords are never written to the evidence directory.
Temporary state is outside the evidence tree and is removed on normal exit.
`fixture-cleanup.json` records the cleanup check. Interrupted runs can leave an
unrelated `/tmp/rustproxy-perf-*` directory; do not include it in a release.

Before timed benchmark, soak or syscall-profile work, `wait_quiescent()` waits
for the existing authenticated ACME runtime API to report `ready` or
`unavailable`. A fresh RustProxy data directory can start a Python/Certbot
environment bootstrap even when no certificates have an issuer. The gate
requires no configured issuers, jobs or renewal observations, the same live Rust
PID and start time, and no child processes for five continuous seconds. It polls
every 0.5 seconds and fails after 180 seconds instead of measuring installation
work. Selected status, process count and timing evidence is saved in
`<label>-quiescence.json`. No PATH, pip settings or application features are
changed. An `unavailable` result permits the HTTP fixture to proceed; these runs
do not test ACME issuance or certificate-provider availability.

## Regression tests

Run from the repository root, providing absolute binary and output paths:

```bash
python3 scripts/performance/regression.py \
  --binary before=/absolute/path/rust-proxy-before \
  --binary after=/absolute/path/rust-proxy-after \
  --out /absolute/path/results/regression
```

`--groups static,proxy,policy,cache,transform` selects groups; all are enabled by
default. `--workers 4` sets the same `TOKIO_WORKER_THREADS` for each binary.
The report contains individual pass/fail/skip observations and a comparison
matrix. An existing failure in the supplied binary remains a failure in its
report; it is not silently converted to a successful baseline expectation.

Coverage includes:

- Static GET/HEAD and byte checksums at 1,040 B, 64 KiB and 1 MiB; empty files;
  ETag/Last-Modified conditional requests; prefix/suffix/open/multipart ranges,
  If-Range and 416; directory redirect/index/listing; decoded path rejection;
  deny/inside-root symlinks; precompressed gzip; MIME/download headers;
  `try_files`, redirects and the compiled `.htaccess` header subset.
- Proxy POST/PUT, 1 MiB upload checksums, chunked upload delivery **before the
  last chunk is sent**, early response delivery, client disconnect, original
  status/redirect preservation, failed-POST no replay, request/response header
  scopes, hop-by-hop stripping, request-ID shape/uniqueness and WebSocket echo.
- Global host/vhost/profile/location ACL composition, upstream suppression on
  denial, trusted versus spoofed forwarding, untrusted country hints, method
  and rate policies, hot changes and rejection of invalid configuration.
- Private administration routes, login, session/CSRF and Origin enforcement,
  Basic Auth, ACL-before-auth, private-origin rejection, retained exchange
  logging fields and byte accounting.
- Cache global/vhost/route OFF and ON, MISS/HIT, Vary isolation, cookie,
  Authorization/private/no-store bypass, PURGE, concurrent fills, validator
  revalidation, static cached ranges/conditionals; transforms OFF/ON with cache
  OFF/ON; compression negotiation, exclusions and scope/global codec ceilings.

The generic suite does not establish TLS/H2/H3, Tinyauth, real PHP-FPM, real DNS
rebinding, concurrent filesystem rename-race safety or long-soak acceptance.
Use the dedicated transport tests and existing Rust integration suites for
their covered cases. The sequential symlink-target test is explicitly not a
concurrent rename-race proof.

## TLS, HTTP/2, HTTP/3 and WebSocket interoperability

```bash
python3 scripts/performance/transport_regression.py \
  --binary before=/absolute/path/rust-proxy-before \
  --binary after=/absolute/path/rust-proxy-after \
  --out /absolute/path/results/transport
```

This suite requires `aioquic`, `httpx[http2]` and `websockets`; use
`--python-deps /absolute/path/pydeps` for a separate dependency directory.
It verifies the temporary server certificate, negotiated protocol, response
status and bytes for TLS H2 and QUIC H3, including HEAD, validators, ranges,
cache ON/OFF, 1 MiB bodies and concurrent streams. Dedicated WebSocket checks
include bidirectional echo and a normal close handshake. Client protocol
self-checks are reported separately from RustProxy acceptance cases.
This establishes covered interoperability, not TLS/H2/H3 throughput or
production-certificate compatibility.

## Two-origin load-balancer regression

```bash
python3 scripts/performance/lb_regression.py \
  --binary before=/absolute/path/rust-proxy-before \
  --binary after=/absolute/path/rust-proxy-after \
  --out /absolute/path/results/lb
```

This separate, small suite sends real requests through RustProxy to two local
TCP origins. It verifies equal round-robin allocation, GET failover after a 503
and after a stopped origin refuses connections, and the absence of POST replay.
A controlled response pauses after its first body fragment while configuration
changes drain or disable its member. New requests must use the other member,
the admitted response must finish intact, and its active lease must remain
counted until completion and then return to zero. The main HTTP case matrix is
unchanged. This suite does not cover sticky sessions, active health probes,
DNS rotation or TLS upstream failures.

## Finite concurrency correctness stress

```bash
python3 scripts/performance/concurrency_regression.py \
  --binary before=/absolute/path/rust-proxy-before \
  --binary after=/absolute/path/rust-proxy-after \
  --nginx /absolute/path/nginx --out /absolute/path/results/concurrency
```

This separate functional check uses 1, 16 and 128 concurrent client tasks for
proxy and static responses at 1,040 B, 64 KiB and 1 MiB. Each finite batch sends
`max(32, 2 * clients)` requests and verifies every status, length and body SHA-256.
After the clients finish, backend active-exchange counters must settle to zero.
RSS and file descriptors are recorded before the batch and after its drain.
These are not peak resource measurements or throughput results. Client tasks
start behind a barrier; configured concurrency does not establish that the same
number of server exchanges were active simultaneously.

## Before/after/nginx benchmark

First verify starts, loopback routing and exact payload hashes without invoking
the load generator:

```bash
python3 scripts/performance/benchmark.py \
  --binary before=/absolute/path/rust-proxy-before \
  --binary after=/absolute/path/rust-proxy-after \
  --nginx /absolute/path/nginx --wrk /absolute/path/wrk \
  --server-cpus 0-3 --origin-cpus 4-5 --client-cpus 6-8 \
  --out /absolute/path/results/preflight --prepare-only
```

For a host with CPUs 0–8 available, a reproducible primary run is:

```bash
python3 scripts/performance/benchmark.py \
  --binary before=/absolute/path/rust-proxy-before \
  --binary after=/absolute/path/rust-proxy-after \
  --nginx /absolute/path/nginx --wrk /absolute/path/wrk \
  --out /absolute/path/results/primary \
  --workers 4 --origin-workers 2 --client-threads 4 \
  --server-cpus 0-3 --origin-cpus 4-5 --client-cpus 6-8 \
  --concurrency 64 --sizes small,large,huge --modes proxy,static \
  --duration 15 --warmup 3 --repeats 5
```

Choose explicit, nonempty and disjoint CPU sets allowed in the run environment;
adapt all three sets on a smaller host. Frontends use the same CPU set and worker
count, the shared origin uses another set, and the generator uses a third. The
harness checks each process/thread's actual affinity and refuses unavailable
resource counters. Four wrk threads share the three client CPUs in this example.
The structured result records thread count and effective connection count; at
64 connections, four threads keep exactly 64 connections, avoiding wrk's integer
division of connections between threads. Only one frontend is loaded at a time.
Stop builds, other tests and profilers during throughput measurement.

The primary run takes approximately 27 minutes: six scenarios, three variants,
five samples, and 18 seconds per sample including excluded warm-up. To explore
concurrency, repeat in a fresh output directory with
`--concurrency 1,16,64,128`. More scenarios intentionally take more time.
Short sanity checks require the explicit `--exploratory` option and cannot
satisfy the >=5 samples / >=15-second measurement gate.

Cache is explicitly disabled for the primary benchmark. Transforms retain their
disabled defaults. Identity encoding is requested. Logging history, analytics,
request IDs, origin policy, body accounting, health/lease plumbing and normal
application functionality remain active. nginx uses `sendfile on`, HTTP/1.1,
keep-alive and streaming proxy delivery. The generated exact configurations are
saved, so this intentional difference in nginx/application features is visible.
No cache-disabled claim is inferred from a cache-on speedup.

Optional keep-alive tuning uses `--max-idle label=64`. Keep this as a separately
labelled experiment. The harness does not turn off bounded pool recycling.

## Recorded evidence and interpretation

- `schedule.json`: complete deterministic crossover order, rotating and
  reversing variants between repetitions. No best-run selection.
- `preflight.json`, `fixture/payloads.json`, binary/version/config metadata:
  exact status, body SHA-256, payload sizes and inputs.
- Per-sample exact status/body checks before and after the measured interval,
  plus a wire-bytes-per-request sanity check, detect wrong content and gross
  transfer-size anomalies. These do not hash every response inside wrk.
- `raw/*.txt` and `raw/*.json`: unmodified wrk output and structured requests,
  duration, bytes, RPS, p50/p95/p99, effective concurrency, and
  connect/read/write/timeout/status errors.
- `raw/*-resources.json`: timestamped `/proc` samples for each server process
  and its children, plus the separate origin process group.
- `raw/*-affinity.json`: observed process/thread CPU masks for the front end,
  origin and generator.
- `runs.csv/json`, `summary.csv/json`: all samples, medians, RPS range and
  coefficient of variation, latency medians, server CPU time per completed
  request, peak/end RSS and file descriptors.
- `status/*.json`: numeric drop and pool-generation/rotation counters read
  outside each measured interval. Warm-up output is kept separately.
- `completion.json` and `SHA256SUMS`: completion state and hashes of evidence.
  Any generator/network/status error stops the run with `failure.json`; partial
  data remains available and is not presented as completed acceptance.

Use `(after_median / before_median - 1) * 100` for throughput improvement and
`after_median / nginx_median * 100` for relative nginx throughput. Rebuilding
both revisions with one compiler, target and resolved lockfile is required for
source-change attribution. Compare the supplied executable separately: build
flags and dependency versions can change its performance.

These are same-host, closed-loop HTTP/1.1 diagnostics. They do not establish
production capacity, TLS/H2/H3 performance, an open-loop latency SLA, allocator
rates, CPU cycles or 24-hour resource stability. wrk latency follows wrk's own
histogram/correction behavior. RSS and descriptor peaks are sampled observations,
so shorter spikes may be missed. Summed nginx RSS can double-count shared pages.
Origin or generator CPU saturation can limit observed throughput; inspect their
resource data and move the generator/origin to separate machines for release
measurements. Run profilers separately so profiling overhead does not enter the
headline A/B throughput numbers.

## Mixed-traffic resource soak

Run this in its own phase, after correctness and throughput checks:

```bash
python3 scripts/performance/soak.py \
  --binary /absolute/path/rust-proxy-after --label after \
  --nginx /absolute/path/nginx --out /absolute/path/results/soak \
  --duration 1800 --rate 100 --clients 8 \
  --workers 4 --origin-workers 2 --server-cpus 0-3 --origin-cpus 4-5 \
  --toggle-seconds 120 --sample-seconds 5 --abort-every 101
```

The bounded client executor alternates proxy/static and 1,040 B/64 KiB/1 MiB
payloads, verifies full bytes and occasional HEAD/range semantics, closes some
clients after an initial body prefix, and applies live cache OFF/ON changes.
Every workload request opens a new frontend HTTP/1.1 connection; upstream
keepalive retains its configured defaults. This exercises connection churn.
The primary wrk benchmark separately uses persistent frontend connections.
The nominal request rate is paced; scheduler delays and actual completion rate
are reported instead of hiding an under-driven client. Pending jobs are bounded
to twice the client count. The script restores base configuration, drains and
rechecks both listeners before shutdown. After each cache apply, two separate
GETs per source verify exact contents and require a HIT on the second GET when
enabled; globally disabled cache must expose no X-Cache header. These probe
requests are recorded separately from paced traffic. Backend active-exchange
counters must settle to zero after the final drain and cooldown.

Five-second resource/status records, body/status failures, observed cache-status
counts, cache transitions/probes, RSS/FD trend and completed traffic totals are
retained. Controlled client closes mean that the client closes after reading a
prefix. The server may already have sent the complete response, so those counts
are not presented as proven server-side cancellations.
Memory trend is descriptive: switching caching and allocator retention affect
RSS. This is a 30-minute resource gate, not a 24-hour leak proof. A short
`--exploratory --duration ...` sanity run stays explicitly outside that gate.

## 0.34.0 hypothesis-only sendfile/nginx gap simulator

```bash
python3 scripts/performance/simulate_nginx_gap_034.py --output /tmp/nginx-gap.json
```

This is **not** a benchmark of RustProxy 0.34.0. It reproduces measured
0.33.8/nginx medians with cache OFF and applies explicitly hypothetical
Amdahl service-time reductions (WWW and reverse proxy separately). A real
RustProxy binary is required to validate the code path, detect regressions
and measure latency/RPS against nginx under identical conditions.