v0.33.8

scripts/performance/README.md

@@ -0,0 +1,276 @@
+# Live performance and regression harness
+
+These scripts exercise actual RustProxy executables through loopback sockets.
+They do not simulate Rust request handling. The Python server in `peer.py` is a
+controllable test **upstream**, used to verify streaming, non-replayable writes,
+headers, cache policy and WebSocket frames. Performance measurements use nginx
+as the shared upstream instead of Python.
+
+## Requirements
+
+- Linux with readable `/proc`, Python 3.10+, executable RustProxy binaries.
+- nginx and wrk for performance comparisons. Both paths are explicit; no
+  system-wide installation is required.
+- `taskset` from util-linux for the benchmark's explicit CPU affinity and for
+  optional affinity settings in the other scripts.
+- Optional Python `brotli` and `zstandard` decoders increase compression
+  acceptance coverage. A missing optional decoder is reported as **skipped**.
+- Permission to bind unprivileged loopback sockets and start local processes.
+  In a sandbox without sockets or `/proc`, run in the approved local execution
+  environment. Do not replace real runtime results with a peer self-test.
+
+Each run creates a new private temporary data directory and initializes its own
+administrator with a random password supplied only over stdin. Configurations
+are applied through RustProxy's existing authenticated CLI/API and listener
+changes receive a restart. No historical SQLite database is copied or edited.
+TLS keys, databases and passwords are never written to the evidence directory.
+Temporary state is outside the evidence tree and is removed on normal exit.
+`fixture-cleanup.json` records the cleanup check. Interrupted runs can leave an
+unrelated `/tmp/rustproxy-perf-*` directory; do not include it in a release.
+
+Before timed benchmark, soak or syscall-profile work, `wait_quiescent()` waits
+for the existing authenticated ACME runtime API to report `ready` or
+`unavailable`. A fresh RustProxy data directory can start a Python/Certbot
+environment bootstrap even when no certificates have an issuer. The gate
+requires no configured issuers, jobs or renewal observations, the same live Rust
+PID and start time, and no child processes for five continuous seconds. It polls
+every 0.5 seconds and fails after 180 seconds instead of measuring installation
+work. Selected status, process count and timing evidence is saved in
+`<label>-quiescence.json`. No PATH, pip settings or application features are
+changed. An `unavailable` result permits the HTTP fixture to proceed; these runs
+do not test ACME issuance or certificate-provider availability.
+
+## Regression tests
+
+Run from the repository root, providing absolute binary and output paths:
+
+```bash
+python3 scripts/performance/regression.py \
+  --binary before=/absolute/path/rust-proxy-before \
+  --binary after=/absolute/path/rust-proxy-after \
+  --out /absolute/path/results/regression
+```
+
+`--groups static,proxy,policy,cache,transform` selects groups; all are enabled by
+default. `--workers 4` sets the same `TOKIO_WORKER_THREADS` for each binary.
+The report contains individual pass/fail/skip observations and a comparison
+matrix. An existing failure in the supplied binary remains a failure in its
+report; it is not silently converted to a successful baseline expectation.
+
+Coverage includes:
+
+- Static GET/HEAD and byte checksums at 1,040 B, 64 KiB and 1 MiB; empty files;
+  ETag/Last-Modified conditional requests; prefix/suffix/open/multipart ranges,
+  If-Range and 416; directory redirect/index/listing; decoded path rejection;
+  deny/inside-root symlinks; precompressed gzip; MIME/download headers;
+  `try_files`, redirects and the compiled `.htaccess` header subset.
+- Proxy POST/PUT, 1 MiB upload checksums, chunked upload delivery **before the
+  last chunk is sent**, early response delivery, client disconnect, original
+  status/redirect preservation, failed-POST no replay, request/response header
+  scopes, hop-by-hop stripping, request-ID shape/uniqueness and WebSocket echo.
+- Global host/vhost/profile/location ACL composition, upstream suppression on
+  denial, trusted versus spoofed forwarding, untrusted country hints, method
+  and rate policies, hot changes and rejection of invalid configuration.
+- Private administration routes, login, session/CSRF and Origin enforcement,
+  Basic Auth, ACL-before-auth, private-origin rejection, retained exchange
+  logging fields and byte accounting.
+- Cache global/vhost/route OFF and ON, MISS/HIT, Vary isolation, cookie,
+  Authorization/private/no-store bypass, PURGE, concurrent fills, validator
+  revalidation, static cached ranges/conditionals; transforms OFF/ON with cache
+  OFF/ON; compression negotiation, exclusions and scope/global codec ceilings.
+
+The generic suite does not establish TLS/H2/H3, Tinyauth, real PHP-FPM, real DNS
+rebinding, concurrent filesystem rename-race safety or long-soak acceptance.
+Use the dedicated transport tests and existing Rust integration suites for
+their covered cases. The sequential symlink-target test is explicitly not a
+concurrent rename-race proof.
+
+## TLS, HTTP/2, HTTP/3 and WebSocket interoperability
+
+```bash
+python3 scripts/performance/transport_regression.py \
+  --binary before=/absolute/path/rust-proxy-before \
+  --binary after=/absolute/path/rust-proxy-after \
+  --out /absolute/path/results/transport
+```
+
+This suite requires `aioquic`, `httpx[http2]` and `websockets`; use
+`--python-deps /absolute/path/pydeps` for a separate dependency directory.
+It verifies the temporary server certificate, negotiated protocol, response
+status and bytes for TLS H2 and QUIC H3, including HEAD, validators, ranges,
+cache ON/OFF, 1 MiB bodies and concurrent streams. Dedicated WebSocket checks
+include bidirectional echo and a normal close handshake. Client protocol
+self-checks are reported separately from RustProxy acceptance cases.
+This establishes covered interoperability, not TLS/H2/H3 throughput or
+production-certificate compatibility.
+
+## Two-origin load-balancer regression
+
+```bash
+python3 scripts/performance/lb_regression.py \
+  --binary before=/absolute/path/rust-proxy-before \
+  --binary after=/absolute/path/rust-proxy-after \
+  --out /absolute/path/results/lb
+```
+
+This separate, small suite sends real requests through RustProxy to two local
+TCP origins. It verifies equal round-robin allocation, GET failover after a 503
+and after a stopped origin refuses connections, and the absence of POST replay.
+A controlled response pauses after its first body fragment while configuration
+changes drain or disable its member. New requests must use the other member,
+the admitted response must finish intact, and its active lease must remain
+counted until completion and then return to zero. The main HTTP case matrix is
+unchanged. This suite does not cover sticky sessions, active health probes,
+DNS rotation or TLS upstream failures.
+
+## Finite concurrency correctness stress
+
+```bash
+python3 scripts/performance/concurrency_regression.py \
+  --binary before=/absolute/path/rust-proxy-before \
+  --binary after=/absolute/path/rust-proxy-after \
+  --nginx /absolute/path/nginx --out /absolute/path/results/concurrency
+```
+
+This separate functional check uses 1, 16 and 128 concurrent client tasks for
+proxy and static responses at 1,040 B, 64 KiB and 1 MiB. Each finite batch sends
+`max(32, 2 * clients)` requests and verifies every status, length and body SHA-256.
+After the clients finish, backend active-exchange counters must settle to zero.
+RSS and file descriptors are recorded before the batch and after its drain.
+These are not peak resource measurements or throughput results. Client tasks
+start behind a barrier; configured concurrency does not establish that the same
+number of server exchanges were active simultaneously.
+
+## Before/after/nginx benchmark
+
+First verify starts, loopback routing and exact payload hashes without invoking
+the load generator:
+
+```bash
+python3 scripts/performance/benchmark.py \
+  --binary before=/absolute/path/rust-proxy-before \
+  --binary after=/absolute/path/rust-proxy-after \
+  --nginx /absolute/path/nginx --wrk /absolute/path/wrk \
+  --server-cpus 0-3 --origin-cpus 4-5 --client-cpus 6-8 \
+  --out /absolute/path/results/preflight --prepare-only
+```
+
+For a host with CPUs 0–8 available, a reproducible primary run is:
+
+```bash
+python3 scripts/performance/benchmark.py \
+  --binary before=/absolute/path/rust-proxy-before \
+  --binary after=/absolute/path/rust-proxy-after \
+  --nginx /absolute/path/nginx --wrk /absolute/path/wrk \
+  --out /absolute/path/results/primary \
+  --workers 4 --origin-workers 2 --client-threads 4 \
+  --server-cpus 0-3 --origin-cpus 4-5 --client-cpus 6-8 \
+  --concurrency 64 --sizes small,large,huge --modes proxy,static \
+  --duration 15 --warmup 3 --repeats 5
+```
+
+Choose explicit, nonempty and disjoint CPU sets allowed in the run environment;
+adapt all three sets on a smaller host. Frontends use the same CPU set and worker
+count, the shared origin uses another set, and the generator uses a third. The
+harness checks each process/thread's actual affinity and refuses unavailable
+resource counters. Four wrk threads share the three client CPUs in this example.
+The structured result records thread count and effective connection count; at
+64 connections, four threads keep exactly 64 connections, avoiding wrk's integer
+division of connections between threads. Only one frontend is loaded at a time.
+Stop builds, other tests and profilers during throughput measurement.
+
+The primary run takes approximately 27 minutes: six scenarios, three variants,
+five samples, and 18 seconds per sample including excluded warm-up. To explore
+concurrency, repeat in a fresh output directory with
+`--concurrency 1,16,64,128`. More scenarios intentionally take more time.
+Short sanity checks require the explicit `--exploratory` option and cannot
+satisfy the >=5 samples / >=15-second measurement gate.
+
+Cache is explicitly disabled for the primary benchmark. Transforms retain their
+disabled defaults. Identity encoding is requested. Logging history, analytics,
+request IDs, origin policy, body accounting, health/lease plumbing and normal
+application functionality remain active. nginx uses `sendfile on`, HTTP/1.1,
+keep-alive and streaming proxy delivery. The generated exact configurations are
+saved, so this intentional difference in nginx/application features is visible.
+No cache-disabled claim is inferred from a cache-on speedup.
+
+Optional keep-alive tuning uses `--max-idle label=64`. Keep this as a separately
+labelled experiment. The harness does not turn off bounded pool recycling.
+
+## Recorded evidence and interpretation
+
+- `schedule.json`: complete deterministic crossover order, rotating and
+  reversing variants between repetitions. No best-run selection.
+- `preflight.json`, `fixture/payloads.json`, binary/version/config metadata:
+  exact status, body SHA-256, payload sizes and inputs.
+- Per-sample exact status/body checks before and after the measured interval,
+  plus a wire-bytes-per-request sanity check, detect wrong content and gross
+  transfer-size anomalies. These do not hash every response inside wrk.
+- `raw/*.txt` and `raw/*.json`: unmodified wrk output and structured requests,
+  duration, bytes, RPS, p50/p95/p99, effective concurrency, and
+  connect/read/write/timeout/status errors.
+- `raw/*-resources.json`: timestamped `/proc` samples for each server process
+  and its children, plus the separate origin process group.
+- `raw/*-affinity.json`: observed process/thread CPU masks for the front end,
+  origin and generator.
+- `runs.csv/json`, `summary.csv/json`: all samples, medians, RPS range and
+  coefficient of variation, latency medians, server CPU time per completed
+  request, peak/end RSS and file descriptors.
+- `status/*.json`: numeric drop and pool-generation/rotation counters read
+  outside each measured interval. Warm-up output is kept separately.
+- `completion.json` and `SHA256SUMS`: completion state and hashes of evidence.
+  Any generator/network/status error stops the run with `failure.json`; partial
+  data remains available and is not presented as completed acceptance.
+
+Use `(after_median / before_median - 1) * 100` for throughput improvement and
+`after_median / nginx_median * 100` for relative nginx throughput. Rebuilding
+both revisions with one compiler, target and resolved lockfile is required for
+source-change attribution. Compare the supplied executable separately: build
+flags and dependency versions can change its performance.
+
+These are same-host, closed-loop HTTP/1.1 diagnostics. They do not establish
+production capacity, TLS/H2/H3 performance, an open-loop latency SLA, allocator
+rates, CPU cycles or 24-hour resource stability. wrk latency follows wrk's own
+histogram/correction behavior. RSS and descriptor peaks are sampled observations,
+so shorter spikes may be missed. Summed nginx RSS can double-count shared pages.
+Origin or generator CPU saturation can limit observed throughput; inspect their
+resource data and move the generator/origin to separate machines for release
+measurements. Run profilers separately so profiling overhead does not enter the
+headline A/B throughput numbers.
+
+## Mixed-traffic resource soak
+
+Run this in its own phase, after correctness and throughput checks:
+
+```bash
+python3 scripts/performance/soak.py \
+  --binary /absolute/path/rust-proxy-after --label after \
+  --nginx /absolute/path/nginx --out /absolute/path/results/soak \
+  --duration 1800 --rate 100 --clients 8 \
+  --workers 4 --origin-workers 2 --server-cpus 0-3 --origin-cpus 4-5 \
+  --toggle-seconds 120 --sample-seconds 5 --abort-every 101
+```
+
+The bounded client executor alternates proxy/static and 1,040 B/64 KiB/1 MiB
+payloads, verifies full bytes and occasional HEAD/range semantics, closes some
+clients after an initial body prefix, and applies live cache OFF/ON changes.
+Every workload request opens a new frontend HTTP/1.1 connection; upstream
+keepalive retains its configured defaults. This exercises connection churn.
+The primary wrk benchmark separately uses persistent frontend connections.
+The nominal request rate is paced; scheduler delays and actual completion rate
+are reported instead of hiding an under-driven client. Pending jobs are bounded
+to twice the client count. The script restores base configuration, drains and
+rechecks both listeners before shutdown. After each cache apply, two separate
+GETs per source verify exact contents and require a HIT on the second GET when
+enabled; globally disabled cache must expose no X-Cache header. These probe
+requests are recorded separately from paced traffic. Backend active-exchange
+counters must settle to zero after the final drain and cooldown.
+
+Five-second resource/status records, body/status failures, observed cache-status
+counts, cache transitions/probes, RSS/FD trend and completed traffic totals are
+retained. Controlled client closes mean that the client closes after reading a
+prefix. The server may already have sent the complete response, so those counts
+are not presented as proven server-side cancellations.
+Memory trend is descriptive: switching caching and allocator retention affect
+RSS. This is a 30-minute resource gate, not a 24-hour leak proof. A short
+`--exploratory --duration ...` sanity run stays explicitly outside that gate.

scripts/performance/benchmark.py

@@ -0,0 +1,479 @@
+#!/usr/bin/env python3
+"""Interleaved before/after/nginx process benchmarks with immutable raw evidence.
+
+This is a same-host diagnostic experiment, not a simulation or release sign-off.
+Runs >=5 measured samples per variant/scenario by default, each >=15 seconds.
+"""
+from __future__ import annotations
+
+import argparse
+import csv
+import hashlib
+import itertools
+import json
+import os
+from pathlib import Path
+import platform
+import resource
+import statistics
+import subprocess
+import sys
+import time
+import traceback
+
+from common import Fixture, http, seal_results, sha256, write_json
+from resources import (Monitor, affinity_snapshot, cgroup_cpu_delta, cgroup_cpu_stat,
+                       parse_cpu_set, validate_cpu_sets, wait_for_affinity)
+from telemetry import traffic_analytics_delta, traffic_analytics_snapshot
+
+
+SIZES = {"small": "small.html", "large": "large.bin", "huge": "huge.bin"}
+DROP_COUNTER_FIELDS = {
+    "logs_dropped": "history_dropped_delta",
+    "logging.sinks.queue_dropped": "logging_sinks_queue_dropped_delta",
+    "logging.error_file.queue_dropped": "logging_error_file_queue_dropped_delta",
+    "cache.cold_queue_dropped": "cache_cold_queue_dropped_delta",
+}
+DROP_INTERVAL_NOTE = ("Delta brackets measured wrk plus one postflight request and possible in-flight completions; "
+                      "percentage uses wrk completed requests as an approximate denominator. "
+                      "History rejection does not imply rejection by live broadcast or external sinks.")
+
+
+def usage_snapshot():
+    usage = resource.getrusage(resource.RUSAGE_CHILDREN)
+    return usage.ru_utime + usage.ru_stime
+
+
+def effective_threads(requested, concurrency):
+    if requested < 1 or concurrency < 1:
+        raise ValueError("client threads and concurrency must be positive")
+    # wrk floor-divides the connection count between threads. Select a divisor
+    # so its advertised and effective connection counts remain identical.
+    return next(value for value in range(min(requested, concurrency), 0, -1)
+                if concurrency % value == 0)
+
+
+def wrk_run(args, port, path, duration, concurrency, prefix, roots=None, expected_processes=None):
+    threads = effective_threads(args.client_threads, concurrency)
+    command = [str(args.wrk), "--latency", "-t", str(threads),
+               "-c", str(concurrency), "-d", f"{duration}s", "--timeout", "5s",
+               "-H", "Accept-Encoding: identity", "-s", str(Path(__file__).with_name("wrk_report.lua")),
+               f"http://127.0.0.1:{port}/{path}"]
+    if args.client_cpus:
+        command = ["taskset", "-c", args.client_cpus, *command]
+    affinities = {}
+    expected_processes = expected_processes or {}
+    if roots:
+        for label, pid in roots.items():
+            expected = parse_cpu_set(args.server_cpus if label == "server" else args.origin_cpus)
+            affinities[label] = affinity_snapshot(pid, expected, expected_processes.get(label))
+    monitor = Monitor(roots, args.resource_interval, expected_processes) if roots else None
+    if monitor:
+        monitor.start()
+    cpu_start = usage_snapshot()
+    started = time.monotonic()
+    process = None
+    stdout, stderr = "", ""
+    try:
+        process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
+        affinities["generator"] = wait_for_affinity(process, parse_cpu_set(args.client_cpus))
+        stdout, stderr = process.communicate(timeout=duration + 30)
+    finally:
+        if process is not None and process.poll() is None:
+            process.kill()
+            stdout, stderr = process.communicate(timeout=5)
+        if monitor:
+            monitor.stop()
+        prefix.parent.mkdir(parents=True, exist_ok=True)
+        prefix.with_suffix(".txt").write_text(stdout + "\n" + stderr)
+        write_json(prefix.with_name(prefix.name + "-affinity.json"), affinities)
+        if monitor:
+            write_json(prefix.with_name(prefix.name + "-resources.json"), monitor.samples)
+            if monitor.errors:
+                write_json(prefix.with_name(prefix.name + "-resource-errors.json"), monitor.errors)
+    cpu_seconds = usage_snapshot() - cpu_start
+    wall_seconds = time.monotonic() - started
+    raw = stdout + "\n" + stderr
+    if process.returncode:
+        raise RuntimeError(f"wrk exited {process.returncode}: {raw[-2000:]}")
+    line = next((line for line in stdout.splitlines() if line.startswith("RUSTPROXY_BENCH_JSON ")), None)
+    if line is None:
+        raise RuntimeError("wrk produced no structured report")
+    result = json.loads(line.removeprefix("RUSTPROXY_BENCH_JSON "))
+    if result["requests"] <= 0 or result["duration_us"] <= 0:
+        write_json(prefix.with_suffix(".json"), result)
+        raise RuntimeError("wrk completed no requests or reported an invalid duration; raw evidence retained")
+    result.update({"command": command, "wall_seconds": wall_seconds,
+                   "client_threads": threads, "requested_connections": concurrency,
+                   "client_threads_requested": args.client_threads,
+                   "effective_connections": threads * (concurrency // threads),
+                   "generator_cpu_seconds": cpu_seconds,
+                   "rps": result["requests"] * 1_000_000 / result["duration_us"],
+                   "error_total": sum(value for key, value in result.items() if key.startswith("errors_"))})
+    if monitor:
+        result["resources"] = monitor.summary(result["requests"])
+    write_json(prefix.with_suffix(".json"), result)
+    return result
+
+
+def payload_check(port, path, expected):
+    """A full exact-body check outside the wrk measurement interval."""
+    try:
+        status, headers, body = http(port, "/" + path, headers={"Accept-Encoding": "identity"})
+        return {"status": status, "bytes": len(body), "sha256": hashlib.sha256(body).hexdigest(),
+                "expected_bytes": len(expected), "expected_sha256": hashlib.sha256(expected).hexdigest(),
+                "content_length": headers.get("content-length"), "content_encoding": headers.get("content-encoding"),
+                "approximate_header_bytes": 19 + sum(len(key) + len(value) + 4 for key, value in headers.items()),
+                "passed": status == 200 and body == expected}
+    except Exception as error:
+        return {"passed": False, "error": str(error), "expected_bytes": len(expected),
+                "expected_sha256": hashlib.sha256(expected).hexdigest()}
+
+
+def transfer_sanity(result, payload_bytes, header_bytes, concurrency):
+    """Check gross truncation/size anomalies; this is not per-response hashing."""
+    requests, wire_bytes = result["requests"], result["bytes"]
+    minimum = requests * payload_bytes
+    # wrk counts headers and partially received final responses in its wire bytes.
+    header_allowance = max(1024, 2 * header_bytes)
+    maximum = (requests + concurrency) * (payload_bytes + header_allowance)
+    return {"passed": requests > 0 and minimum <= wire_bytes <= maximum,
+            "wire_bytes_per_request": wire_bytes / requests if requests else None,
+            "expected_payload_bytes_per_request": payload_bytes,
+            "minimum_wire_bytes": minimum, "maximum_wire_bytes": maximum,
+            "header_allowance_per_response": header_allowance,
+            "inflight_response_allowance": concurrency}
+
+
+def counter_values(value, prefix=""):
+    """Retain numeric drop and rotation counters without session/config data."""
+    found = {}
+    if isinstance(value, dict):
+        for key, child in value.items():
+            path = f"{prefix}.{key}" if prefix else key
+            if isinstance(child, (int, float)) and any(part in key for part in ["dropped", "rotations", "generation", "requests_in_generation"]):
+                found[path] = child
+            found.update(counter_values(child, path))
+    elif isinstance(value, list):
+        for index, child in enumerate(value):
+            found.update(counter_values(child, f"{prefix}[{index}]"))
+    return found
+
+
+def validate_drop_counters(values):
+    for name in DROP_COUNTER_FIELDS:
+        if type(values.get(name)) is not int or values[name] < 0:
+            raise RuntimeError(f"missing or invalid cumulative status counter: {name}")
+    return values
+
+
+def status_counter_snapshot(rust):
+    if rust is None:
+        return None
+    status, _, body = rust.admin("/api/status")
+    if status != 200:
+        raise RuntimeError(f"{rust.label} status capture failed: HTTP {status}")
+    return counter_values(json.loads(body))
+
+
+def drop_counter_report(before, after, requests):
+    if requests <= 0:
+        raise ValueError("drop-rate denominator must contain completed requests")
+    if (before is None) != (after is None):
+        raise RuntimeError("both status snapshots are required for a drop delta")
+    if before is not None:
+        validate_drop_counters(before)
+        validate_drop_counters(after)
+    deltas = {name: after[name] - before[name] if before is not None else None
+              for name in DROP_COUNTER_FIELDS}
+    if any(value is not None and value < 0 for value in deltas.values()):
+        raise RuntimeError("cumulative drop counter decreased during measured interval")
+    fields = {field: deltas[name] for name, field in DROP_COUNTER_FIELDS.items()}
+    fields["history_drop_percent_of_completed_requests"] = (
+        100 * deltas["logs_dropped"] / requests if before is not None else None)
+    return {"applicable": before is not None, "counter_deltas": deltas, "row_fields": fields,
+            "completed_requests_denominator": requests, "denominator_is_approximate": True,
+            "interval_note": DROP_INTERVAL_NOTE}
+
+
+def analytics_counter_report(before, after, requests):
+    if requests <= 0 or (before is None) != (after is None):
+        raise ValueError("analytics delta requires paired snapshots and completed requests")
+    delta = traffic_analytics_delta(before, after) if before is not None else {
+        "queue_dropped_delta": None, "recorded_requests_delta": None, "interval_seconds": None}
+    fields = {"analytics_queue_dropped_delta": delta["queue_dropped_delta"],
+              "analytics_recorded_requests_delta": delta["recorded_requests_delta"],
+              "analytics_queue_drop_percent_of_completed_requests": (
+                  100 * delta["queue_dropped_delta"] / requests if before is not None else None)}
+    return {"applicable": before is not None, **delta, "row_fields": fields,
+            "completed_requests_denominator": requests, "denominator_is_approximate": True,
+            "interval_note": DROP_INTERVAL_NOTE,
+            "recorded_requests_note": "Asynchronous analytics records; not HTTP success or completion counts."}
+
+
+def persist_rows(out, rows):
+    write_json(out / "runs.json", rows)
+    if not rows:
+        return
+    with (out / "runs.csv").open("w", newline="") as stream:
+        writer = csv.DictWriter(stream, fieldnames=list(rows[0]))
+        writer.writeheader()
+        writer.writerows(rows)
+    groups = {}
+    for row in rows:
+        groups.setdefault((row["mode"], row["size"], row["concurrency"], row["variant"]), []).append(row)
+    summary = []
+    for (mode, size, concurrency, variant), group in sorted(groups.items()):
+        rps = [row["rps"] for row in group]
+        summary.append({"mode": mode, "size": size, "concurrency": concurrency, "variant": variant,
+                        "samples": len(group), "rps_median": statistics.median(rps),
+                        "rps_min": min(rps), "rps_max": max(rps),
+                        "rps_cv_percent": statistics.stdev(rps) / statistics.mean(rps) * 100 if len(rps) > 1 else None,
+                        "p50_us_median": statistics.median(row["p50_us"] for row in group),
+                        "p95_us_median": statistics.median(row["p95_us"] for row in group),
+                        "p99_us_median": statistics.median(row["p99_us"] for row in group),
+                        "server_cpu_us_per_request_median": statistics.median(row["server_cpu_us_per_request"] for row in group),
+                        "server_rss_peak_bytes": max(row["server_rss_peak_bytes"] for row in group),
+                        "server_fd_peak": max(row["server_fd_peak"] for row in group),
+                        "requests": sum(row["requests"] for row in group),
+                        "cgroup_throttled_samples": sum(row["cgroup_nr_throttled_delta"] > 0 for row in group),
+                        "cgroup_nr_throttled_delta": sum(row["cgroup_nr_throttled_delta"] for row in group),
+                        "cgroup_throttled_usec_delta": sum(row["cgroup_throttled_usec_delta"] for row in group),
+                        "validation_failures": sum(not (row["payload_before_passed"] and row["payload_after_passed"]
+                                                          and row["transfer_sanity_passed"]) for row in group),
+                        "errors": sum(row["error_total"] for row in group)})
+    write_json(out / "summary.json", summary)
+    with (out / "summary.csv").open("w", newline="") as stream:
+        writer = csv.DictWriter(stream, fieldnames=list(summary[0]))
+        writer.writeheader()
+        writer.writerows(summary)
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--binary", action="append", required=True, metavar="LABEL=/ABS/PATH")
+    parser.add_argument("--max-idle", action="append", default=[], metavar="LABEL=COUNT")
+    parser.add_argument("--nginx", type=Path, required=True)
+    parser.add_argument("--wrk", type=Path, required=True)
+    parser.add_argument("--out", type=Path, required=True)
+    parser.add_argument("--workers", type=int, default=4)
+    parser.add_argument("--origin-workers", type=int, default=4)
+    parser.add_argument("--client-threads", type=int, default=4)
+    parser.add_argument("--server-cpus")
+    parser.add_argument("--origin-cpus")
+    parser.add_argument("--client-cpus")
+    parser.add_argument("--duration", type=int, default=15)
+    parser.add_argument("--warmup", type=int, default=3)
+    parser.add_argument("--repeats", type=int, default=5)
+    parser.add_argument("--concurrency", default="64")
+    parser.add_argument("--sizes", default="small,large")
+    parser.add_argument("--modes", default="proxy,static")
+    parser.add_argument("--resource-interval", type=float, default=.5)
+    parser.add_argument("--exploratory", action="store_true", help="explicitly allow <5 repetitions or <15s samples")
+    parser.add_argument("--prepare-only", action="store_true", help="start processes and verify hashes; run no load generator")
+    args = parser.parse_args()
+    if (args.duration < 15 or args.repeats < 5) and not args.exploratory:
+        parser.error("acceptance requires >=15-second runs and >=5 repetitions; use --exploratory for labelled short diagnostics")
+    if args.out.exists() and any(args.out.iterdir()):
+        parser.error("--out must be a new or empty directory so raw evidence is never overwritten")
+    modes, sizes = args.modes.split(","), args.sizes.split(",")
+    concurrencies = [int(value) for value in args.concurrency.split(",")]
+    if any(mode not in ("proxy", "static") for mode in modes) or any(size not in SIZES for size in sizes):
+        parser.error("supported modes: proxy,static; sizes: small,large,huge")
+    if min(concurrencies) < 1 or max(concurrencies) > 4096:
+        parser.error("concurrency must be 1..4096")
+    if args.client_threads < 1:
+        parser.error("--client-threads must be positive")
+    if args.workers < 1 or args.origin_workers < 1:
+        parser.error("--workers and --origin-workers must be positive")
+    try:
+        cpu_sets = validate_cpu_sets({"server": args.server_cpus, "origin": args.origin_cpus,
+                                      "client": args.client_cpus}, set(os.sched_getaffinity(0)))
+    except ValueError as error:
+        parser.error(str(error))
+    if args.resource_interval <= 0:
+        parser.error("--resource-interval must be positive")
+    args.nginx, args.wrk, args.out = args.nginx.resolve(), args.wrk.resolve(), args.out.resolve()
+    variants = dict(item.split("=", 1) for item in args.binary)
+    if len(variants) != len(args.binary) or "nginx" in variants:
+        parser.error("variant names must be unique and must not be nginx")
+    max_idle = dict(item.split("=", 1) for item in args.max_idle)
+    args.out.mkdir(parents=True, exist_ok=True)
+    environment = {"kind": "same-host real HTTP/1.1 process benchmark", "simulation": False,
+        "exploratory": args.exploratory, "prepare_only": args.prepare_only,
+        "acceptance_sampling": args.duration >= 15 and args.repeats >= 5 and not args.prepare_only,
+        "platform": platform.platform(), "python": sys.version,
+        "started_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
+        "cpu_affinity_available": sorted(os.sched_getaffinity(0)),
+        "cpu_sets": {label: sorted(cpus) for label, cpus in cpu_sets.items()},
+        "resource_process_discovery": "PPid map from /proc/*/stat; root and configured nginx process counts must be readable",
+        "expected_nginx_processes": {"frontend": args.workers + 1, "origin": args.origin_workers + 1},
+        "generator_plan": [{"connections": concurrency, "requested_threads": args.client_threads,
+                            "effective_threads": effective_threads(args.client_threads, concurrency)}
+                           for concurrency in concurrencies],
+        "settings": {key: str(value) if isinstance(value, Path) else value for key, value in vars(args).items()},
+        "wrk_sha256": sha256(args.wrk), "nginx_sha256": sha256(args.nginx),
+        "limitations": ["Generator, upstream and frontend share one host and kernel; not production capacity",
+                        "Closed-loop wrk latency includes its histogram correction; no open-loop arrival-rate SLA claim",
+                        "RSS sums processes and may double-count shared pages in nginx",
+                        "RSS and FD peaks are sampled, not absolute instantaneous peaks",
+                        "Exact HTTP 200 and payload hashes are checked before/after each measured run, not for every response during load",
+                        "wrk status errors count only status >399; aggregate bytes/request is a gross sanity check, not response integrity proof",
+                        "cgroup CPU-throttling deltas cover all processes in the shared cgroup, not only the measured server",
+                        DROP_INTERVAL_NOTE,
+                        "No TLS/H2/H3 speed inference; no CPU cycles/allocation profiler claim",
+                        "nginx has a smaller feature set; request logging/ACL/analytics work is not feature-matched"]}
+    for path in ["/sys/fs/cgroup/cpu.max", "/sys/fs/cgroup/memory.max", "/proc/version"]:
+        try:
+            environment[path] = Path(path).read_text().strip()
+        except OSError:
+            environment[path] = None
+    write_json(args.out / "environment.json", environment)
+    expected_samples = len(modes) * len(sizes) * len(concurrencies) * args.repeats * (len(variants) + 1)
+    write_json(args.out / "completion.json", {"completed": False, "samples": 0,
+                                              "expected_samples": expected_samples,
+                                              "simulation": False, "prepare_only": args.prepare_only})
+    rows, error = [], None
+    try:
+        with Fixture(args.out / "fixture", workers=args.workers, server_cpus=args.server_cpus,
+                     origin_cpus=args.origin_cpus, origin_workers=args.origin_workers) as fixture:
+            nginx = fixture.nginx(args.nginx)
+            servers = {"nginx": {"process": nginx["process"], "proxy": nginx["reverse_port"],
+                                  "static": nginx["static_port"], "rust": None}}
+            for label, binary in variants.items():
+                rust = fixture.rust(Path(binary), label, max_idle=int(max_idle[label]) if label in max_idle else None)
+                rust.login()
+                servers[label] = {"process": rust.process, "proxy": rust.reverse_port, "static": rust.static_port, "rust": rust}
+            quiescence = {label: server["rust"].wait_quiescent()
+                          for label, server in servers.items() if server["rust"] is not None}
+            write_json(args.out / "quiescence-preflight.json", quiescence)
+            initial_affinity = {"origin": affinity_snapshot(nginx["origin"].pid, cpu_sets["origin"], args.origin_workers + 1)}
+            initial_affinity.update({label: affinity_snapshot(server["process"].pid, cpu_sets["server"],
+                                                               args.workers + 1 if label == "nginx" else 1)
+                                     for label, server in servers.items()})
+            write_json(args.out / "affinity-preflight.json", initial_affinity)
+            # The fixture has only master + configured workers. Refuse a
+            # master-only snapshot even if HTTP payload checks would succeed.
+            prepare_monitor = Monitor({"server": nginx["process"].pid, "origin": nginx["origin"].pid},
+                                      args.resource_interval,
+                                      {"server": args.workers + 1, "origin": args.origin_workers + 1})
+            prepare_monitor.sample()
+            write_json(args.out / "resources-preflight.json", prepare_monitor.samples)
+            telemetry_preflight = {label: {"status": status_counter_snapshot(server["rust"]),
+                "traffic": traffic_analytics_snapshot(server["rust"]) if server["rust"] else None}
+                for label, server in servers.items()}
+            write_json(args.out / "telemetry-preflight.json", telemetry_preflight)
+            for snapshot in telemetry_preflight.values():
+                if snapshot["status"] is not None:
+                    validate_drop_counters(snapshot["status"])
+            preflight = []
+            for label, server in servers.items():
+                for mode, size in itertools.product(modes, sizes):
+                    payload = (fixture.site / SIZES[size]).read_bytes()
+                    check = payload_check(server[mode], SIZES[size], payload)
+                    preflight.append({"variant": label, "mode": mode, "size": size, **check})
+                    write_json(args.out / "preflight.json", preflight)
+                    if not check["passed"]:
+                        raise AssertionError(f"payload preflight failed: {label} {mode} {size}: {check}")
+            write_json(args.out / "preflight.json", preflight)
+            labels = [*variants, "nginx"]
+            schedule = []
+            for mode, size, concurrency in itertools.product(modes, sizes, concurrencies):
+                for repetition in range(args.repeats):
+                    offset = repetition % len(labels)
+                    order = labels[offset:] + labels[:offset]
+                    if repetition % 2:
+                        order.reverse()
+                    for variant in order:
+                        schedule.append({"sequence": len(schedule) + 1, "mode": mode, "size": size,
+                                         "concurrency": concurrency, "repetition": repetition + 1, "variant": variant})
+            write_json(args.out / "schedule.json", schedule)
+            if not args.prepare_only:
+                for item in schedule:
+                    server = servers[item["variant"]]
+                    expected_processes = {"origin": args.origin_workers + 1,
+                                          "server": args.workers + 1 if item["variant"] == "nginx" else 1}
+                    stem = f"{item['sequence']:04}-{item['mode']}-{item['size']}-c{item['concurrency']}-{item['variant']}"
+                    if args.warmup:
+                        warmup = wrk_run(args, server[item["mode"]], SIZES[item["size"]], args.warmup,
+                                         item["concurrency"], args.out / "warmup" / stem)
+                        if warmup["error_total"]:
+                            raise RuntimeError(f"warmup errors in {stem}; raw evidence retained")
+                    payload = (fixture.site / SIZES[item["size"]]).read_bytes()
+                    before = payload_check(server[item["mode"]], SIZES[item["size"]], payload)
+                    write_json(args.out / "checks" / (stem + "-before.json"), before)
+                    if not before["passed"]:
+                        raise RuntimeError(f"exact payload check failed before {stem}: {before}")
+                    counters_before = status_counter_snapshot(server["rust"])
+                    write_json(args.out / "status" / (stem + "-before.json"), counters_before)
+                    if counters_before is not None:
+                        validate_drop_counters(counters_before)
+                    analytics_before = traffic_analytics_snapshot(server["rust"]) if server["rust"] else None
+                    write_json(args.out / "status" / (stem + "-traffic-before.json"), analytics_before)
+                    cpu_before = cgroup_cpu_stat()
+                    try:
+                        result = wrk_run(args, server[item["mode"]], SIZES[item["size"]], args.duration,
+                                         item["concurrency"], args.out / "raw" / stem,
+                                         {"server": server["process"].pid, "origin": nginx["origin"].pid},
+                                         expected_processes)
+                    finally:
+                        cpu_after = cgroup_cpu_stat()
+                        cpu_delta = cgroup_cpu_delta(cpu_before, cpu_after)
+                        throttling = {"before": cpu_before, "after": cpu_after, "delta": cpu_delta}
+                        write_json(args.out / "checks" / (stem + "-cgroup-cpu.json"), throttling)
+                    after = payload_check(server[item["mode"]], SIZES[item["size"]], payload)
+                    write_json(args.out / "checks" / (stem + "-after.json"), after)
+                    write_json(args.out / "checks" / (stem + "-affinity-after.json"), {
+                        "server": affinity_snapshot(server["process"].pid, cpu_sets["server"], expected_processes.get("server")),
+                        "origin": affinity_snapshot(nginx["origin"].pid, cpu_sets["origin"], expected_processes["origin"])})
+                    counters_after = status_counter_snapshot(server["rust"])
+                    write_json(args.out / "status" / (stem + ".json"), counters_after)
+                    analytics_after = traffic_analytics_snapshot(server["rust"]) if server["rust"] else None
+                    write_json(args.out / "status" / (stem + "-traffic-after.json"), analytics_after)
+                    drops = drop_counter_report(counters_before, counters_after, result["requests"])
+                    write_json(args.out / "status" / (stem + "-delta.json"), drops)
+                    analytics = analytics_counter_report(analytics_before, analytics_after, result["requests"])
+                    write_json(args.out / "status" / (stem + "-traffic-delta.json"), analytics)
+                    sanity = transfer_sanity(result, len(payload), before["approximate_header_bytes"], item["concurrency"])
+                    result.update({"payload_before": before, "payload_after": after, "transfer_sanity": sanity,
+                                   "cgroup_cpu": throttling, "observability": drops, "traffic_analytics": analytics})
+                    write_json((args.out / "raw" / stem).with_suffix(".json"), result)
+                    row = {**item, **{key: result[key] for key in ["requests", "bytes", "duration_us", "rps", "p50_us", "p95_us", "p99_us",
+                        "client_threads_requested", "client_threads", "requested_connections", "effective_connections",
+                        "error_total", "errors_connect", "errors_read", "errors_write", "errors_timeout", "errors_status", "generator_cpu_seconds"]}}
+                    row.update({"payload_before_passed": before["passed"], "payload_after_passed": after["passed"],
+                                "transfer_sanity_passed": sanity["passed"], "wire_bytes_per_request": sanity["wire_bytes_per_request"],
+                                "cgroup_nr_throttled_delta": cpu_delta["nr_throttled"],
+                                "cgroup_throttled_usec_delta": cpu_delta["throttled_usec"]})
+                    row.update(drops["row_fields"])
+                    row.update(analytics["row_fields"])
+                    for group in ["server", "origin"]:
+                        row.update({f"{group}_{key}": value for key, value in result["resources"][group].items()})
+                    rows.append(row)
+                    persist_rows(args.out, rows)
+                    print(json.dumps({"sequence": item["sequence"], "total": len(schedule), "variant": item["variant"],
+                                      "scenario": f"{item['mode']}_{item['size']}_c{item['concurrency']}",
+                                      "rps": round(result["rps"], 2), "p95_ms": result["p95_us"] / 1000,
+                                      "errors": result["error_total"]}), flush=True)
+                    if result["error_total"]:
+                        raise RuntimeError(f"benchmark errors in {stem}; raw evidence retained")
+                    if not after["passed"] or not sanity["passed"]:
+                        raise RuntimeError(f"payload/byte-count validation failed in {stem}; raw evidence retained")
+    except (Exception, KeyboardInterrupt) as exc:
+        error = {"error": str(exc) or type(exc).__name__, "type": type(exc).__name__,
+                 "interrupted": isinstance(exc, KeyboardInterrupt), "traceback": traceback.format_exc()}
+        write_json(args.out / "failure.json", error)
+        print(json.dumps(error), flush=True)
+    finally:
+        persist_rows(args.out, rows)
+        completed = error is None and (args.prepare_only or len(rows) == expected_samples)
+        write_json(args.out / "completion.json", {"completed": completed, "samples": len(rows),
+                                                  "expected_samples": expected_samples,
+                                                  "simulation": False, "prepare_only": args.prepare_only,
+                                                  "finished_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())})
+        seal_results(args.out)
+    return (130 if error.get("interrupted") else 1) if error else (0 if completed else 1)
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/performance/checks_cache.py

@@ -0,0 +1,180 @@
+"""Cache and response-processing paths must survive disabled-path optimization."""
+from concurrent.futures import ThreadPoolExecutor
+import copy
+import gzip
+import io
+import json
+import time
+import zlib
+
+from common import http
+
+
+def run(check, rust, peer):
+    port = rust.reverse_port
+    before = peer.count("GET", "/fresh")
+    results = [http(port, "/fresh") for _ in range(2)]
+    check("cache.global_off", all(s == 200 and h.get("x-cache") != "HIT" for s, h, b in results)
+          and peer.count("GET", "/fresh") - before == 2)
+    config = copy.deepcopy(rust.base)
+    config["cache"] = {"enabled": True, "expose_status": True, "expose_hits": True,
+                       "expose_cache_status": True, "http_invalidation_allowed_cidrs": ["127.0.0.1/32"]}
+    config["web_services"][0]["cache_control"] = "public, max-age=60"
+    rust.apply(config, "cache-enabled")
+    first = http(port, "/fresh")
+    second = http(port, "/fresh")
+    check("cache.miss_then_hit", first[0] == 200 and first[1].get("x-cache") == "MISS"
+          and second[1].get("x-cache") == "HIT" and first[2] == second[2],
+          first=first[1].get("x-cache"), second=second[1].get("x-cache"))
+    results = {}
+    for language in ["en", "pl", "en", "pl"]:
+        response = http(port, "/vary", headers={"Accept-Language": language})
+        previous = results.get(language)
+        if previous:
+            check("cache.vary_hit." + language, response[1].get("x-cache") == "HIT" and response[2] == previous[2],
+                  cache=response[1].get("x-cache"))
+        results[language] = response
+    check("cache.vary_isolation", results["en"][2] != results["pl"][2])
+    for path in ["/private", "/no-store", "/cookie"]:
+        before = peer.count("GET", path)
+        first, second = http(port, path), http(port, path)
+        check("cache.nonpublic_bypass." + path, first[0] == second[0] == 200
+              and first[1].get("x-cache") != "HIT" and second[1].get("x-cache") != "HIT"
+              and peer.count("GET", path) - before == 2,
+              first=first[1].get("x-cache"), second=second[1].get("x-cache"))
+    for name, headers in [("authorization", {"Authorization": "Bearer fixture-value"}),
+                           ("cookie", {"Cookie": "application_session=fixture"})]:
+        before = peer.count("GET", "/fresh")
+        s, h, b = http(port, "/fresh", headers=headers)
+        check("cache.request_bypass." + name, s == 200 and h.get("x-cache") != "HIT"
+              and peer.count("GET", "/fresh") - before == 1, cache=h.get("x-cache"))
+    before = peer.count("GET", "/fresh")
+    s, h, b = http(port, "/fresh", "GET", {"Content-Type": "application/octet-stream"}, b"nonempty-get-body")
+    check("cache.nonempty_get_bypass_and_preserve", s == 200 and h.get("x-cache") != "HIT"
+          and h.get("x-received-body-bytes") == "17" and peer.count("GET", "/fresh") - before == 1,
+          cache=h.get("x-cache"), origin_body_bytes=h.get("x-received-body-bytes"))
+    s, h, b = http(port, "/fresh", "PURGE")
+    after = http(port, "/fresh")
+    check("cache.http_purge", s in (200, 202, 204) and after[1].get("x-cache") == "MISS", status=s)
+    with ThreadPoolExecutor(max_workers=8) as pool:
+        results = list(pool.map(lambda _: http(port, "/concurrent"), range(8)))
+    check("cache.concurrent_fill", all(item[0] == 200 for item in results)
+          and len({item[2] for item in results}) == 1, origin_requests=peer.count("GET", "/concurrent"))
+    first = http(port, "/revalidate")
+    time.sleep(1.2)
+    second = http(port, "/revalidate")
+    check("cache.expired_validator_revalidation", first[0] == second[0] == 200 and first[2] == second[2]
+          and peer.count("GET", "/revalidate") >= 2, cache=second[1].get("x-cache"))
+
+    first, second = http(rust.static_port), http(rust.static_port)
+    check("cache.static_hit", first[0] == second[0] == 200 and second[1].get("x-cache") == "HIT"
+          and first[2] == second[2], cache=second[1].get("x-cache"))
+    empty_first, empty_second = http(rust.static_port, "/empty.txt"), http(rust.static_port, "/empty.txt")
+    check("cache.static_empty_hit", empty_first[0] == empty_second[0] == 200
+          and empty_second[1].get("x-cache") == "HIT" and empty_second[2] == b"",
+          first=empty_first[1].get("x-cache"), second=empty_second[1].get("x-cache"))
+    hs, hh, hb = http(rust.static_port, "/download.txt", "HEAD")
+    gs, gh, gb = http(rust.static_port, "/download.txt")
+    check("cache.static_head_cannot_fill_empty_get", hs == gs == 200 and hb == b""
+          and gb == (rust.fixture.site / "download.txt").read_bytes(), head_status=hs, get_status=gs)
+    s, h, b = http(rust.static_port, headers={"Range": "bytes=0-9"})
+    check("cache.static_range_survives", s == 206 and b == first[2][:10], status=s)
+    s, h, b = http(rust.static_port, headers={"If-None-Match": first[1]["etag"]})
+    check("cache.static_conditional_survives", s == 304 and b == b"", status=s)
+
+    config["vhosts"][0]["cache"] = {"enabled": False}
+    rust.apply(config, "cache-vhost-disabled")
+    first, second = http(port, "/fresh"), http(port, "/fresh")
+    check("cache.vhost_off", all(response[1].get("x-cache") != "HIT" for response in [first, second])
+          and first[2] != second[2])
+    config["vhosts"][0]["cache"] = {"enabled": True}
+    config["vhosts"][0]["routes"] = [{"id": "fresh-no-cache", "path_prefix": "/fresh", "cache": False}]
+    rust.apply(config, "cache-route-disabled")
+    first, second = http(port, "/fresh"), http(port, "/fresh")
+    check("cache.route_off", all(response[1].get("x-cache") != "HIT" for response in [first, second])
+          and first[2] != second[2])
+    config["cache"]["enabled"] = False
+    config["vhosts"][0]["routes"] = []
+    rust.apply(config, "cache-global-disabled-vhost-enabled")
+    first, second = http(port, "/fresh"), http(port, "/fresh")
+    check("cache.vhost_cannot_override_global_off", first[2] != second[2]
+          and second[1].get("x-cache") != "HIT")
+
+
+def transforms(check, rust, peer):
+    config = copy.deepcopy(rust.base)
+    original = (rust.fixture.site / "app.json").read_bytes()
+    for port, source in [(rust.reverse_port, "proxy"), (rust.static_port, "static")]:
+        s, h, b = http(port, "/app.json")
+        check("transform.off." + source, s == 200 and b == original, status=s)
+    config["transform"] = {"enabled": True, "json": True, "html": True}
+    for vhost in config["vhosts"]:
+        vhost["transform"] = {"enabled": True, "json": True, "html": True}
+    rust.apply(config, "transform-enabled-cache-off")
+    for port, source in [(rust.reverse_port, "proxy"), (rust.static_port, "static")]:
+        s, h, b = http(port, "/app.json")
+        check("transform.on." + source, s == 200 and json.loads(b) == json.loads(original)
+              and len(b) < len(original), original_bytes=len(original), result_bytes=len(b), status=s)
+        s, h, b = http(port, "/large.bin")
+        check("transform.binary_unchanged." + source, s == 200 and b == (rust.fixture.site / "large.bin").read_bytes())
+        s, h, b = http(port, "/app.json", headers={"Range": "bytes=0-9"})
+        check("transform.range_exclusion." + source, s == 206 and b == original[:10], status=s)
+    config["cache"] = {"enabled": True, "expose_status": True}
+    rust.apply(config, "transform-enabled-cache-on")
+    first, second = http(rust.reverse_port, "/app.json"), http(rust.reverse_port, "/app.json")
+    check("transform.cache_combination", second[0] == 200 and second[1].get("x-cache") == "HIT"
+          and first[2] == second[2] and json.loads(second[2]) == json.loads(original),
+          cache=second[1].get("x-cache"))
+
+    rust.reset()
+    expected = (rust.fixture.site / "compress.txt").read_bytes()
+    for port, source in [(rust.reverse_port, "proxy"), (rust.static_port, "static")]:
+        s, h, b = http(port, "/compress.txt", headers={"Accept-Encoding": "gzip"})
+        check("compression.gzip." + source, s == 200 and h.get("content-encoding") == "gzip"
+              and gzip.decompress(b) == expected and len(b) < len(expected), bytes=len(b))
+        s, h, b = http(port, "/compress.txt", headers={"Accept-Encoding": "gzip;q=0, identity"})
+        check("compression.identity_qvalue." + source, s == 200 and "content-encoding" not in h and b == expected)
+        s, h, b = http(port, "/compress.txt", headers={"Accept-Encoding": "gzip", "Range": "bytes=0-9"})
+        check("compression.range_exclusion." + source, s == 206 and "content-encoding" not in h and b == expected[:10])
+    s, h, b = http(rust.reverse_port, "/no-transform", headers={"Accept-Encoding": "gzip"})
+    check("compression.no_transform_exclusion", s == 200 and "content-encoding" not in h)
+    config = copy.deepcopy(rust.base)
+    config["vhosts"][0]["compression"] = {"enabled": False}
+    rust.apply(config, "compression-vhost-disabled")
+    s, h, b = http(rust.reverse_port, "/compress.txt", headers={"Accept-Encoding": "gzip"})
+    check("compression.vhost_off", s == 200 and "content-encoding" not in h and b == expected)
+    config["compression"] = {"gzip": False, "br": False, "zstd": False, "deflate": False}
+    config["vhosts"][0]["compression"] = {"enabled": True, "gzip": True}
+    rust.apply(config, "compression-global-disabled")
+    s, h, b = http(rust.reverse_port, "/compress.txt", headers={"Accept-Encoding": "gzip"})
+    check("compression.global_codec_ceiling", s == 200 and "content-encoding" not in h and b == expected)
+
+    config = copy.deepcopy(rust.base)
+    config["compression"] = {"gzip": True, "br": True, "zstd": True, "deflate": True}
+    config["cache"] = {"enabled": True, "expose_status": True}
+    rust.apply(config, "compression-cache-enabled")
+    for codec in ["gzip", "deflate", "br", "zstd"]:
+        decoder = None
+        if codec == "gzip":
+            decoder = gzip.decompress
+        elif codec == "deflate":
+            decoder = zlib.decompress
+        elif codec == "br":
+            try:
+                import brotli
+                decoder = brotli.decompress
+            except ImportError:
+                pass
+        else:
+            try:
+                import zstandard
+                decoder = lambda data: zstandard.ZstdDecompressor().stream_reader(io.BytesIO(data)).read()
+            except ImportError:
+                pass
+        if decoder is None:
+            check("compression.codec." + codec, None, skip_reason="optional Python decoder is not installed")
+            continue
+        first = http(rust.reverse_port, "/compress.txt", headers={"Accept-Encoding": codec})
+        second = http(rust.reverse_port, "/compress.txt", headers={"Accept-Encoding": codec})
+        check("compression.codec." + codec, second[0] == 200 and second[1].get("content-encoding") == codec
+              and decoder(first[2]) == decoder(second[2]) == expected, cache=second[1].get("x-cache"))

scripts/performance/checks_policy.py

@@ -0,0 +1,184 @@
+"""Live ACL, authentication, atomic reload and logging regressions."""
+import base64
+import copy
+import json
+import re
+import time
+
+from common import http, write_json
+
+
+DENY = {"default": "deny", "rules": []}
+ALLOW_LOCAL = {"default": "deny", "rules": [{"network": "127.0.0.1/32", "action": "allow"}]}
+
+
+def run(check, rust, peer):
+    port = rust.reverse_port
+    config = copy.deepcopy(rust.base)
+    config["host_policies"] = [{"hosts": ["fixture.test"], "acl": DENY}]
+    rust.apply(config, "policy-global-host")
+    count = peer.count("GET", "/echo")
+    s, h, b = http(port, "/echo", headers={"Host": "fixture.test"})
+    check("acl.global_host_without_vhost_acl", s == 403 and peer.count("GET", "/echo") == count,
+          status=s, upstream_requests=peer.count("GET", "/echo") - count)
+    s, h, b = http(port, "/echo", headers={"Host": "localhost"})
+    check("acl.global_other_host_unchanged", s == 200, status=s)
+
+    config = copy.deepcopy(rust.base)
+    config["vhosts"][0]["acl"] = DENY
+    rust.apply(config, "policy-vhost-deny")
+    s, h, b = http(port)
+    check("acl.vhost_deny", s == 403, status=s)
+    s, h, b = http(rust.static_port)
+    check("acl.vhost_isolation", s == 200, status=s)
+    config["vhosts"][0]["acl"] = ALLOW_LOCAL
+    rust.apply(config, "policy-vhost-allow")
+    s, h, b = http(port)
+    check("acl.hot_enable_allow", s == 200, status=s)
+
+    config = copy.deepcopy(rust.base)
+    config["acl_profiles"] = [{"id": "restricted", "name": "Restricted", "acl": DENY}]
+    config["vhosts"][0]["acl_profile_id"] = "restricted"
+    config["vhosts"][0]["acl"] = ALLOW_LOCAL
+    rust.apply(config, "policy-profile-deny")
+    s, h, b = http(port)
+    check("acl.profile_and_vhost_stack", s == 403, status=s)
+    config = copy.deepcopy(rust.base)
+    config["vhosts"][0]["routes"] = [{"id": "denied", "path_prefix": "/echo", "acl": DENY}]
+    rust.apply(config, "policy-location-deny")
+    count = peer.count("GET", "/echo")
+    s, h, b = http(port, "/echo")
+    check("acl.location_before_upstream", s == 403 and peer.count("GET", "/echo") == count, status=s)
+    s, h, b = http(port)
+    check("acl.location_other_path_unchanged", s == 200, status=s)
+    config["vhosts"][0]["acl"] = DENY
+    config["vhosts"][0]["routes"][0]["acl"] = ALLOW_LOCAL
+    rust.apply(config, "policy-location-allow-vhost-deny")
+    s, h, b = http(port, "/echo")
+    check("acl.location_cannot_override_vhost_deny", s == 403, status=s)
+
+    config = copy.deepcopy(rust.base)
+    config["vhosts"][0]["acl"] = {"default": "deny", "rules": [{"network": "203.0.113.22/32", "action": "allow"}]}
+    rust.apply(config, "policy-untrusted-forwarded")
+    s, h, b = http(port, headers={"X-Forwarded-For": "203.0.113.22", "X-Real-IP": "203.0.113.22"})
+    check("acl.spoofed_forwarded_ip_denied", s == 403, status=s)
+    config["forwarding"] = {"trusted_proxies": ["127.0.0.1/32"]}
+    rust.apply(config, "policy-trusted-forwarded")
+    s, h, b = http(port, headers={"X-Forwarded-For": "203.0.113.22"})
+    check("acl.trusted_forwarded_ip_allowed", s == 200, status=s)
+
+    config = copy.deepcopy(rust.base)
+    config["vhosts"][0]["rate_limit"] = {"country_access": {"mode": "allow", "countries": ["PL"]}}
+    config["vhosts"][0]["cloudflare_client_ip"] = True
+    rust.apply(config, "policy-country")
+    s, h, b = http(port, headers={"CF-IPCountry": "PL", "CF-Connecting-IP": "203.0.113.22"})
+    check("acl.untrusted_country_header_cannot_allow", s == 403, status=s)
+
+    config = copy.deepcopy(rust.base)
+    config["vhosts"][0]["method_policy"] = {"mode": "allow", "methods": ["GET", "HEAD"]}
+    rust.apply(config, "policy-method")
+    s, h, b = http(port, "/echo", "POST", body=b"denied method")
+    check("policy.method_allowlist", s in (403, 405), status=s)
+    s, h, b = http(port)
+    check("policy.allowed_method_survives", s == 200, status=s)
+    config = copy.deepcopy(rust.base)
+    config["vhosts"][0]["rate_limit"] = {"enabled": True, "requests": 2, "window_seconds": 60, "burst": 0}
+    rust.apply(config, "policy-rate-limit")
+    statuses = [http(port)[0] for _ in range(3)]
+    check("policy.rate_limit", statuses[:2] == [200, 200] and statuses[2] == 429, statuses=statuses)
+    rust.reset()
+    s, h, b = http(port)
+    check("policy.hot_disable_restores_request", s == 200, status=s)
+
+    # Every route check uses the real admin socket and no bearer/backdoor token.
+    for path in ["/api/config", "/api/status", "/api/logs", "/api/events", "/api/openapi.json", "/assets/app.js"]:
+        s, h, b = http(rust.admin_port, path)
+        check("admin.private." + path, s in (401, 403), status=s)
+    check("admin.login", rust.login() == 200)
+    s, h, b = rust.admin("/api/config")
+    envelope = json.loads(b)
+    check("admin.authenticated_config", s == 200 and "config" in envelope and "revision" in envelope, status=s)
+    s, h, b = rust.admin("/api/config", "PUT", {"expected_revision": envelope["revision"], "config": rust.base}, csrf=False)
+    check("admin.csrf_required", s == 403, status=s)
+    s, h, b = rust.admin("/api/config", origin="http://foreign.example")
+    check("admin.foreign_origin_denied", s == 403, status=s)
+
+    # Generate the disposable Basic Auth hash through the existing secured API.
+    s, h, b = rust.admin("/api/basic-auth/hash", "POST", {"username": "fixtureuser", "password": rust.password})
+    hashed = json.loads(b)
+    check("basic.hash_api", s == 200 and "password_hash" in hashed, status=s)
+    config = copy.deepcopy(rust.base)
+    config["vhosts"][0]["basic_auth"] = {"enabled": True, "require_tls": False,
+        "users": [{"username": "fixtureuser", "password_hash": hashed["password_hash"]}]}
+    rust.apply(config, "policy-basic-auth")
+    s, h, b = http(port, "/echo")
+    check("basic.challenge", s == 401 and "Basic" in h.get("www-authenticate", ""), status=s)
+    credentials = base64.b64encode(("fixtureuser:" + rust.password).encode()).decode()
+    s, h, b = http(port, "/echo", headers={"Authorization": "Basic " + credentials})
+    check("basic.correct_password_and_strip", s == 200 and "authorization" not in json.loads(b)["headers"], status=s)
+    s, h, b = http(port, "/echo", headers={"Authorization": "Basic Zml4dHVyZXVzZXI6d3Jvbmc="})
+    check("basic.wrong_password_denied", s == 401, status=s)
+    config["vhosts"][0]["acl"] = DENY
+    rust.apply(config, "policy-acl-before-basic")
+    s, h, b = http(port, "/echo")
+    check("basic.acl_precedes_auth", s == 403, status=s)
+    rust.reset()
+
+    config = copy.deepcopy(rust.base)
+    config["vhosts"][0]["frontend"]["listener_ids"] = ["does-not-exist"]
+    rust.apply(config, "invalid-config-retains-snapshot", expected_success=False)
+    s, h, b = http(port)
+    check("reload.invalid_document_retains_previous_snapshot", s == 200, status=s)
+    s, h, b = http(rust.static_port)
+    check("reload.invalid_document_keeps_other_vhost", s == 200, status=s)
+    config = copy.deepcopy(rust.base)
+    config["vhosts"][0]["backends"][0]["allow_private"] = False
+    # Explicit private-address rejection is checked through apply, preserving the
+    # active configuration if compile-time validation rejects the backend.
+    rust.apply(config, "ssrf-private-origin-disallowed", expected_success=False)
+    s, h, b = http(port)
+    check("ssrf.rejected_origin_does_not_replace_snapshot", s == 200, status=s)
+
+    query = "/echo?observability=" + rust.label
+    s, h, b = http(port, query)
+    request_id = json.loads(b)["headers"]["x-request-id"]
+    log = None
+    for _ in range(20):
+        ls, lh, lb = rust.admin("/api/logs?vhost=reverse&limit=512")
+        logs = json.loads(lb)
+        log = next((entry for entry in logs.get("entries", []) if entry.get("request_id") == request_id), None)
+        if log:
+            break
+        time.sleep(.05)
+    check("observability.log_fields_and_body_bytes", log is not None and log.get("status") == 200
+          and log.get("response_bytes") == len(b) and log.get("request_bytes") == 0
+          and log.get("cache_effective_enabled") is False and log.get("transform_effective_enabled") is False
+          and bool(re.fullmatch("[0-9a-f]{64}", log.get("request_id", ""))),
+          recorded_status=log.get("status") if log else None,
+          recorded_response_bytes=log.get("response_bytes") if log else None, response_bytes=len(b))
+    if log:
+        write_json(rust.fixture.results / f"{rust.label}-sample-exchange.json", log)
+    config = copy.deepcopy(rust.base)
+    config["diagnostic_headers"] = {"request_id": True}
+    rust.apply(config, "observability-static-eos")
+    s, h, b = http(rust.static_port)
+    static_request_id = h.get("x-request-id")
+    static_log = None
+    for _ in range(20):
+        ls, lh, lb = rust.admin("/api/logs?vhost=static&limit=64")
+        static_log = next((entry for entry in json.loads(lb).get("entries", [])
+                           if entry.get("request_id") == static_request_id), None)
+        if static_log:
+            break
+        time.sleep(.05)
+    check("observability.static_complete_after_content_length", s == 200 and static_log is not None
+          and static_log.get("outcome") == "complete" and static_log.get("response_bytes") == len(b),
+          outcome=static_log.get("outcome") if static_log else None,
+          recorded_response_bytes=static_log.get("response_bytes") if static_log else None,
+          response_bytes=len(b))
+    if static_log:
+        write_json(rust.fixture.results / f"{rust.label}-sample-static-exchange.json", static_log)
+    s, h, b = rust.admin("/api/status")
+    status = json.loads(b)
+    write_json(rust.fixture.results / f"{rust.label}-runtime-status.json", status)
+    check("observability.status_available", s == 200, status=s)

scripts/performance/checks_proxy.py

@@ -0,0 +1,135 @@
+"""Proxy streaming, status, header, request-ID and WebSocket tests."""
+import copy
+import hashlib
+from http.client import HTTPConnection, HTTPResponse
+import json
+import re
+import socket
+import time
+
+from common import http
+
+
+def run(check, rust, peer):
+    port = rust.reverse_port
+    for name in ["small.html", "large.bin", "huge.bin"]:
+        expected = (rust.fixture.site / name).read_bytes()
+        s, h, b = http(port, "/" + name)
+        check("proxy.get." + name, s == 200 and b == expected, status=s, bytes=len(b))
+        s, h, b = http(port, "/" + name, "HEAD")
+        check("proxy.head." + name, s == 200 and b == b"" and h.get("content-length") == str(len(expected)), status=s)
+    s, h, b = http(port, "/large.bin", headers={"Range": "bytes=0-127"})
+    check("proxy.range", s == 206 and b == bytes(range(128)), status=s, bytes=len(b))
+    s, h, b = http(port)
+    s, h, b = http(port, headers={"If-None-Match": h["etag"]})
+    check("proxy.conditional304", s == 304 and b == b"", status=s)
+    for status in [201, 400, 404, 500, 503]:
+        s, h, b = http(port, f"/status/{status}")
+        check(f"proxy.origin_status.{status}", s == status and b == b"origin status payload\n"
+              and h.get("x-origin-fixture") == "yes", status=s)
+    s, h, b = http(port, "/redirect")
+    check("proxy.redirect", s == 302 and h.get("location") == "/small.html", status=s)
+
+    payload = bytes(range(256)) * 4096
+    ids = []
+    for method in ["POST", "PUT"]:
+        s, h, b = http(port, "/echo?query=preserved", method,
+                       {"Content-Type": "application/octet-stream", "X-Request-ID": "untrusted-client-id"}, payload)
+        data = json.loads(b)
+        request_id = data["headers"].get("x-request-id", "")
+        ids.append(request_id)
+        check("proxy.upload." + method, s == 200 and data["method"] == method
+              and data["path"] == "/echo?query=preserved" and data["body_bytes"] == len(payload)
+              and data["body_sha256"] == hashlib.sha256(payload).hexdigest(), status=s, bytes=data["body_bytes"])
+        check("proxy.request_id." + method, bool(re.fullmatch("[0-9a-f]{64}", request_id)),
+              valid_64_hex=bool(re.fullmatch("[0-9a-f]{64}", request_id)))
+    check("proxy.request_id_unique", len(set(ids)) == 2)
+    s, h, b = http(port, "/echo", "GET", {"Content-Type": "application/octet-stream"}, b"nonempty-get-body")
+    data = json.loads(b)
+    check("proxy.nonempty_get_body_preserved", s == 200 and data["body_bytes"] == 17
+          and data["body_sha256"] == hashlib.sha256(b"nonempty-get-body").hexdigest(), status=s)
+
+    peer.started_upload.clear()
+    first, last = b"first streamed chunk" * 64, b"final streamed chunk" * 64
+    with socket.create_connection(("127.0.0.1", port), timeout=5) as sock:
+        sock.sendall((f"POST /stream-upload HTTP/1.1\r\nHost: 127.0.0.1:{port}\r\n"
+                      "Transfer-Encoding: chunked\r\nContent-Type: application/octet-stream\r\n\r\n").encode())
+        sock.sendall(f"{len(first):x}\r\n".encode() + first + b"\r\n")
+        forwarded_before_end = peer.started_upload.wait(2)
+        sock.sendall(f"{len(last):x}\r\n".encode() + last + b"\r\n0\r\n\r\n")
+        response = HTTPResponse(sock)
+        response.begin()
+        data = json.loads(response.read())
+        check("proxy.chunked_streaming_upload", forwarded_before_end and response.status == 200
+              and data["body_sha256"] == hashlib.sha256(first + last).hexdigest(),
+              first_chunk_arrived_before_end=forwarded_before_end, status=response.status)
+
+    peer.release_stream.clear()
+    connection = HTTPConnection("127.0.0.1", port, timeout=3)
+    connection.request("GET", "/stream-download")
+    started = time.monotonic()
+    response = connection.getresponse()
+    first = response.read(5)
+    first_seconds = time.monotonic() - started
+    peer.release_stream.set()
+    last = response.read()
+    connection.close()
+    check("proxy.streaming_response", response.status == 200 and first == b"first" and last == b"last"
+          and first_seconds < 2, first_chunk_seconds=first_seconds)
+    peer.release_stream.clear()
+    connection = HTTPConnection("127.0.0.1", port, timeout=3)
+    connection.request("GET", "/stream-download")
+    response = connection.getresponse()
+    response.read(5)
+    connection.close()
+    peer.release_stream.set()
+    s, h, b = http(port)
+    check("proxy.client_disconnect_survives", s == 200, status=s)
+
+    before = peer.count("POST", "/no-replay")
+    s, h, b = http(port, "/no-replay", "POST", {"Content-Type": "application/octet-stream"}, b"write-once")
+    after = peer.count("POST", "/no-replay")
+    check("proxy.failed_post_not_replayed", s in (502, 503, 504) and after - before == 1,
+          status=s, upstream_writes=after - before)
+
+    config = copy.deepcopy(rust.base)
+    config["headers"] = {"request": [{"op": "set", "name": "X-Fixture-Global", "value": "global"}],
+                         "response": [{"op": "set", "name": "X-Fixture-Global", "value": "response"}]}
+    config["vhosts"][0]["headers"] = {"request": [{"op": "set", "name": "X-Fixture-Vhost", "value": "vhost"}]}
+    config["vhosts"][0]["routes"] = [{"id": "echo", "path_prefix": "/echo", "headers": {
+        "request": [{"op": "set", "name": "X-Fixture-Route", "value": "route"}],
+        "response": [{"op": "set", "name": "X-Fixture-Route", "value": "response"}]}}]
+    rust.apply(config, "proxy-headers")
+    s, h, b = http(port, "/echo", headers={"Connection": "keep-alive, x-hop-fixture", "X-Hop-Fixture": "must-be-stripped"})
+    observed = json.loads(b)["headers"]
+    check("proxy.header_scopes", s == 200 and observed.get("x-fixture-global") == "global"
+          and observed.get("x-fixture-vhost") == "vhost" and observed.get("x-fixture-route") == "route"
+          and h.get("x-fixture-global") == "response" and h.get("x-fixture-route") == "response", status=s)
+    check("proxy.hop_by_hop_removed", "x-hop-fixture" not in observed)
+    s, h, b = http(port, "/echo", headers={"X-Forwarded-For": "203.0.113.22", "X-Real-IP": "203.0.113.22"})
+    observed = json.loads(b)["headers"]
+    check("proxy.untrusted_forwarded_ip", observed.get("x-real-ip") == "127.0.0.1"
+          and "203.0.113.22" not in observed.get("x-forwarded-for", ""),
+          observed_x_real_ip=observed.get("x-real-ip"), observed_x_forwarded_for=observed.get("x-forwarded-for"))
+
+    rust.reset()
+    with socket.create_connection(("127.0.0.1", port), timeout=4) as sock:
+        sock.sendall((f"GET /ws HTTP/1.1\r\nHost: 127.0.0.1:{port}\r\nUpgrade: websocket\r\n"
+                      "Connection: Upgrade\r\nSec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==\r\n"
+                      "Sec-WebSocket-Version: 13\r\n\r\n").encode())
+        buffer = b""
+        while b"\r\n\r\n" not in buffer:
+            piece = sock.recv(4096)
+            if not piece:
+                break
+            buffer += piece
+        upgraded = buffer.startswith(b"HTTP/1.1 101")
+        echoed = b""
+        if upgraded:
+            message, mask = b"hello", b"\x11\x22\x33\x44"
+            sock.sendall(bytes([0x81, 0x80 | len(message)]) + mask +
+                         bytes(c ^ mask[i % 4] for i, c in enumerate(message)))
+            frame = sock.recv(2)
+            if len(frame) == 2:
+                echoed = sock.recv(frame[1] & 127)
+        check("proxy.websocket_echo", upgraded and echoed == b"hello", upgraded=upgraded, bytes=len(echoed))

scripts/performance/checks_static.py

@@ -0,0 +1,132 @@
+"""Live static-file regression cases against RustProxy's network listener."""
+import copy
+from email.parser import BytesParser
+from email.policy import default
+import gzip
+
+from common import http
+
+
+def run(check, rust, peer):
+    port, fixture = rust.static_port, rust.fixture
+    for name in ["small.html", "large.bin", "huge.bin", "empty.txt"]:
+        expected = (fixture.site / name).read_bytes()
+        status, headers, body = http(port, "/" + name)
+        check(f"static.get.{name}", status == 200 and body == expected,
+              status=status, bytes=len(body))
+        head_status, head_headers, head_body = http(port, "/" + name, "HEAD")
+        check(f"static.head.{name}", head_status == 200 and head_body == b""
+              and head_headers.get("content-length") == str(len(expected)),
+              status=head_status, content_length=head_headers.get("content-length"))
+
+    status, headers, body = http(port)
+    tag = headers.get("etag")
+    modified = headers.get("last-modified")
+    check("static.validators_present", bool(tag and modified), etag=tag, last_modified=modified)
+    for label, condition in [("etag", {"If-None-Match": tag}),
+                             ("modified", {"If-Modified-Since": modified})]:
+        s, h, b = http(port, headers=condition)
+        check("static.conditional304." + label, s == 304 and b == b"", status=s, bytes=len(b))
+    s, h, b = http(port, headers={"If-None-Match": '"missing"'})
+    check("static.conditional_miss", s == 200 and b == body, status=s)
+
+    data = (fixture.site / "large.bin").read_bytes()
+    cases = [("prefix", "bytes=0-127", data[:128], "bytes 0-127/65536"),
+             ("suffix", "bytes=-37", data[-37:], "bytes 65499-65535/65536"),
+             ("open", "bytes=65500-", data[65500:], "bytes 65500-65535/65536")]
+    for name, range_header, expected, content_range in cases:
+        s, h, b = http(port, "/large.bin", headers={"Range": range_header})
+        check("static.range." + name, s == 206 and b == expected
+              and h.get("content-range") == content_range,
+              status=s, bytes=len(b), content_range=h.get("content-range"))
+    s, h, b = http(port, "/large.bin", headers={"Range": "bytes=999999-"})
+    check("static.range.unsatisfiable", s == 416 and h.get("content-range") == "bytes */65536",
+          status=s, content_range=h.get("content-range"))
+    s, h, b = http(port, "/large.bin", headers={"Range": "bytes=0-15,100-115"})
+    mime = BytesParser(policy=default).parsebytes(
+        b"Content-Type: " + h.get("content-type", "").encode() + b"\r\n\r\n" + b)
+    parts = list(mime.iter_parts())
+    check("static.range.multipart", s == 206 and len(parts) == 2
+          and parts[0].get_payload(decode=True) == data[:16]
+          and parts[1].get_payload(decode=True) == data[100:116], status=s, parts=len(parts))
+    for name, value, expected_status in [("match", tag, 206), ("mismatch", '"other"', 200)]:
+        s, h, b = http(port, headers={"Range": "bytes=0-15", "If-Range": value})
+        expected = body[:16] if expected_status == 206 else body
+        check("static.if_range." + name, s == expected_status and b == expected,
+              status=s, bytes=len(b))
+
+    s, h, b = http(port, "/directory")
+    check("static.directory_slash", s in (301, 308) and h.get("location", "").endswith("/directory/"),
+          status=s, location=h.get("location"))
+    s, h, b = http(port, "/directory/")
+    check("static.directory_index", s == 200 and b == b"directory index\n", status=s)
+    s, h, b = http(port, "/listing/")
+    check("static.directory_listing_denied", s == 403, status=s)
+    for path in ["/escape-link.txt", "/inside-link.txt"]:
+        s, h, b = http(port, path)
+        check("static.symlink_deny." + path, s == 403 and b"MUST-NOT" not in b, status=s)
+    for path in ["/%2e%2e/outside-secret.txt", "/%2fetc/passwd", "/%252e%252e/outside-secret.txt",
+                 "/%00", "/..%5coutside-secret.txt"]:
+        s, h, b = http(port, path)
+        check("static.invalid_path." + path, s in (400, 403, 404) and b"MUST-NOT" not in b,
+              status=s)
+
+    config = copy.deepcopy(rust.base)
+    service = config["web_services"][0]
+    service["symlink_policy"] = "inside_root"
+    service["directory_listing"] = True
+    rust.apply(config, "static-inside-root")
+    s, h, b = http(port, "/inside-link.txt")
+    check("static.symlink_inside_root_allowed", s == 200 and b == b"download fixture\n", status=s)
+    s, h, b = http(port, "/escape-link.txt")
+    check("static.symlink_inside_root_escape_denied", s == 403 and b"MUST-NOT" not in b, status=s)
+    s, h, b = http(port, "/listing/")
+    check("static.directory_listing_enabled", s == 200 and b"visible.txt" in b, status=s)
+    # Alternate a symlink target between valid and outside-root files. The file
+    # read must stay confined even when a pathname changes between requests.
+    link = fixture.site / "changing-link.txt"
+    race_statuses = []
+    escaped = False
+    for index in range(12):
+        if link.is_symlink():
+            link.unlink()
+        link.symlink_to(fixture.outside if index % 2 else fixture.site / "download.txt")
+        s, h, b = http(port, "/changing-link.txt")
+        race_statuses.append(s)
+        escaped = escaped or b"MUST-NOT" in b
+    check("static.changing_symlink_confinement", not escaped and all(s in (200, 403) for s in race_statuses),
+          statuses=race_statuses, note="Sequential target replacement; not a concurrent rename-race proof")
+
+    config = copy.deepcopy(rust.base)
+    config["web_services"][0]["precompressed"] = {"enabled": True, "gzip": True, "brotli": False}
+    rust.apply(config, "static-precompressed")
+    s, h, b = http(port, headers={"Accept-Encoding": "gzip"})
+    sidecar = (fixture.site / "small.html.gz").read_bytes()
+    check("static.precompressed_gzip", s == 200 and h.get("content-encoding") == "gzip"
+          and b == sidecar and gzip.decompress(b) == body, status=s, bytes=len(b))
+    s, h, b = http(port, headers={"Accept-Encoding": "gzip", "Range": "bytes=0-9"})
+    check("static.precompressed_range", s == 206 and b == sidecar[:10], status=s, bytes=len(b))
+
+    config = copy.deepcopy(rust.base)
+    config["web_services"][0]["try_files"] = ["$uri", "/small.html"]
+    config["web_services"][0]["rewrites"] = [{"pattern": "^/old$", "replacement": "/small.html", "status": 302}]
+    config["web_services"][0]["download_extensions"] = ["txt"]
+    config["web_services"][0]["mime_types"] = {"bin": "application/x-fixture"}
+    rust.apply(config, "static-routing")
+    s, h, b = http(port, "/spa/deep/link")
+    check("static.try_files_spa", s == 200 and b == body, status=s)
+    s, h, b = http(port, "/old")
+    check("static.rewrite_redirect", s == 302 and h.get("location") == "/small.html", status=s)
+    s, h, b = http(port, "/download.txt")
+    check("static.download_header", s == 200 and "attachment" in h.get("content-disposition", ""), status=s)
+    s, h, b = http(port, "/large.bin")
+    check("static.mime_override", s == 200 and h.get("content-type") == "application/x-fixture", status=s)
+
+    config = copy.deepcopy(rust.base)
+    (fixture.site / ".htaccess").write_text("Header set X-Fixture-Htaccess yes\nDirectoryIndex small.html\n")
+    config["web_services"][0]["htaccess"] = {"enabled": True, "strict": True}
+    rust.apply(config, "static-htaccess")
+    s, h, b = http(port)
+    check("static.htaccess_header", s == 200 and h.get("x-fixture-htaccess") == "yes", status=s)
+    s, h, b = http(port, "/.htaccess")
+    check("static.htaccess_not_served", s in (403, 404) and b"Header set" not in b, status=s)

scripts/performance/common.py

@@ -0,0 +1,448 @@
+#!/usr/bin/env python3
+"""Disposable, loopback-only fixtures for testing actual RustProxy processes.
+
+Runtime databases and credentials live in TemporaryDirectory, never in results.
+No pre-existing SQLite database, fixed port, or historical /mnt/data path is used.
+"""
+from __future__ import annotations
+
+import copy
+import gzip
+import hashlib
+from http.client import HTTPConnection, HTTPException
+import json
+import os
+from pathlib import Path
+import pwd
+import secrets
+import shutil
+import signal
+import socket
+import subprocess
+import tempfile
+import time
+
+
+def sha256(path: Path) -> str:
+    result = hashlib.sha256()
+    with path.open("rb") as stream:
+        for block in iter(lambda: stream.read(1024 * 1024), b""):
+            result.update(block)
+    return result.hexdigest()
+
+
+def write_json(path: Path, value) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    path.write_text(json.dumps(value, indent=2, sort_keys=True) + "\n")
+
+
+def seal_results(directory: Path) -> None:
+    """Checksums cover every regular evidence file, except the checksum manifest."""
+    paths = sorted(p for p in directory.rglob("*") if p.is_file()
+                   and p.name != "SHA256SUMS")
+    (directory / "SHA256SUMS").write_text("".join(
+        f"{sha256(path)}  {path.relative_to(directory)}\n" for path in paths))
+
+
+def http(port: int, path="/small.html", method="GET", headers=None, body=None,
+         timeout=10, source_address=None):
+    connection = HTTPConnection("127.0.0.1", port, timeout=timeout,
+                                source_address=source_address)
+    try:
+        connection.request(method, path, body, headers or {})
+        response = connection.getresponse()
+        payload = response.read()
+        return response.status, {k.lower(): v for k, v in response.getheaders()}, payload
+    finally:
+        connection.close()
+
+
+def free_port() -> int:
+    with socket.socket() as sock:
+        sock.bind(("127.0.0.1", 0))
+        return sock.getsockname()[1]
+
+
+def wait_ready(port: int, process: subprocess.Popen, timeout=30):
+    deadline = time.monotonic() + timeout
+    while time.monotonic() < deadline:
+        if process.poll() is not None:
+            raise RuntimeError(f"process {process.pid} exited with {process.returncode}")
+        try:
+            http(port, "/", timeout=.5)
+            return
+        except (OSError, HTTPException):
+            time.sleep(.05)
+    raise RuntimeError(f"loopback listener {port} did not start")
+
+
+class Fixture:
+    def __init__(self, results: Path, workers=4, temp_parent: Path | None = None,
+                 server_cpus=None, origin_cpus=None, origin_workers=None):
+        self.results = results.resolve()
+        self.results.mkdir(parents=True, exist_ok=True)
+        # Keep live state outside the evidence tree. Even an interrupted run or
+        # workspace synchronization must never put databases/keys in results.
+        self.temp = tempfile.TemporaryDirectory(prefix="rustproxy-perf-", dir=temp_parent)
+        self.root = Path(self.temp.name)
+        self.root.chmod(0o755)  # nginx workers must traverse the public fixture root
+        self.site = self.root / "site"
+        self.site.mkdir(mode=0o755)
+        self.workers = workers
+        self.server_cpus = server_cpus
+        self.origin_cpus = origin_cpus
+        self.origin_workers = origin_workers or workers
+        self.children = []
+        self.handles = []
+        self.ports = set()
+        self.origin_port = self.port()
+        self.make_site()
+
+    def port(self):
+        while True:
+            port = free_port()
+            if port not in self.ports:
+                self.ports.add(port)
+                return port
+
+    def make_site(self):
+        # Exact, reproducible payload sizes match the original report's cases.
+        small = b"<!doctype html><html><body>RustProxy benchmark "
+        small += b"x" * (1040 - len(small) - len(b"</body></html>"))
+        small += b"</body></html>"
+        files = {"small.html": small,
+                 "large.bin": bytes(range(256)) * 256,
+                 "huge.bin": bytes(range(256)) * 4096,
+                 "empty.txt": b"",
+                 "app.json": (b'{\n  "message": "stable fixture",\n  "values": [1, 2, 3]\n}\n'),
+                 "app.css": b"/* fixture comment */\nbody { color: red; padding: 0px; }\n",
+                 "compress.txt": (b"compressible RustProxy text fixture. " * 4096),
+                 "download.txt": b"download fixture\n"}
+        for name, body in files.items():
+            (self.site / name).write_bytes(body)
+        (self.site / "small.html.gz").write_bytes(gzip.compress(small, mtime=0))
+        (self.site / "directory").mkdir()
+        (self.site / "directory" / "index.html").write_bytes(b"directory index\n")
+        (self.site / "listing").mkdir()
+        (self.site / "listing" / "visible.txt").write_bytes(b"listing fixture\n")
+        (self.site / "inside-link.txt").symlink_to(self.site / "download.txt")
+        self.outside = self.root / "outside-secret.txt"
+        self.outside.write_bytes(b"MUST-NOT-BE-SERVED-OUTSIDE-ROOT\n")
+        (self.site / "escape-link.txt").symlink_to(self.outside)
+        write_json(self.results / "payloads.json", {
+            name: {"bytes": len(body), "sha256": hashlib.sha256(body).hexdigest()}
+            for name, body in files.items()})
+
+    def spawn(self, argv, name, env=None, affinity=None):
+        log = (self.results / f"{name}.log").open("w")
+        self.handles.append(log)
+        if affinity:
+            argv = ["taskset", "-c", affinity, *argv]
+        child = subprocess.Popen([str(x) for x in argv], stdout=log,
+                                 stderr=subprocess.STDOUT, env=env, start_new_session=True)
+        self.children.append(child)
+        return child
+
+    @staticmethod
+    def stop(process):
+        if process.poll() is None:
+            os.killpg(process.pid, signal.SIGTERM)
+            try:
+                process.wait(timeout=12)
+            except subprocess.TimeoutExpired:
+                os.killpg(process.pid, signal.SIGKILL)
+                process.wait(timeout=3)
+
+    def close(self):
+        for process in reversed(self.children):
+            self.stop(process)
+        for handle in self.handles:
+            handle.close()
+        self.temp.cleanup()
+        write_json(self.results / "fixture-cleanup.json", {
+            "temporary_state_removed": not self.root.exists(),
+            "state_was_outside_results": not self.root.is_relative_to(self.results)})
+        if self.root.exists():
+            raise RuntimeError("temporary runtime state was not removed; do not package the fixture directory")
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, *_):
+        self.close()
+
+    def base_config(self, reverse_port, static_port):
+        endpoints = []
+        for label, port in [("reverse", reverse_port), ("static", static_port)]:
+            endpoints.append({"id": label, "name": label, "scope": "internal",
+                              "enabled": True, "binding": {
+                                  "address": f"127.0.0.1:{port}", "tls": False,
+                                  "http2": False, "http3": None}})
+        return {
+            "cache": {"enabled": False},
+            "geoip_updates": {"enabled": False},
+            "listeners": {"endpoints": endpoints},
+            "web_services": [{"id": "fixture-static", "name": "Fixture static",
+                              "document_root": str(self.site), "index": ["index.html", "small.html"],
+                              "directory_listing": False}],
+            "vhosts": [{"id": "reverse", "frontend": {
+                "hosts": ["127.0.0.1", "localhost", "fixture.test"],
+                "listener_ids": ["reverse"]},
+                "backends": [{"id": "origin", "url": f"http://127.0.0.1:{self.origin_port}",
+                              "allow_private": True, "preserve_host": False}],
+                "load_balancing": {"health_check": {"mode": "off"}}, "routes": []},
+                {"id": "static", "frontend": {
+                    "hosts": ["127.0.0.1", "localhost", "fixture.test"],
+                    "listener_ids": ["static"]},
+                 "source": {"type": "web_server", "web_service_id": "fixture-static"},
+                 "routes": []}]
+        }
+
+    def rust(self, binary: Path, label: str, max_idle=None):
+        return RustInstance(self, binary.resolve(), label, max_idle=max_idle)
+
+    def nginx(self, executable: Path, frontend=True):
+        """Use a separate nginx master for the shared origin and comparator."""
+        executable = executable.resolve()
+        result = subprocess.run([str(executable), "-V"], capture_output=True, text=True, timeout=10)
+        (self.results / "nginx-version.txt").write_text(result.stdout + result.stderr)
+        origin_conf = self.nginx_config("nginx-origin", f"""
+          server {{ listen 127.0.0.1:{self.origin_port}; server_name _;
+            root "{self.site}"; location / {{ add_header Cache-Control "public, max-age=60"; }}
+          }}
+        """, workers=self.origin_workers)
+        origin = self.spawn([executable, "-p", self.root, "-c", origin_conf], "nginx-origin",
+                            affinity=self.origin_cpus)
+        wait_ready(self.origin_port, origin)
+        result = {"origin": origin, "origin_port": self.origin_port}
+        if frontend:
+            reverse_port, static_port = self.port(), self.port()
+            conf = self.nginx_config("nginx-frontend", f"""
+              upstream fixture_origin {{ server 127.0.0.1:{self.origin_port}; keepalive 64; }}
+              server {{ listen 127.0.0.1:{reverse_port}; server_name _;
+                location / {{ proxy_pass http://fixture_origin; proxy_http_version 1.1;
+                    proxy_set_header Connection ""; proxy_buffering off; }}
+              }}
+              server {{ listen 127.0.0.1:{static_port}; server_name _; root "{self.site}"; }}
+            """)
+            process = self.spawn([executable, "-p", self.root, "-c", conf], "nginx-frontend",
+                                 affinity=self.server_cpus)
+            wait_ready(reverse_port, process)
+            wait_ready(static_port, process)
+            result.update({"process": process, "reverse_port": reverse_port,
+                           "static_port": static_port})
+        return result
+
+    def nginx_config(self, name, servers, workers=None):
+        path = self.root / f"{name}.conf"
+        text = f"""daemon off;
+master_process on;
+user {pwd.getpwuid(os.geteuid()).pw_name};
+worker_processes {workers or self.workers};
+pid {self.root}/{name}.pid;
+error_log stderr warn;
+events {{ worker_connections 8192; }}
+http {{
+  access_log off;
+  sendfile on;
+  tcp_nopush on;
+  keepalive_timeout 65;
+  keepalive_requests 1000000;
+  default_type application/octet-stream;
+  types {{ text/html html; text/css css; application/json json; text/plain txt; }}
+  client_body_temp_path {self.root}/{name}-body;
+  proxy_temp_path {self.root}/{name}-proxy;
+  {servers}
+}}
+"""
+        path.write_text(text)
+        (self.results / path.name).write_text(text)
+        return path
+
+
+class RustInstance:
+    def __init__(self, fixture, binary, label, max_idle=None):
+        self.fixture, self.binary, self.label = fixture, binary, label
+        if not os.access(binary, os.X_OK):
+            target = fixture.root / f"{label}-rust-proxy"
+            shutil.copyfile(binary, target)
+            target.chmod(0o700)
+            self.binary = target
+        self.admin_port, self.reverse_port, self.static_port = (fixture.port() for _ in range(3))
+        self.data = fixture.root / f"{label}-data"
+        self.base = fixture.base_config(self.reverse_port, self.static_port)
+        if max_idle is not None:
+            self.base["vhosts"][0]["load_balancing"]["keepalive"] = {"max_idle_per_host": max_idle}
+        self.password = "Fixture!" + secrets.token_hex(20)
+        self.username = "performance"
+        self.env = os.environ.copy()
+        self.env["TOKIO_WORKER_THREADS"] = str(fixture.workers)
+        self.env["RUST_PROXY_PROXY_LISTEN"] = f"127.0.0.1:{self.reverse_port}"
+        self.env["RUST_PROXY_UPDATE_CHECK_INTERVAL_SECS"] = "0"
+        self.env.pop("RUST_PROXY_TLS_LISTEN", None)
+        self.applies = []
+        self.cli("admin", "init", "--username", self.username, "--password-stdin",
+                 input=self.password + "\n")
+        self.start("bootstrap")
+        self.apply(self.base, "base")
+        fixture.stop(self.process)
+        self.start("runtime")
+        wait_ready(self.static_port, self.process)
+        version = self.cli("--version").stdout.strip()
+        write_json(fixture.results / f"{label}-binary.json", {
+            "label": label, "path": str(binary), "sha256": sha256(binary),
+            "version": version, "worker_threads": fixture.workers,
+            "pid": self.process.pid, "protocol": "HTTP/1.1 plaintext loopback",
+            "cpu_affinity": fixture.server_cpus,
+            "max_idle_per_host_override": max_idle})
+
+    def cli(self, *args, input=None, check=True):
+        result = subprocess.run([str(self.binary), "--data-dir", str(self.data),
+                                 "--admin-listen", f"127.0.0.1:{self.admin_port}", *args],
+                                input=input, text=True, capture_output=True,
+                                env=self.env, timeout=40)
+        if check and result.returncode:
+            raise RuntimeError(f"{self.label} CLI {' '.join(args[:2])}: "
+                               f"{result.stdout[-2000:]} {result.stderr[-3000:]}")
+        return result
+
+    def start(self, phase):
+        self.process = self.fixture.spawn([
+            self.binary, "--data-dir", self.data, "--admin-listen", f"127.0.0.1:{self.admin_port}",
+            "--update-check-interval-secs", "0", "serve", "--proxy-listen",
+            f"127.0.0.1:{self.reverse_port}", "--max-connections", "4096", "--max-inflight", "4096"],
+            f"{self.label}-{phase}", env=self.env, affinity=self.fixture.server_cpus)
+        wait_ready(self.admin_port, self.process)
+        wait_ready(self.reverse_port, self.process)
+
+    def apply(self, config, name, expected_success=True):
+        config_path = self.fixture.root / f"{self.label}-{name}.json"
+        write_json(config_path, config)
+        result = self.cli("config", "apply", str(config_path), check=False)
+        # Configurations with ephemeral test passwords/hashes are intentionally not copied.
+        safe = copy.deepcopy(config)
+        for vhost in safe.get("vhosts", []):
+            if "basic_auth" in vhost:
+                vhost["basic_auth"]["users"] = "REDACTED fixture users"
+        write_json(self.fixture.results / "configs" / f"{self.label}-{name}.json", safe)
+        event = {"name": name, "sha256": sha256(config_path), "returncode": result.returncode,
+                 "expected_success": expected_success, "stdout": result.stdout[-2000:],
+                 "stderr": result.stderr[-3000:]}
+        self.applies.append(event)
+        write_json(self.fixture.results / f"{self.label}-config-applies.json", self.applies)
+        if (result.returncode == 0) != expected_success:
+            raise RuntimeError(f"config {name}: {result.stdout} {result.stderr}")
+        return result
+
+    def reset(self):
+        return self.apply(self.base, "restored-base")
+
+    def login(self):
+        status, headers, body = http(self.admin_port, "/api/auth/login", "POST",
+            {"Origin": f"http://127.0.0.1:{self.admin_port}", "Content-Type": "application/json",
+             "X-RustProxy-Request": "1"},
+            json.dumps({"username": self.username, "password": self.password}))
+        if status != 200:
+            raise RuntimeError(f"fixture login failed: HTTP {status}: {body[:200]!r}")
+        self.cookie = headers["set-cookie"].split(";", 1)[0]
+        self.csrf = json.loads(body)["csrf_token"]
+        return status
+
+    def wait_quiescent(self, timeout=180, quiet_seconds=5, poll_seconds=.5):
+        """Wait for the real, one-time ACME bootstrap before any timed work.
+
+        This does not disable ACME or change the process environment. The
+        private fixture has no issuers/jobs; a terminal unavailable state is
+        valid for HTTP tests, but does not establish ACME issuance support.
+        """
+        from resources import descendants, process_stats
+
+        if timeout <= 0 or quiet_seconds <= 0 or poll_seconds <= 0 or timeout <= quiet_seconds:
+            raise ValueError("quiescence timeout must exceed the positive quiet period")
+        started = time.monotonic()
+        deadline = started + timeout
+        output = self.fixture.results / f"{self.label}-quiescence.json"
+        report = {"completed": False, "label": self.label, "pid": self.process.pid,
+                  "timeout_seconds": timeout, "required_quiet_seconds": quiet_seconds,
+                  "poll_seconds": poll_seconds, "observations": [],
+                  "acme_issuance_tested": False}
+
+        def read_admin(path):
+            remaining = deadline - time.monotonic()
+            if remaining <= 0:
+                raise TimeoutError(f"{self.label} did not become quiescent within {timeout}s")
+            if self.process.poll() is not None:
+                raise RuntimeError(f"{self.label} exited during quiescence")
+            status, _, body = self.admin(path, timeout=min(1, remaining))
+            if status != 200:
+                raise RuntimeError(f"{self.label} quiescence {path}: HTTP {status}")
+            return json.loads(body)
+
+        try:
+            root = process_stats(self.process.pid)
+            if root is None:
+                raise RuntimeError(f"{self.label} has no live root process")
+            identity = (root["pid"], root["start_ticks"])
+            report["start_ticks"] = root["start_ticks"]
+            envelope = read_admin("/api/config")
+            if not isinstance(envelope, dict) or not isinstance(envelope.get("config"), dict):
+                raise RuntimeError("quiescence requires the actual configuration envelope")
+            certificates = envelope["config"].get("tls", {}).get("certificates", [])
+            if not isinstance(certificates, list) or any(not isinstance(cert, dict) for cert in certificates):
+                raise RuntimeError("unexpected certificate configuration during quiescence")
+            report["configured_issuers"] = sum(cert.get("issuer") is not None for cert in certificates)
+            if report["configured_issuers"]:
+                raise RuntimeError("performance fixture must have no configured ACME issuers")
+            quiet_started, previous_terminal = None, None
+            while True:
+                payload = read_admin("/api/acme")
+                if not isinstance(payload, dict) or not {"runtime", "job", "renewals"} <= payload.keys():
+                    raise RuntimeError("quiescence requires explicit ACME runtime/job/renewal state")
+                if payload["job"] is not None or payload["renewals"] != {}:
+                    raise RuntimeError("performance fixture has an unexpected ACME job or renewal")
+                runtime = payload["runtime"]
+                if not isinstance(runtime, dict) or runtime.get("status") not in {
+                        "checking", "initializing", "ready", "unavailable"}:
+                    raise RuntimeError("unexpected ACME runtime state during quiescence")
+                pids = descendants(self.process.pid)
+                root = process_stats(self.process.pid)
+                if root is None or (root["pid"], root["start_ticks"]) != identity:
+                    raise RuntimeError(f"{self.label} root process identity changed during quiescence")
+                now = time.monotonic()
+                if now >= deadline:
+                    raise TimeoutError(f"{self.label} did not become quiescent within {timeout}s")
+                terminal = tuple(runtime.get(key) for key in ("status", "mode", "message"))
+                quiet = runtime["status"] in {"ready", "unavailable"} and pids == [self.process.pid]
+                if not quiet:
+                    quiet_started, previous_terminal = None, None
+                elif quiet_started is None or terminal != previous_terminal:
+                    quiet_started, previous_terminal = now, terminal
+                elapsed_quiet = now - quiet_started if quiet_started is not None else 0
+                report["observations"].append({
+                    "elapsed_seconds": now - started, "process_count": len(pids),
+                    "acme_status": runtime["status"], "acme_mode": runtime.get("mode"),
+                    "acme_message": runtime.get("message"), "quiet_seconds": elapsed_quiet})
+                if elapsed_quiet >= quiet_seconds:
+                    report.update({"completed": True, "elapsed_seconds": now - started,
+                                   "terminal_status": runtime["status"], "process_count": len(pids)})
+                    write_json(output, report)
+                    return report
+                write_json(output, report)
+                time.sleep(min(poll_seconds, max(0, deadline - time.monotonic())))
+        except (Exception, KeyboardInterrupt) as error:
+            report.update({"elapsed_seconds": time.monotonic() - started,
+                           "error_type": type(error).__name__, "error": str(error)})
+            write_json(output, report)
+            raise
+
+    def admin(self, path, method="GET", body=None, csrf=True, cookie=True, origin=None,
+              timeout=10):
+        headers = {"Origin": origin or f"http://127.0.0.1:{self.admin_port}", "X-RustProxy-Request": "1"}
+        if cookie:
+            headers["Cookie"] = self.cookie
+        if csrf:
+            headers["X-CSRF-Token"] = self.csrf
+        if body is not None:
+            headers["Content-Type"] = "application/json"
+            body = json.dumps(body)
+        return http(self.admin_port, path, method, headers, body, timeout=timeout)

scripts/performance/concurrency_regression.py

@@ -0,0 +1,138 @@
+#!/usr/bin/env python3
+"""Finite live HTTP concurrency correctness stress; no throughput/latency claim."""
+from __future__ import annotations
+
+import argparse
+from concurrent.futures import ThreadPoolExecutor
+import hashlib
+import json
+from pathlib import Path
+import threading
+import time
+import traceback
+
+from common import Fixture, http, seal_results, sha256, write_json
+from resources import process_stats
+
+
+PAYLOADS = ["small.html", "large.bin", "huge.bin"]
+
+
+def active_backends(rust):
+    status, _, body = rust.admin("/api/status")
+    if status != 200:
+        return {"error": f"HTTP {status}"}
+    values = []
+
+    def walk(value):
+        if isinstance(value, dict):
+            if "id" in value and "active" in value and "status" in value and "requests" in value:
+                values.append({key: value[key] for key in ("id", "active", "status", "requests")})
+            for child in value.values():
+                walk(child)
+        elif isinstance(value, list):
+            for child in value:
+                walk(child)
+    walk(json.loads(body))
+    return values
+
+
+def batch(rust, port, name, expected, concurrency):
+    count = max(32, concurrency * 2)
+    expected_sha = hashlib.sha256(expected).hexdigest()
+    ready = threading.Barrier(concurrency, timeout=15)
+    lock = threading.Lock()
+    running = peak = 0
+
+    def fetch(index):
+        nonlocal running, peak
+        with lock:
+            running += 1
+            peak = max(peak, running)
+        try:
+            if index < concurrency:
+                ready.wait()
+            status, headers, body = http(port, "/" + name, timeout=15)
+            digest = hashlib.sha256(body).hexdigest()
+            return {"index": index, "status": status, "bytes": len(body), "sha256": digest,
+                    "passed": status == 200 and len(body) == len(expected) and digest == expected_sha}
+        except Exception as error:
+            return {"index": index, "passed": False, "error": str(error)}
+        finally:
+            with lock:
+                running -= 1
+
+    before = process_stats(rust.process.pid)
+    started = time.monotonic()
+    with ThreadPoolExecutor(max_workers=concurrency) as executor:
+        responses = list(executor.map(fetch, range(count)))
+    elapsed = time.monotonic() - started
+    # Completion means every client consumed or failed its full response. Give
+    # asynchronous server bookkeeping a bounded interval to release its leases.
+    for _ in range(40):
+        active = active_backends(rust)
+        if isinstance(active, list) and active and all(row["active"] == 0 for row in active):
+            break
+        time.sleep(.05)
+    after = process_stats(rust.process.pid)
+    return {"configured_concurrent_clients": concurrency, "peak_running_client_tasks": peak,
+            "client_tasks_include_barrier": True, "requests": count, "elapsed_seconds": elapsed,
+            "expected_bytes": len(expected), "expected_sha256": expected_sha,
+            "response_failures": sum(not response["passed"] for response in responses),
+            "passed": all(response["passed"] for response in responses)
+                      and isinstance(active, list) and bool(active) and all(row["active"] == 0 for row in active),
+            "responses": responses, "active_backends_after_drain": active,
+            "server_before": before, "server_after_drain": after}
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--binary", action="append", required=True, metavar="LABEL=/ABS/PATH")
+    parser.add_argument("--nginx", type=Path, required=True)
+    parser.add_argument("--out", type=Path, required=True)
+    args = parser.parse_args()
+    if args.out.exists() and any(args.out.iterdir()):
+        parser.error("--out must be new or empty")
+    reports = {}
+    for specification in args.binary:
+        label, binary = specification.split("=", 1)
+        cases, error = [], None
+        try:
+            with Fixture(args.out / label, workers=4, origin_workers=2) as fixture:
+                fixture.nginx(args.nginx, frontend=False)
+                rust = fixture.rust(Path(binary), label)
+                rust.login()
+                for source, port in [("proxy", rust.reverse_port), ("static", rust.static_port)]:
+                    for name in PAYLOADS:
+                        expected = (fixture.site / name).read_bytes()
+                        for concurrency in [1, 16, 128]:
+                            result = batch(rust, port, name, expected, concurrency)
+                            case = {"case": f"concurrency.{source}.{name}.c{concurrency}", **result}
+                            cases.append(case)
+                            write_json(args.out / label / "cases" / f"{source}-{name}-c{concurrency}.json", case)
+                            print(f"{label} {'PASS' if case['passed'] else 'FAIL'} {case['case']}", flush=True)
+                if rust.process.poll() is not None:
+                    raise RuntimeError("RustProxy exited during concurrency regression")
+        except Exception as exc:
+            error = {"error": str(exc), "traceback": traceback.format_exc()}
+            write_json(args.out / label / "failure.json", error)
+        report = {"kind": "live HTTP/1.1 finite concurrency functional stress", "simulation": False,
+                  "binary_sha256": sha256(Path(binary)), "throughput_measurement": False,
+                  "passed_cases": sum(case["passed"] for case in cases),
+                  "failed_cases": sum(not case["passed"] for case in cases),
+                  "requests": sum(case["requests"] for case in cases),
+                  "response_failures": sum(case["response_failures"] for case in cases),
+                  "completed": error is None and len(cases) == 18,
+                  "cases": [{key: case[key] for key in ("case", "passed", "requests", "response_failures")} for case in cases],
+                  "limitations": ["configured client concurrency is not proof of simultaneous active server exchanges",
+                                  "Python client is not a throughput or latency benchmark",
+                                  "FD/RSS are before/after snapshots, not peaks or a long-term leak test"]}
+        reports[label] = report
+        write_json(args.out / label / "report.json", report)
+    write_json(args.out / "comparison.json", reports)
+    seal_results(args.out)
+    return 0 if all(report["completed"] and not report["failed_cases"] for report in reports.values()) else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

scripts/performance/lb_regression.py

@@ -0,0 +1,223 @@
+#!/usr/bin/env python3
+"""Small live two-origin failover and active-exchange regression, not a benchmark."""
+from __future__ import annotations
+
+import argparse
+from collections import Counter
+import copy
+import hashlib
+from http.client import HTTPConnection
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+import json
+from pathlib import Path
+import threading
+import time
+import traceback
+
+from common import Fixture, http, seal_results, write_json
+
+
+class Origin:
+    def __init__(self, port, name):
+        self.name, self.port = name, port
+        self.fail_get = False
+        self.counts, self.post_hashes = Counter(), []
+        self.release, self.lock = threading.Event(), threading.Lock()
+        origin = self
+
+        class Handler(BaseHTTPRequestHandler):
+            protocol_version = "HTTP/1.1"
+
+            def log_message(self, *_):
+                pass
+
+            def do_GET(self):
+                self.respond()
+
+            def do_POST(self):
+                self.respond()
+
+            def respond(self):
+                body = self.rfile.read(int(self.headers.get("Content-Length", "0")))
+                with origin.lock:
+                    origin.counts[(self.command, self.path)] += 1
+                    if self.command == "POST":
+                        origin.post_hashes.append(hashlib.sha256(body).hexdigest())
+                hold = self.path == "/hold"
+                status = 503 if self.command == "POST" or origin.fail_get else 200
+                payload = b"firstlast" if hold else origin.name.encode()
+                try:
+                    self.send_response(status)
+                    self.send_header("Content-Length", str(len(payload)))
+                    self.send_header("Content-Type", "application/octet-stream")
+                    self.send_header("Cache-Control", "no-store")
+                    # Fresh connections make refused-listener tests independent
+                    # of already accepted upstream keepalive connections.
+                    self.send_header("Connection", "close")
+                    self.end_headers()
+                    if hold:
+                        self.wfile.write(payload[:5])
+                        self.wfile.flush()
+                        origin.release.wait(15)
+                        self.wfile.write(payload[5:])
+                    else:
+                        self.wfile.write(payload)
+                    self.wfile.flush()
+                except (BrokenPipeError, ConnectionResetError):
+                    pass
+                self.close_connection = True
+
+        self.server = ThreadingHTTPServer(("127.0.0.1", port), Handler)
+        self.thread = threading.Thread(target=self.server.serve_forever, daemon=True)
+        self.thread.start()
+        self.closed = False
+
+    def close(self):
+        self.release.set()
+        if not self.closed:
+            self.server.shutdown()
+            self.server.server_close()
+            self.thread.join(timeout=3)
+            self.closed = True
+
+
+def backend_views(rust):
+    status, _, body = rust.admin("/api/status")
+    if status != 200:
+        raise RuntimeError(f"status unavailable: {status}")
+    found = {}
+
+    def walk(value):
+        if isinstance(value, dict):
+            if value.get("id") in ("a", "b") and "active" in value and "status" in value:
+                found[value["id"]] = {key: value[key] for key in value
+                    if key in ("id", "status", "active", "requests", "errors", "bytes_out")
+                    or key.startswith("retry_") or key.startswith("connect_errors")}
+            for child in value.values():
+                walk(child)
+        elif isinstance(value, list):
+            for child in value:
+                walk(child)
+    walk(json.loads(body))
+    return found
+
+
+def run(check, fixture, binary, label):
+    a, b = Origin(fixture.origin_port, "a"), Origin(fixture.port(), "b")
+    try:
+        rust = fixture.rust(binary, label)
+        rust.login()
+        config = copy.deepcopy(rust.base)
+        vhost = config["vhosts"][0]
+        vhost["backends"] = [{"id": origin.name, "url": f"http://127.0.0.1:{origin.port}",
+                               "allow_private": True, "preserve_host": False} for origin in [a, b]]
+        vhost["load_balancing"] = {"algorithm": "round_robin", "health_check": {"mode": "off"},
+            "slow_start_seconds": 0, "passive_health": {"enabled": False},
+            "retry": {"max_retries": 1, "retry_statuses": [503]}}
+        vhost["routes"] = [{"id": "hold-a", "path_prefix": "/hold", "backend_id": "a"}]
+        rust.apply(config, "lb-two-origins")
+        replies = [http(rust.reverse_port, "/rr") for _ in range(8)]
+        counts = Counter(body.decode() for status, _, body in replies if status == 200)
+        check("lb.round_robin_both_origins", counts == {"a": 4, "b": 4}, counts=dict(counts))
+
+        a.fail_get = True
+        replies = [http(rust.reverse_port, "/retry-status") for _ in range(6)]
+        check("lb.get_503_failover", all(status == 200 and body == b"b" for status, _, body in replies)
+              and a.counts[("GET", "/retry-status")] > 0,
+              failed_origin_attempts=a.counts[("GET", "/retry-status")])
+        views = backend_views(rust)
+        check("lb.retry_observations_recorded", sum(view.get("retry_attempts_total", 0) for view in views.values()) > 0,
+              backends=views)
+        a.fail_get = False
+        payload = b"one POST must be attempted once"
+        replies = [http(rust.reverse_port, "/post-no-replay", "POST", body=payload) for _ in range(4)]
+        attempts = sum(origin.counts[("POST", "/post-no-replay")] for origin in [a, b])
+        check("lb.post_503_never_replayed", all(status == 503 for status, _, _ in replies) and attempts == 4,
+              client_requests=4, origin_attempts=attempts)
+        check("lb.post_body_preserved", len(a.post_hashes + b.post_hashes) == 4 and
+              all(digest == hashlib.sha256(payload).hexdigest() for digest in a.post_hashes + b.post_hashes))
+
+        for mode in ["drain", "disable"]:
+            vhost["backends"][0].update({"enabled": True, "drain": False})
+            rust.apply(config, f"lb-{mode}-before")
+            a.release.clear()
+            connection = HTTPConnection("127.0.0.1", rust.reverse_port, timeout=10)
+            try:
+                connection.request("GET", "/hold")
+                response = connection.getresponse()
+                first = response.read(5)
+                before = backend_views(rust)
+                check(f"lb.{mode}.lease_spans_response_body", response.status == 200 and first == b"first"
+                      and before.get("a", {}).get("active") == 1, backends=before)
+                vhost["backends"][0].update({"drain": mode == "drain", "enabled": mode != "disable"})
+                rust.apply(config, f"lb-{mode}-active")
+                replies = [http(rust.reverse_port, "/new") for _ in range(4)]
+                check(f"lb.{mode}.new_requests_use_healthy_member",
+                      all(status == 200 and body == b"b" for status, _, body in replies))
+                during = backend_views(rust)
+                check(f"lb.{mode}.active_lease_survives_hot_edit", during.get("a", {}).get("active") == 1,
+                      backends=during)
+                a.release.set()
+                last = response.read()
+                check(f"lb.{mode}.admitted_body_finishes", first + last == b"firstlast")
+                for _ in range(20):
+                    after = backend_views(rust)
+                    if after.get("a", {}).get("active") == 0:
+                        break
+                    time.sleep(.05)
+                check(f"lb.{mode}.lease_released_after_body", after.get("a", {}).get("active") == 0,
+                      backends=after)
+            finally:
+                a.release.set()
+                connection.close()
+
+        vhost["backends"][0].update({"enabled": True, "drain": False})
+        rust.apply(config, "lb-connect-refused")
+        a.close()
+        replies = [http(rust.reverse_port, "/retry-connect") for _ in range(6)]
+        views = backend_views(rust)
+        check("lb.connect_refused_failover", all(status == 200 and body == b"b" for status, _, body in replies)
+              and views.get("a", {}).get("connect_errors_total", 0) > 0, backends=views)
+        rust.reset()
+        check("lb.static_vhost_survives_reload", http(rust.static_port)[0] == 200)
+    finally:
+        a.close()
+        b.close()
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--binary", action="append", required=True, metavar="LABEL=/ABS/PATH")
+    parser.add_argument("--out", type=Path, required=True)
+    args = parser.parse_args()
+    if args.out.exists() and any(args.out.iterdir()):
+        parser.error("--out must be new or empty")
+    reports = {}
+    for specification in args.binary:
+        label, binary = specification.split("=", 1)
+        cases = []
+
+        def check(case, passed, **details):
+            cases.append({"case": case, "passed": passed, **details})
+            print(f"{label} {'PASS' if passed else 'FAIL'} {case}", flush=True)
+            write_json(args.out / label / "checks.json", cases)
+
+        try:
+            with Fixture(args.out / label) as fixture:
+                run(check, fixture, Path(binary), label)
+        except Exception as exc:
+            check("lb.runner", False, error=str(exc), traceback=traceback.format_exc())
+        report = {"passed": sum(case["passed"] for case in cases),
+                  "failed": sum(not case["passed"] for case in cases), "cases": cases,
+                  "simulation": False, "kind": "live RustProxy two-origin HTTP/1.1 regression",
+                  "not_covered": ["sticky affinity", "active health probes", "long-running connection soak",
+                                  "TLS origin failures", "DNS rotation"]}
+        reports[label] = report
+        write_json(args.out / label / "report.json", report)
+    write_json(args.out / "comparison.json", reports)
+    seal_results(args.out)
+    return 1 if any(report["failed"] for report in reports.values()) else 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

scripts/performance/peer.py

@@ -0,0 +1,223 @@
+#!/usr/bin/env python3
+"""A protocol peer, not a RustProxy emulator; used only as the test upstream.
+
+Performance comparisons use nginx as the upstream, never this Python fixture.
+"""
+from __future__ import annotations
+
+import base64
+import collections
+import hashlib
+import json
+import socket
+import struct
+import threading
+import time
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+from urllib.parse import urlsplit, parse_qs
+
+
+class Peer:
+    def __init__(self, port, site):
+        self.site = site
+        self.counts = collections.Counter()
+        self.received = {}
+        self.lock = threading.Lock()
+        self.started_upload = threading.Event()
+        self.release_stream = threading.Event()
+        peer = self
+
+        class Handler(BaseHTTPRequestHandler):
+            protocol_version = "HTTP/1.1"
+
+            def log_message(self, *_):
+                pass
+
+            def emit(self, status, body=b"", headers=None):
+                self.send_response(status)
+                self.send_header("Content-Length", str(len(body)))
+                if "Content-Type" not in (headers or {}):
+                    self.send_header("Content-Type", "application/octet-stream")
+                for key, value in (headers or {}).items():
+                    self.send_header(key, value)
+                self.end_headers()
+                if self.command != "HEAD" and body:
+                    self.wfile.write(body)
+
+            def upload(self):
+                chunks = []
+                if self.headers.get("Transfer-Encoding", "").lower() == "chunked":
+                    while True:
+                        length = int(self.rfile.readline().split(b";", 1)[0], 16)
+                        if length == 0:
+                            while self.rfile.readline().strip():
+                                pass
+                            break
+                        payload = self.rfile.read(length)
+                        self.rfile.read(2)
+                        chunks.append(payload)
+                        peer.started_upload.set()
+                else:
+                    remaining = int(self.headers.get("Content-Length", 0))
+                    while remaining:
+                        payload = self.rfile.read(min(remaining, 1024))
+                        if not payload:
+                            break
+                        chunks.append(payload)
+                        remaining -= len(payload)
+                        peer.started_upload.set()
+                return b"".join(chunks)
+
+            def do_POST(self):
+                self.handle_method()
+
+            def do_PUT(self):
+                self.handle_method()
+
+            def do_HEAD(self):
+                self.handle_method()
+
+            def do_GET(self):
+                self.handle_method()
+
+            def handle_method(self):
+                try:
+                    self.route()
+                except (BrokenPipeError, ConnectionResetError, socket.timeout):
+                    self.close_connection = True
+
+            def route(self):
+                parsed = urlsplit(self.path)
+                path, query = parsed.path, parse_qs(parsed.query)
+                with peer.lock:
+                    peer.counts[(self.command, path)] += 1
+                    count = peer.counts[(self.command, path)]
+                body = self.upload() if self.command in ("POST", "PUT") or self.headers.get("Content-Length") or self.headers.get("Transfer-Encoding") else b""
+                if path == "/no-replay":
+                    self.close_connection = True
+                    self.connection.shutdown(socket.SHUT_RDWR)
+                    self.connection.close()
+                    return
+                if path in ("/echo", "/stream-upload"):
+                    data = {"method": self.command, "path": self.path,
+                            "headers": {key.lower(): value for key, value in self.headers.items()},
+                            "body_bytes": len(body), "body_sha256": hashlib.sha256(body).hexdigest()}
+                    with peer.lock:
+                        peer.received[path] = data
+                    self.emit(200, json.dumps(data).encode(), {"Content-Type": "application/json",
+                              "Cache-Control": "no-store", "X-Origin-Fixture": "yes"})
+                    return
+                if path == "/stream-download":
+                    self.send_response(200)
+                    self.send_header("Transfer-Encoding", "chunked")
+                    self.send_header("Content-Type", "application/octet-stream")
+                    self.send_header("Cache-Control", "no-store")
+                    self.end_headers()
+                    self.wfile.write(b"5\r\nfirst\r\n")
+                    self.wfile.flush()
+                    peer.release_stream.wait(5)
+                    self.wfile.write(b"4\r\nlast\r\n0\r\n\r\n")
+                    self.wfile.flush()
+                    return
+                if path == "/ws":
+                    key = self.headers.get("Sec-WebSocket-Key", "")
+                    accept = base64.b64encode(hashlib.sha1(
+                        (key + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11").encode()).digest()).decode()
+                    self.send_response(101)
+                    self.send_header("Upgrade", "websocket")
+                    self.send_header("Connection", "Upgrade")
+                    self.send_header("Sec-WebSocket-Accept", accept)
+                    self.end_headers()
+                    self.wfile.flush()
+                    frame = self.rfile.read(2)
+                    if len(frame) == 2:
+                        length = frame[1] & 127
+                        if length == 126:
+                            length = struct.unpack("!H", self.rfile.read(2))[0]
+                        mask = self.rfile.read(4) if frame[1] & 128 else b"\0\0\0\0"
+                        data = self.rfile.read(length)
+                        data = bytes(byte ^ mask[i % 4] for i, byte in enumerate(data))
+                        self.wfile.write(bytes([0x81, len(data)]) + data)
+                        self.wfile.flush()
+                    self.close_connection = True
+                    return
+                if path.startswith("/status/"):
+                    self.emit(int(path.rsplit("/", 1)[1]), b"origin status payload\n",
+                              {"Cache-Control": "no-store", "X-Origin-Fixture": "yes"})
+                    return
+                if path == "/redirect":
+                    self.emit(302, b"redirect body\n", {"Location": "/small.html", "Cache-Control": "no-store"})
+                    return
+                if path in ("/private", "/cookie", "/no-store", "/vary", "/fresh", "/revalidate", "/concurrent"):
+                    cache_control = "public, max-age=60"
+                    headers = {"Content-Type": "text/plain"}
+                    if body:
+                        headers["X-Received-Body-Bytes"] = str(len(body))
+                    if path == "/private":
+                        cache_control = "private, max-age=60"
+                    if path == "/no-store":
+                        cache_control = "no-store"
+                    if path == "/cookie":
+                        headers["Set-Cookie"] = "application-session=fixture; Path=/"
+                    if path == "/vary":
+                        headers["Vary"] = "Accept-Language"
+                    if path == "/concurrent":
+                        time.sleep(.08)
+                    if path == "/revalidate":
+                        cache_control = "public, max-age=1, stale-if-error=10"
+                        headers["ETag"] = '"fixture-v1"'
+                        if self.headers.get("If-None-Match") == headers["ETag"]:
+                            self.emit(304, b"", {**headers, "Cache-Control": cache_control})
+                            return
+                    data = f"{path}:count={count}:lang={self.headers.get('Accept-Language', '')}".encode()
+                    if path == "/revalidate":
+                        data = b"stable revalidated object"
+                    self.emit(200, data, {**headers, "Cache-Control": cache_control})
+                    return
+                if path == "/no-transform":
+                    self.emit(200, b"unchanged compression fixture " * 1024,
+                              {"Content-Type": "text/plain", "Cache-Control": "no-transform, public, max-age=60"})
+                    return
+                file = peer.site / path.lstrip("/")
+                if not file.is_file() or ".." in file.parts:
+                    self.emit(404, b"fixture not found")
+                    return
+                payload = file.read_bytes()
+                tag = '"' + hashlib.sha256(payload).hexdigest() + '"'
+                headers = {"Cache-Control": "public, max-age=60", "ETag": tag,
+                           "Accept-Ranges": "bytes", "X-Origin-Fixture": "yes"}
+                headers["Content-Type"] = {".html": "text/html", ".txt": "text/plain",
+                    ".css": "text/css", ".json": "application/json"}.get(file.suffix, "application/octet-stream")
+                if self.headers.get("If-None-Match") == tag:
+                    self.emit(304, b"", headers)
+                    return
+                range_value = self.headers.get("Range")
+                if range_value and self.headers.get("If-Range", tag) == tag:
+                    try:
+                        first, last = range_value.removeprefix("bytes=").split("-", 1)
+                        start = int(first) if first else max(0, len(payload) - int(last))
+                        end = min(int(last), len(payload) - 1) if first and last else len(payload) - 1
+                        if start >= len(payload) or end < start:
+                            self.emit(416, b"", {**headers, "Content-Range": f"bytes */{len(payload)}"})
+                        else:
+                            self.emit(206, payload[start:end + 1], {**headers,
+                                "Content-Range": f"bytes {start}-{end}/{len(payload)}"})
+                        return
+                    except ValueError:
+                        pass
+                self.emit(200, payload, headers)
+
+        self.server = ThreadingHTTPServer(("127.0.0.1", port), Handler)
+        self.server.daemon_threads = True
+        self.thread = threading.Thread(target=self.server.serve_forever, daemon=True)
+        self.thread.start()
+
+    def count(self, method, path):
+        with self.lock:
+            return self.counts[(method, path)]
+
+    def close(self):
+        self.release_stream.set()
+        self.server.shutdown()
+        self.server.server_close()
+        self.thread.join(timeout=2)

scripts/performance/profile_syscalls.py

@@ -0,0 +1,284 @@
+#!/usr/bin/env python3
+"""Profile selected server syscalls separately from throughput measurements.
+
+Runs each RustProxy binary sequentially against a disposable loopback fixture.
+The measured operation is one persistent HTTP/1.1 connection requesting the
+64-KiB static file with cache disabled and Accept-Encoding: identity.
+Every response is validated; syscall counts include background server work.
+strace elapsed/CPU times must not be interpreted as uninstrumented performance.
+"""
+from __future__ import annotations
+
+import argparse
+import hashlib
+from http.client import HTTPConnection
+import json
+import os
+from pathlib import Path
+import platform
+import re
+import shutil
+import signal
+import subprocess
+import sys
+import time
+import traceback
+
+from common import Fixture, seal_results, sha256, write_json
+from resources import descendants
+
+
+SYSCALLS = ("read", "pread64", "write", "writev", "openat", "close", "fcntl", "lseek", "futex")
+REQUEST_PATH = "/large.bin"
+
+
+def thread_tracers(pid: int) -> dict[int, int]:
+    result = {}
+    for task in Path(f"/proc/{pid}/task").iterdir():
+        try:
+            status = (task / "status").read_text()
+        except FileNotFoundError:
+            continue  # A thread may exit between the directory read and status.
+        match = re.search(r"^TracerPid:\s+(\d+)$", status, re.MULTILINE)
+        if match:
+            result[int(task.name)] = int(match.group(1))
+    return result
+
+
+def wait_attached(tracer, server, timeout=10):
+    deadline = time.monotonic() + timeout
+    while time.monotonic() < deadline:
+        if tracer.poll() is not None:
+            raise RuntimeError("strace exited before attachment; see strace diagnostic log (ptrace may be unavailable)")
+        if server.poll() is not None:
+            raise RuntimeError("server exited during strace attachment")
+        traced = thread_tracers(server.pid)
+        if traced and all(traced.values()):
+            return sorted(traced)
+        time.sleep(.01)
+    raise RuntimeError("strace did not attach to every server thread within 10 seconds")
+
+
+def detach(tracer, server):
+    # Do not signal the process group: only the independently started tracer.
+    if tracer.poll() is None:
+        tracer.send_signal(signal.SIGINT)
+        try:
+            tracer.wait(timeout=10)
+        except subprocess.TimeoutExpired:
+            tracer.terminate()
+            try:
+                tracer.wait(timeout=3)
+            except subprocess.TimeoutExpired:
+                tracer.kill()
+                tracer.wait(timeout=3)
+            raise RuntimeError("strace failed to finish on SIGINT; profile is incomplete")
+    if server.poll() is not None:
+        raise RuntimeError("server exited while the tracer was detached")
+    traced = thread_tracers(server.pid)
+    if not traced or any(traced.values()):
+        raise RuntimeError("server still has traced threads after strace exited")
+    return {"signal": "SIGINT to strace PID only", "strace_returncode": tracer.returncode,
+            "server_alive": True, "remaining_traced_threads": 0}
+
+
+def parse_summary(path: Path, requests: int):
+    rows = {name: {"calls": 0, "errors": 0, "calls_per_request": 0.0}
+            for name in SYSCALLS}
+    total = None
+    for line in path.read_text().splitlines():
+        fields = line.split()
+        if len(fields) not in (5, 6) or fields[-1] not in (*SYSCALLS, "total"):
+            continue
+        percent, seconds, usecs, calls = map(float, fields[:4])
+        errors = int(fields[4]) if len(fields) == 6 else 0
+        row = {"calls": int(calls), "errors": errors,
+               "calls_per_request": int(calls) / requests,
+               "strace_system_seconds": seconds, "strace_percent_time": percent,
+               "strace_us_per_call": usecs}
+        if fields[-1] == "total":
+            total = row
+        else:
+            rows[fields[-1]] = row
+    if total is None:
+        raise RuntimeError(f"no complete strace summary found in {path}")
+    if sum(row["calls"] for row in rows.values()) != total["calls"]:
+        raise RuntimeError("parsed syscall counts do not match the raw strace total")
+    return {"selected_syscalls": rows, "selected_total": total}
+
+
+def request_checked(connection, expected_socket, expected_sha, phase, sequence, evidence):
+    if connection.sock is not expected_socket:
+        raise RuntimeError("keepalive connection was replaced before the next request")
+    connection.request("GET", REQUEST_PATH,
+                       headers={"Accept-Encoding": "identity", "Connection": "keep-alive"})
+    response = connection.getresponse()
+    payload = response.read()
+    digest = hashlib.sha256(payload).hexdigest()
+    headers = {key.lower(): value for key, value in response.getheaders()}
+    valid = (response.status == 200 and len(payload) == 65536 and digest == expected_sha
+             and headers.get("content-length") == "65536"
+             and headers.get("content-encoding", "identity") == "identity"
+             and connection.sock is expected_socket and not response.will_close)
+    result = {"phase": phase, "sequence": sequence, "status": response.status,
+              "bytes": len(payload), "sha256": digest,
+              "content_length": headers.get("content-length"),
+              "content_encoding": headers.get("content-encoding"),
+              "connection_reused": connection.sock is expected_socket,
+              "will_close": response.will_close, "passed": valid}
+    evidence.write(json.dumps(result, sort_keys=True) + "\n")
+    evidence.flush()
+    if not valid:
+        raise RuntimeError(f"response verification failed in {phase} request {sequence}: {result}")
+
+
+def profile_variant(fixture, label, binary, args):
+    rust = fixture.rust(binary, label)
+    rust.login()
+    rust.wait_quiescent()
+    raw = args.out / "raw"
+    raw.mkdir(exist_ok=True)
+    summary_path = raw / f"{label}.strace.txt"
+    diagnostic_path = raw / f"{label}.strace.log"
+    responses_path = raw / f"{label}.responses.jsonl"
+    expected_sha = sha256(fixture.site / "large.bin")
+    quiescence_path = fixture.results / f"{label}-quiescence.json"
+    config_path = fixture.results / "configs" / f"{label}-base.json"
+    config_sha = sha256(config_path)
+    if config_sha != rust.applies[-1]["sha256"]:
+        raise RuntimeError("saved fixture configuration differs from the applied configuration")
+    command = [str(args.strace), "-f", "-c", "-e", "trace=" + ",".join(SYSCALLS),
+               "-p", str(rust.process.pid), "-o", str(summary_path)]
+    connection = HTTPConnection("127.0.0.1", rust.static_port, timeout=args.timeout)
+    tracer = None
+    result = {"variant": label, "binary": str(binary), "binary_sha256": sha256(binary),
+              "config": str(config_path.relative_to(args.out)), "config_sha256": config_sha,
+              "quiescence": str(quiescence_path.relative_to(args.out)),
+              "quiescence_sha256": sha256(quiescence_path),
+              "pid": rust.process.pid, "request_path": REQUEST_PATH, "payload_bytes": 65536,
+              "payload_sha256": expected_sha, "cache_enabled": False,
+              "request_encoding": "identity", "concurrency": 1,
+              "warmup_requests": args.warmup, "profile_requests": args.requests,
+              "strace_command": command, "raw_summary": str(summary_path.relative_to(args.out)),
+              "raw_diagnostics": str(diagnostic_path.relative_to(args.out)),
+              "responses": str(responses_path.relative_to(args.out)), "passed": False}
+    try:
+        with responses_path.open("w") as responses, diagnostic_path.open("w") as diagnostics:
+            connection.connect()
+            expected_socket = connection.sock
+            result["client_local_address"] = expected_socket.getsockname()
+            for index in range(args.warmup):
+                request_checked(connection, expected_socket, expected_sha, "warmup", index + 1, responses)
+            result["processes_before_attach"] = descendants(rust.process.pid)
+            if result["processes_before_attach"] != [rust.process.pid]:
+                raise RuntimeError("server spawned a child after quiescence; refusing contaminated syscall trace")
+            environment = os.environ.copy()
+            environment["LC_ALL"] = "C"
+            tracer = subprocess.Popen(command, stdout=diagnostics, stderr=diagnostics,
+                                      env=environment, start_new_session=True)
+            result["initial_traced_threads"] = wait_attached(tracer, rust.process)
+            started = time.monotonic()
+            try:
+                for index in range(args.requests):
+                    request_checked(connection, expected_socket, expected_sha, "profile", index + 1, responses)
+                    if tracer.poll() is not None:
+                        raise RuntimeError("strace exited before all profile requests completed")
+            finally:
+                result["instrumented_request_loop_seconds"] = time.monotonic() - started
+                result["detach"] = detach(tracer, rust.process)
+                tracer = None
+            # An actual HTTP response after detaching proves the server stayed usable.
+            request_checked(connection, expected_socket, expected_sha, "after_detach", 1, responses)
+            result["processes_after_detach"] = descendants(rust.process.pid)
+            if result["processes_after_detach"] != [rust.process.pid]:
+                raise RuntimeError("server has a child after profiling; syscall trace needs review")
+        result.update(parse_summary(summary_path, args.requests))
+        result["verified_responses"] = args.requests + args.warmup + 1
+        result["passed"] = True
+        return result
+    finally:
+        if tracer is not None:
+            detach(tracer, rust.process)
+        connection.close()
+        fixture.stop(rust.process)
+        write_json(args.out / f"{label}.json", result)
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--binary", action="append", required=True, metavar="LABEL=/ABS/PATH")
+    parser.add_argument("--nginx", type=Path, required=True, help="nginx executable for the shared fixture origin")
+    parser.add_argument("--strace", type=Path, default=shutil.which("strace"))
+    parser.add_argument("--out", type=Path, required=True)
+    parser.add_argument("--requests", type=int, default=500)
+    parser.add_argument("--warmup", type=int, default=20)
+    parser.add_argument("--workers", type=int, default=4)
+    parser.add_argument("--origin-workers", type=int, default=4)
+    parser.add_argument("--server-cpus")
+    parser.add_argument("--origin-cpus")
+    parser.add_argument("--timeout", type=float, default=20)
+    args = parser.parse_args()
+    if args.requests < 1 or args.warmup < 0 or args.workers < 1 or args.origin_workers < 1 or args.timeout <= 0:
+        parser.error("requests/workers/timeout must be positive, warmup must be nonnegative")
+    if args.strace is None:
+        parser.error("strace is required; pass --strace /absolute/path")
+    if args.out.exists() and (not args.out.is_dir() or any(args.out.iterdir())):
+        parser.error("--out must be a new or empty directory; existing evidence is never overwritten")
+    variants = {}
+    for value in args.binary:
+        label, separator, filename = value.partition("=")
+        if not separator or not re.fullmatch(r"[A-Za-z][A-Za-z0-9_-]{0,48}", label) or label in variants:
+            parser.error("each --binary must use a unique filesystem-safe LABEL=/ABS/PATH")
+        binary = Path(filename).resolve()
+        if not binary.is_file():
+            parser.error(f"binary does not exist: {binary}")
+        variants[label] = binary
+    args.out, args.nginx, args.strace = args.out.resolve(), args.nginx.resolve(), args.strace.resolve()
+    for executable in (args.nginx, args.strace):
+        if not executable.is_file() or not os.access(executable, os.X_OK):
+            parser.error(f"executable is missing or is not executable: {executable}")
+    args.out.mkdir(parents=True, exist_ok=True)
+    version = subprocess.run([str(args.strace), "-V"], capture_output=True, text=True, timeout=10)
+    environment = {"kind": "selected syscall counts with ptrace instrumentation", "simulation": False,
+        "platform": platform.platform(), "python": sys.version, "strace_version": version.stdout + version.stderr,
+        "strace_sha256": sha256(args.strace), "nginx_sha256": sha256(args.nginx),
+        "variant_order": list(variants), "selected_syscalls": SYSCALLS,
+        "started_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
+        "settings": {key: str(value) if isinstance(value, Path) else value for key, value in vars(args).items()},
+        "limitations": ["strace changes scheduling and latency; do not use instrumented times as RPS evidence",
+                        "Counts cover selected syscalls in every traced server thread, including background work",
+                        "Normalized counts divide the whole trace by verified profile requests; they do not isolate per-request causality",
+                        "This is a warm-file-cache, c1, HTTP/1.1 plaintext static-file scenario; no general capacity inference",
+                        "Syscall counts do not directly count Tokio task dispatches or memory allocations"]}
+    for path in ("/proc/sys/kernel/yama/ptrace_scope", "/sys/fs/cgroup/cpu.max", "/sys/fs/cgroup/memory.max"):
+        try:
+            environment[path] = Path(path).read_text().strip()
+        except OSError:
+            environment[path] = None
+    write_json(args.out / "environment.json", environment)
+    rows, error = [], None
+    try:
+        with Fixture(args.out / "fixture", workers=args.workers, server_cpus=args.server_cpus,
+                     origin_cpus=args.origin_cpus, origin_workers=args.origin_workers) as fixture:
+            fixture.nginx(args.nginx, frontend=False)
+            for label, binary in variants.items():
+                result = profile_variant(fixture, label, binary, args)
+                rows.append(result)
+                write_json(args.out / "profiles.json", rows)
+                print(json.dumps({"variant": label, "requests": args.requests, "passed": result["passed"],
+                                  "selected_syscall_calls": result["selected_total"]["calls"]}), flush=True)
+    except Exception as exc:
+        error = {"error": str(exc), "traceback": traceback.format_exc()}
+        write_json(args.out / "failure.json", error)
+        print(json.dumps(error), flush=True)
+    finally:
+        write_json(args.out / "profiles.json", rows)
+        write_json(args.out / "completion.json", {"completed": error is None, "profiles": len(rows),
+            "simulation": False, "throughput_measurement": False,
+            "finished_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())})
+        seal_results(args.out)
+    return 1 if error else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/performance/regression.py

@@ -0,0 +1,107 @@
+#!/usr/bin/env python3
+"""Run black-box acceptance tests against one or more real RustProxy binaries.
+
+Example:
+  python3 scripts/performance/regression.py --binary before=/abs/before \
+    --binary after=/abs/after --out /abs/results/regression
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+from pathlib import Path
+import platform
+import sys
+import time
+import traceback
+
+from common import Fixture, seal_results, write_json
+from peer import Peer
+import checks_cache
+import checks_policy
+import checks_proxy
+import checks_static
+
+
+GROUPS = {"static": checks_static.run, "proxy": checks_proxy.run,
+          "policy": checks_policy.run, "cache": checks_cache.run,
+          "transform": checks_cache.transforms}
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--binary", action="append", required=True, metavar="LABEL=/ABS/PATH")
+    parser.add_argument("--out", required=True, type=Path)
+    parser.add_argument("--workers", type=int, default=4)
+    parser.add_argument("--groups", default=",".join(GROUPS))
+    args = parser.parse_args()
+    groups = args.groups.split(",")
+    if any(group not in GROUPS for group in groups):
+        parser.error(f"groups must be selected from {','.join(GROUPS)}")
+    args.out = args.out.resolve()
+    args.out.mkdir(parents=True, exist_ok=True)
+    metadata = {"kind": "live RustProxy black-box HTTP acceptance", "simulation": False,
+                "python": sys.version, "platform": platform.platform(),
+                "cpu_affinity": sorted(os.sched_getaffinity(0)) if hasattr(os, "sched_getaffinity") else None,
+                "started_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
+                "groups": groups,
+                "not_covered": ["TLS/ALPN", "HTTP/2", "HTTP/3", "PHP-FPM",
+                                "Tinyauth verifier", "real DNS rebinding", "24-hour soak",
+                                "concurrent filesystem rename-race proof"]}
+    write_json(args.out / "environment.json", metadata)
+    reports = {}
+    for binary_arg in args.binary:
+        label, separator, name = binary_arg.partition("=")
+        if not separator or not label.replace("-", "").replace("_", "").isalnum():
+            parser.error("--binary must be LABEL=/ABS/PATH with a simple unique label")
+        if label in reports:
+            parser.error("binary labels must be unique")
+        cases = []
+        started = time.monotonic()
+        output = args.out / label
+
+        def check(case, passed, **details):
+            row = {"case": case, "passed": passed, **details}
+            cases.append(row)
+            print(f"{label} {'SKIP' if passed is None else 'PASS' if passed else 'FAIL'} {case}", flush=True)
+            write_json(output / "checks.json", cases)
+
+        try:
+            with Fixture(output, workers=args.workers) as fixture:
+                peer = Peer(fixture.origin_port, fixture.site)
+                try:
+                    rust = fixture.rust(Path(name), label)
+                    check("runtime.started", True)
+                    for group in groups:
+                        try:
+                            rust.reset()
+                            GROUPS[group](check, rust, peer)
+                        except Exception as exc:
+                            check(group + ".runner", False, error=str(exc), traceback=traceback.format_exc())
+                        finally:
+                            rust.reset()
+                    check("runtime.still_alive", rust.process.poll() is None)
+                finally:
+                    peer.close()
+        except Exception as exc:
+            check("runtime.runner", False, error=str(exc), traceback=traceback.format_exc())
+        report = {"label": label, "kind": metadata["kind"], "simulation": False,
+                  "seconds": time.monotonic() - started, "passed": sum(c["passed"] is True for c in cases),
+                  "failed": sum(c["passed"] is False for c in cases),
+                  "skipped": sum(c["passed"] is None for c in cases), "cases": cases}
+        reports[label] = report
+        write_json(output / "report.json", report)
+        print(f"{label}: {report['passed']} passed, {report['failed']} failed, {report['skipped']} skipped", flush=True)
+    comparison = {"variants": {name: {key: report[key] for key in ["passed", "failed", "skipped", "seconds"]}
+                               for name, report in reports.items()},
+                  "cases": [{"case": case, **{name: next((entry["passed"] for entry in report["cases"]
+                              if entry["case"] == case), "not_run") for name, report in reports.items()}}
+                            for case in sorted({entry["case"] for report in reports.values() for entry in report["cases"]})]}
+    write_json(args.out / "comparison.json", comparison)
+    seal_results(args.out)
+    return 1 if any(report["failed"] for report in reports.values()) else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/performance/resources.py

@@ -0,0 +1,228 @@
+"""Linux /proc sampling outside the server process; no synthetic resource data."""
+from __future__ import annotations
+
+import os
+from pathlib import Path
+import threading
+import time
+
+
+def cgroup_cpu_stat(path=Path("/sys/fs/cgroup/cpu.stat")):
+    values = {key: int(value) for key, value in (line.split() for line in path.read_text().splitlines())}
+    if not {"nr_throttled", "throttled_usec"} <= values.keys():
+        raise RuntimeError(f"{path} lacks cgroup v2 throttling counters")
+    return {"path": str(path), "observed_monotonic": time.monotonic(), "values": values}
+
+
+def cgroup_cpu_delta(before, after):
+    delta = {key: after["values"][key] - value for key, value in before["values"].items()
+             if key in after["values"]}
+    if any(value < 0 for value in delta.values()):
+        raise RuntimeError("cgroup CPU counters reset during the measured run")
+    return delta
+
+
+def parse_cpu_set(value):
+    if not value:
+        raise ValueError("an explicit, nonempty CPU set is required")
+    result = set()
+    for part in value.split(","):
+        bounds = part.strip().split("-")
+        if len(bounds) not in (1, 2) or any(not item.isascii() or not item.isdigit() for item in bounds):
+            raise ValueError(f"invalid CPU set {value!r}; use comma-separated numbers or ranges")
+        start, end = int(bounds[0]), int(bounds[-1])
+        if start > end or end - start > 65536:
+            raise ValueError(f"invalid CPU range {part!r}")
+        result.update(range(start, end + 1))
+    return result
+
+
+def validate_cpu_sets(values, allowed):
+    groups = {label: parse_cpu_set(value) for label, value in values.items()}
+    for label, cpus in groups.items():
+        if not cpus <= allowed:
+            raise ValueError(f"{label} CPUs {sorted(cpus - allowed)} are outside the allowed affinity")
+        for other, other_cpus in groups.items():
+            if label < other and cpus & other_cpus:
+                raise ValueError(f"{label} and {other} CPU sets overlap: {sorted(cpus & other_cpus)}")
+    return groups
+
+
+def affinity_snapshot(root, expected, expected_processes=None):
+    """Verify every currently visible thread in the root process and children."""
+    tasks = []
+    for pid in descendants(root):
+        try:
+            threads = list(Path(f"/proc/{pid}/task").iterdir())
+        except FileNotFoundError:
+            continue
+        for thread in threads:
+            try:
+                actual = set(os.sched_getaffinity(int(thread.name)))
+            except ProcessLookupError:
+                continue
+            if actual != expected:
+                raise RuntimeError(f"affinity mismatch for PID {pid} TID {thread.name}: "
+                                   f"expected {sorted(expected)}, got {sorted(actual)}")
+            tasks.append({"pid": pid, "tid": int(thread.name), "cpus": sorted(actual)})
+    if not tasks or not any(task["pid"] == root for task in tasks):
+        raise RuntimeError(f"cannot inspect affinity of live process {root}")
+    processes = len({task["pid"] for task in tasks})
+    if expected_processes is not None and processes != expected_processes:
+        raise RuntimeError(f"process count mismatch for PID {root}: "
+                           f"expected {expected_processes}, inspected {processes}")
+    return {"root_pid": root, "processes": processes,
+            "threads": len(tasks), "tasks": tasks}
+
+
+def wait_for_affinity(process, expected, timeout=2):
+    """Allow taskset to execute before inspecting the generator's inherited mask."""
+    deadline = time.monotonic() + timeout
+    while True:
+        if process.poll() is not None:
+            raise RuntimeError(f"generator exited before affinity verification: {process.returncode}")
+        try:
+            return affinity_snapshot(process.pid, expected)
+        except RuntimeError:
+            if time.monotonic() >= deadline:
+                raise
+            time.sleep(.01)
+
+
+def descendants(pid, proc_root=Path("/proc")):
+    """Find the process tree through stat PPids, including nginx workers.
+
+    /proc/PID/task/PID/children is optional on some Linux kernels and may also
+    omit children created by another thread. Missing that file is not evidence
+    that a live process is a leaf. A single PPid-map scan avoids either case.
+    """
+    children, seen = {}, set()
+    for entry in proc_root.iterdir():
+        if not entry.name.isdecimal():
+            continue
+        current = int(entry.name)
+        try:
+            fields = (entry / "stat").read_text().rsplit(")", 1)[1].split()
+            parent = int(fields[1])
+        except (FileNotFoundError, ProcessLookupError):
+            # A non-root process may finish while the snapshot is assembled.
+            continue
+        except (OSError, ValueError, IndexError) as error:
+            raise RuntimeError(f"cannot discover process tree from PID {current}: {error}") from error
+        if fields[0] in {"Z", "X", "x"}:
+            # A dead/zombie worker cannot satisfy the configured live count.
+            continue
+        seen.add(current)
+        children.setdefault(parent, []).append(current)
+    if pid not in seen:
+        raise RuntimeError(f"no readable live root process for PID {pid}")
+    result, pending = [], [pid]
+    while pending:
+        current = pending.pop()
+        if current in result:
+            continue
+        result.append(current)
+        pending.extend(sorted(children.get(current, []), reverse=True))
+    return result
+
+
+def process_stats(pid):
+    try:
+        data = Path(f"/proc/{pid}/stat").read_text().rsplit(")", 1)[1].split()
+        if data[0] in {"Z", "X", "x"}:
+            return None
+        ticks = os.sysconf("SC_CLK_TCK")
+        pages = os.sysconf("SC_PAGE_SIZE")
+        return {"pid": pid, "start_ticks": int(data[19]),
+                "cpu_seconds": (int(data[11]) + int(data[12])) / ticks,
+                "rss_bytes": int(data[21]) * pages,
+                "fd": len(list(Path(f"/proc/{pid}/fd").iterdir())),
+                "threads": int(data[17])}
+    except (FileNotFoundError, ProcessLookupError):
+        return None
+    except (OSError, ValueError, IndexError) as error:
+        raise RuntimeError(f"cannot read resource counters for PID {pid}: {error}") from error
+
+
+class Monitor:
+    def __init__(self, roots, interval=.5, expected_processes=None):
+        self.roots = roots
+        self.expected_processes = expected_processes or {}
+        self.process_identities = {}
+        self.interval = interval
+        if interval <= 0:
+            raise ValueError("resource interval must be positive")
+        self.samples = []
+        self.errors = []
+        self.stop_event = threading.Event()
+        self.started = time.monotonic()
+        self.thread = threading.Thread(target=self.run, daemon=True)
+
+    def sample(self):
+        groups = {}
+        for label, root in self.roots.items():
+            processes = [stats for pid in descendants(root) if (stats := process_stats(pid)) is not None]
+            if not any(process["pid"] == root for process in processes):
+                raise RuntimeError(f"no readable live root process for {label} (PID {root})")
+            expected = self.expected_processes.get(label)
+            if expected is not None and len(processes) != expected:
+                raise RuntimeError(f"resource process count mismatch for {label} (PID {root}): "
+                                   f"expected {expected}, read {len(processes)}")
+            identities = {(process["pid"], process["start_ticks"]) for process in processes}
+            if expected is not None:
+                original = self.process_identities.setdefault(label, identities)
+                if identities != original:
+                    raise RuntimeError(f"process identities changed while measuring {label}: "
+                                       f"expected {sorted(original)}, read {sorted(identities)}")
+            groups[label] = {"processes": processes,
+                             "rss_bytes": sum(p["rss_bytes"] for p in processes),
+                             "fd": sum(p["fd"] for p in processes),
+                             "threads": sum(p["threads"] for p in processes)}
+        self.samples.append({"seconds": time.monotonic() - self.started, "groups": groups})
+
+    def run(self):
+        while not self.stop_event.wait(self.interval):
+            try:
+                self.sample()
+            except Exception as error:
+                self.errors.append(str(error))
+                self.stop_event.set()
+
+    def start(self):
+        self.sample()
+        self.thread.start()
+
+    def stop(self):
+        self.stop_event.set()
+        self.thread.join(timeout=3)
+        if self.thread.is_alive():
+            self.errors.append("resource sampler did not stop")
+        try:
+            self.sample()
+        except Exception as error:
+            self.errors.append(str(error))
+
+    def summary(self, requests):
+        if self.errors:
+            raise RuntimeError("resource sampling failed: " + "; ".join(self.errors))
+        result = {}
+        for label in self.roots:
+            observations = [sample["groups"][label] for sample in self.samples]
+            totals = {}
+            for group in observations:
+                for process in group["processes"]:
+                    key = (process["pid"], process["start_ticks"])
+                    low, high = totals.get(key, (process["cpu_seconds"], process["cpu_seconds"]))
+                    totals[key] = (min(low, process["cpu_seconds"]), max(high, process["cpu_seconds"]))
+            if not totals:
+                raise RuntimeError(f"no readable process samples for {label}; refusing zero resource metrics")
+            seconds = sum(high - low for low, high in totals.values())
+            result[label] = {"cpu_seconds": seconds,
+                             "cpu_us_per_request": seconds * 1_000_000 / requests if requests else None,
+                             "rss_peak_bytes": max(group["rss_bytes"] for group in observations),
+                             "rss_start_bytes": observations[0]["rss_bytes"],
+                             "rss_end_bytes": observations[-1]["rss_bytes"],
+                             "fd_peak": max(group["fd"] for group in observations),
+                             "fd_start": observations[0]["fd"], "fd_end": observations[-1]["fd"],
+                             "processes_seen": len(totals)}
+        return result

scripts/performance/soak.py

@@ -0,0 +1,325 @@
+#!/usr/bin/env python3
+"""Fixed-rate mixed traffic, controlled client closes and cache hot-toggle soak.
+
+The default is a 30-minute live run. Short invocations must be explicitly marked
+--exploratory and cannot satisfy the release soak gate. No run is auto-scheduled.
+"""
+from __future__ import annotations
+
+import argparse
+from collections import Counter
+from concurrent.futures import ThreadPoolExecutor, wait, FIRST_COMPLETED
+import copy
+import hashlib
+from http.client import HTTPConnection
+import json
+import os
+from pathlib import Path
+import platform
+import statistics
+import sys
+import time
+import traceback
+
+from benchmark import counter_values
+from common import Fixture, http, seal_results, sha256, write_json
+from resources import Monitor
+from telemetry import traffic_analytics_snapshot, traffic_analytics_delta
+
+
+PAYLOADS = ["small.html", "large.bin", "huge.bin"]
+
+
+def fetch(index, rust, expected, abort_every):
+    name = PAYLOADS[index % len(PAYLOADS)]
+    source = "proxy" if (index // len(PAYLOADS)) % 2 else "static"
+    port = rust.reverse_port if source == "proxy" else rust.static_port
+    started = time.monotonic()
+    case = {"index": index, "source": source, "file": name, "method": "GET", "intentional_abort": False}
+    try:
+        if abort_every and index % abort_every == 0:
+            case.update({"file": "huge.bin", "intentional_abort": True})
+            connection = HTTPConnection("127.0.0.1", port, timeout=10)
+            try:
+                connection.request("GET", "/huge.bin", headers={"Accept-Encoding": "identity"})
+                response = connection.getresponse()
+                body = response.read(128)
+                case.update({"status": response.status, "bytes": len(body),
+                             "cache": response.getheader("X-Cache"),
+                             "passed": response.status == 200 and body == expected["huge.bin"][:128]})
+            finally:
+                connection.close()
+        else:
+            headers = {"Accept-Encoding": "identity"}
+            method = "HEAD" if index % 29 == 0 else "GET"
+            if method == "GET" and index % 37 == 0:
+                headers["Range"] = "bytes=0-127"
+                expected_status, expected_body = 206, expected[name][:128]
+            else:
+                expected_status, expected_body = 200, b"" if method == "HEAD" else expected[name]
+            status, response_headers, body = http(port, "/" + name, method, headers)
+            case.update({"method": method, "status": status, "bytes": len(body),
+                         "cache": response_headers.get("x-cache"),
+                         "passed": status == expected_status and body == expected_body})
+            if not case["passed"]:
+                case.update({"expected_status": expected_status, "expected_bytes": len(expected_body),
+                             "body_sha256": hashlib.sha256(body).hexdigest()})
+    except Exception as exc:
+        case.update({"passed": False, "error": str(exc)})
+    case["seconds"] = time.monotonic() - started
+    return case
+
+
+def probe_cache_mode(rust, expected, enabled, phase, out, observations):
+    """Record two exact GETs per source, separate from the paced workload."""
+    phase_observations = []
+    for source, port in [("proxy", rust.reverse_port), ("static", rust.static_port)]:
+        replies = []
+        for attempt in [1, 2]:
+            try:
+                status, headers, body = http(port, "/small.html?soak_cache_probe=1",
+                                             headers={"Accept-Encoding": "identity"})
+                replies.append({"attempt": attempt, "status": status, "cache": headers.get("x-cache"),
+                                "bytes": len(body), "sha256": hashlib.sha256(body).hexdigest(),
+                                "body_passed": status == 200 and body == expected["small.html"]})
+            except Exception as error:
+                replies.append({"attempt": attempt, "body_passed": False, "error": str(error)})
+        mode_passed = replies[1].get("cache") == "HIT" if enabled else all(
+            reply.get("cache") is None for reply in replies)
+        phase_observations.append({"phase": phase, "source": source, "cache_enabled": enabled,
+                                   "responses": replies,
+                                   "passed": all(reply["body_passed"] for reply in replies) and mode_passed})
+    observations.extend(phase_observations)
+    write_json(out / "cache-probes.json", observations)
+    if not all(observation["passed"] for observation in phase_observations):
+        raise RuntimeError(f"cache mode/body probe failed after {phase}; see cache-probes.json")
+
+
+def backend_drain_snapshot(rust):
+    status, _, body = rust.admin("/api/status")
+    if status != 200:
+        return {"passed": False, "error": f"admin status HTTP {status}"}
+    backends = []
+
+    def walk(value):
+        if isinstance(value, dict):
+            if "id" in value and "active" in value and "status" in value and "requests" in value:
+                backends.append({key: value[key] for key in ("id", "active", "status", "requests")})
+            for child in value.values():
+                walk(child)
+        elif isinstance(value, list):
+            for child in value:
+                walk(child)
+    walk(json.loads(body))
+    return {"passed": bool(backends) and all(backend["active"] == 0 for backend in backends),
+            "backends": backends}
+
+
+def memory_trend(samples, label="server"):
+    observations = [(sample["seconds"], sample["groups"][label]["rss_bytes"], sample["groups"][label]["fd"])
+                    for sample in samples if sample["seconds"] >= 60]
+    if len(observations) < 3:
+        return {"available": False, "reason": "less than three samples after the first minute"}
+    tail_seconds = observations[-1][0]
+    early = [row[1] for row in observations if row[0] < observations[0][0] + 60]
+    late = [row[1] for row in observations if row[0] >= tail_seconds - 60]
+    x_mean = statistics.mean(row[0] for row in observations)
+    y_mean = statistics.mean(row[1] for row in observations)
+    denominator = sum((row[0] - x_mean) ** 2 for row in observations)
+    slope = sum((row[0] - x_mean) * (row[1] - y_mean) for row in observations) / denominator if denominator else 0
+    return {"available": True, "rss_first_minute_after_warmup_median": statistics.median(early),
+            "rss_last_minute_median": statistics.median(late),
+            "rss_growth_bytes": statistics.median(late) - statistics.median(early),
+            "rss_linear_bytes_per_hour": slope * 3600, "fd_first": observations[0][2],
+            "fd_last": observations[-1][2], "fd_peak": max(row[2] for row in observations),
+            "interpretation": "Descriptive only: cache toggles and allocator retention can change RSS without a leak"}
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--binary", type=Path, required=True)
+    parser.add_argument("--label", default="after")
+    parser.add_argument("--nginx", type=Path, required=True)
+    parser.add_argument("--out", type=Path, required=True)
+    parser.add_argument("--duration", type=int, default=1800)
+    parser.add_argument("--rate", type=float, default=100, help="offered requests per second, not a saturation benchmark")
+    parser.add_argument("--clients", type=int, default=8)
+    parser.add_argument("--workers", type=int, default=4)
+    parser.add_argument("--origin-workers", type=int, default=2)
+    parser.add_argument("--server-cpus")
+    parser.add_argument("--origin-cpus")
+    parser.add_argument("--toggle-seconds", type=int, default=120)
+    parser.add_argument("--sample-seconds", type=int, default=5)
+    parser.add_argument("--abort-every", type=int, default=101)
+    parser.add_argument("--cooldown-seconds", type=int, default=15)
+    parser.add_argument("--exploratory", action="store_true")
+    args = parser.parse_args()
+    if args.duration < 1800 and not args.exploratory:
+        parser.error("the soak gate requires >=1800 seconds; label short diagnostics with --exploratory")
+    if args.rate <= 0 or not 1 <= args.clients <= 128 or args.toggle_seconds < 1 or args.sample_seconds < 1:
+        parser.error("rate, sample/toggle intervals and clients must be positive; clients <=128")
+    args.out = args.out.resolve()
+    if args.out.exists() and any(args.out.iterdir()):
+        parser.error("--out must be a new or empty evidence directory")
+    args.out.mkdir(parents=True, exist_ok=True)
+    settings = {key: str(value) if isinstance(value, Path) else value for key, value in vars(args).items()}
+    write_json(args.out / "environment.json", {
+        "kind": "actual RustProxy fixed-rate mixed-traffic resource soak", "simulation": False,
+        "settings": settings, "platform": platform.platform(), "binary_sha256": sha256(args.binary),
+        "started_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
+        "frontend_connection_policy": "A new HTTP/1.1 TCP connection for each workload request and probe",
+        "controlled_close_interpretation": "Client closes after reading 128 bytes; the server may already have sent the full body. No server-cancellation count is inferred.",
+        "traffic_analytics_observation_policy": "Global counters are read before traffic, periodically with status diagnostics, and after final drain. Deltas include cache-mode probes and cooldown, and are separate from HTTP errors.",
+        "not_covered": ["24-hour soak", "TLS/H2/H3 soak", "multi-backend failures", "PHP-FPM",
+                        "real application write load", "origin DNS/TLS rotation", "production traffic",
+                        "long-lived frontend keepalive connections"]})
+    totals = {"submitted": 0, "completed": 0, "passed": 0, "failed": 0, "controlled_client_closes": 0,
+              "response_bytes": 0, "scheduler_late_events": 0, "peak_pending": 0}
+    failure_examples, progress, toggles, counter_samples = [], [], [], []
+    traffic_samples, traffic_delta = [], None
+    cache_probes, cache_status_counts = [], {"proxy": Counter(), "static": Counter()}
+    drain_snapshot, configuration_restored = None, False
+    samples, resources = [], {}
+    fatal, runtime_seconds = None, 0
+    try:
+        with Fixture(args.out / "fixture", workers=args.workers, server_cpus=args.server_cpus,
+                     origin_cpus=args.origin_cpus, origin_workers=args.origin_workers) as fixture:
+            nginx = fixture.nginx(args.nginx, frontend=False)
+            rust = fixture.rust(args.binary, args.label)
+            rust.login()
+            rust.wait_quiescent()
+            expected = {name: (fixture.site / name).read_bytes() for name in PAYLOADS}
+            config = copy.deepcopy(rust.base)
+            config["cache"].update({"expose_status": True})
+            config["web_services"][0]["cache_control"] = "public, max-age=60"
+            rust.apply(config, "soak-initial")
+            probe_cache_mode(rust, expected, False, "initial-off", args.out, cache_probes)
+            traffic_before = traffic_analytics_snapshot(rust)
+            traffic_samples.append({"phase": "before", "seconds": 0, "snapshot": traffic_before})
+            write_json(args.out / "traffic-analytics.json", traffic_samples)
+            monitor = Monitor({"server": rust.process.pid, "origin": nginx["origin"].pid}, args.sample_seconds)
+            monitor.start()
+            start = time.monotonic()
+            deadline = start + args.duration
+            next_toggle, next_snapshot, next_submit = start + args.toggle_seconds, start, start
+            pending = set()
+
+            def consume(done):
+                for future in done:
+                    result = future.result()
+                    totals["completed"] += 1
+                    totals["passed" if result["passed"] else "failed"] += 1
+                    totals["controlled_client_closes"] += int(result["intentional_abort"])
+                    totals["response_bytes"] += result.get("bytes", 0)
+                    cache_status = (result.get("cache") or "absent") if "status" in result else "no_response"
+                    cache_status_counts[result["source"]][cache_status] += 1
+                    if not result["passed"] and len(failure_examples) < 100:
+                        failure_examples.append(result)
+
+            try:
+                with ThreadPoolExecutor(max_workers=args.clients) as pool:
+                    while time.monotonic() < deadline:
+                        now = time.monotonic()
+                        done = {future for future in pending if future.done()}
+                        consume(done)
+                        pending -= done
+                        if now >= next_toggle:
+                            config["cache"]["enabled"] = not config["cache"]["enabled"]
+                            rust.apply(config, f"soak-toggle-{len(toggles) + 1}")
+                            toggles.append({"seconds": time.monotonic() - start, "cache_enabled": config["cache"]["enabled"]})
+                            write_json(args.out / "cache-toggles.json", toggles)
+                            probe_cache_mode(rust, expected, config["cache"]["enabled"],
+                                             f"toggle-{len(toggles)}", args.out, cache_probes)
+                            next_toggle += args.toggle_seconds
+                        if now >= next_snapshot:
+                            status, headers, body = rust.admin("/api/status")
+                            if status != 200:
+                                raise RuntimeError(f"admin status became unavailable: HTTP {status}")
+                            counter_samples.append({"seconds": time.monotonic() - start,
+                                                    "cache_enabled": config["cache"]["enabled"],
+                                                    "counters": counter_values(json.loads(body))})
+                            traffic_now = traffic_analytics_snapshot(rust)
+                            traffic_delta = traffic_analytics_delta(traffic_before, traffic_now)
+                            traffic_samples.append({"phase": "during", "seconds": time.monotonic() - start,
+                                                    "snapshot": traffic_now, "delta_from_start": traffic_delta})
+                            row = {"seconds": time.monotonic() - start, **totals,
+                                   "cache_enabled": config["cache"]["enabled"], "pending": len(pending),
+                                   "observed_cache_status_counts": {source: dict(counts)
+                                       for source, counts in cache_status_counts.items()}}
+                            progress.append(row)
+                            write_json(args.out / "progress.json", progress)
+                            write_json(args.out / "status-counters.json", counter_samples)
+                            write_json(args.out / "traffic-analytics.json", traffic_samples)
+                            write_json(args.out / "resources.json", list(monitor.samples))
+                            print(json.dumps(row), flush=True)
+                            next_snapshot = time.monotonic() + args.sample_seconds
+                        now = time.monotonic()
+                        if len(pending) >= args.clients * 2:
+                            done, pending = wait(pending, timeout=.1, return_when=FIRST_COMPLETED)
+                            consume(done)
+                            continue
+                        if now < next_submit:
+                            time.sleep(min(.02, next_submit - now))
+                            continue
+                        if now > next_submit + .25:
+                            totals["scheduler_late_events"] += 1
+                            next_submit = now
+                        totals["submitted"] += 1
+                        pending.add(pool.submit(fetch, totals["submitted"], rust, expected, args.abort_every))
+                        totals["peak_pending"] = max(totals["peak_pending"], len(pending))
+                        next_submit += 1 / args.rate
+                    done, pending = wait(pending, timeout=30)
+                    consume(done)
+                    if pending:
+                        raise RuntimeError("requests failed to settle during bounded soak drain")
+                runtime_seconds = time.monotonic() - start
+                rust.reset()
+                configuration_restored = True
+                cooldown_until = time.monotonic() + args.cooldown_seconds
+                while time.monotonic() < cooldown_until:
+                    time.sleep(max(0, min(.5, cooldown_until - time.monotonic())))
+                probe_cache_mode(rust, expected, False, "restored-off", args.out, cache_probes)
+                for _ in range(40):
+                    drain_snapshot = backend_drain_snapshot(rust)
+                    if drain_snapshot["passed"]:
+                        break
+                    time.sleep(.05)
+                write_json(args.out / "backend-drain.json", drain_snapshot)
+                if not drain_snapshot["passed"]:
+                    raise RuntimeError("backend active exchanges did not drain to zero")
+                traffic_after = traffic_analytics_snapshot(rust)
+                traffic_delta = traffic_analytics_delta(traffic_before, traffic_after)
+                traffic_samples.append({"phase": "after_drain", "seconds": time.monotonic() - start,
+                                        "snapshot": traffic_after, "delta_from_start": traffic_delta})
+            finally:
+                monitor.stop()
+                samples = monitor.samples
+                resources = monitor.summary(totals["completed"])
+                write_json(args.out / "resources.json", samples)
+                write_json(args.out / "status-counters.json", counter_samples)
+                write_json(args.out / "traffic-analytics.json", traffic_samples)
+                write_json(args.out / "failure-examples.json", failure_examples)
+    except Exception as exc:
+        fatal = {"error": str(exc), "traceback": traceback.format_exc()}
+        write_json(args.out / "failure.json", fatal)
+    report = {"completed": fatal is None, "simulation": False, "exploratory": args.exploratory,
+              "duration_seconds": runtime_seconds, "soak_duration_gate": runtime_seconds >= 1800,
+              "totals": totals, "resources": resources, "memory_trend": memory_trend(samples) if samples else None,
+              "cache_toggles": len(toggles), "errors": failure_examples,
+              "traffic_analytics": {"observations": len(traffic_samples), "delta": traffic_delta,
+                                    "includes_cache_mode_probes_and_cooldown": True},
+              "cache_mode_probes": {"source_checks": len(cache_probes),
+                                   "requests": sum(len(probe["responses"]) for probe in cache_probes),
+                                   "passed": bool(cache_probes) and all(probe["passed"] for probe in cache_probes)},
+              "observed_cache_status_counts": cache_status_counts,
+              "configuration_restored": configuration_restored, "backend_drain": drain_snapshot,
+              "actual_completed_rps": totals["completed"] / runtime_seconds if runtime_seconds else None,
+              "passed": fatal is None and totals["failed"] == 0 and runtime_seconds >= args.duration}
+    write_json(args.out / "report.json", report)
+    seal_results(args.out)
+    print(json.dumps(report), flush=True)
+    return 0 if report["passed"] else 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/performance/telemetry.py

@@ -0,0 +1,47 @@
+"""Selected counters from existing authenticated APIs, without traffic records."""
+from __future__ import annotations
+
+import json
+import time
+
+
+def traffic_analytics_snapshot(rust, timeout=2):
+    """Read global analytics queue loss; never retain paths, IPs or user agents.
+
+    queue_dropped is process-global even when the API's vhost query is used.
+    recorded_requests is asynchronous analytics work, not an HTTP success count.
+    Call outside a timed load interval so this control-plane read is not charged
+    to the request path being measured.
+    """
+    status, _, body = rust.admin("/api/traffic", timeout=timeout)
+    if status != 200:
+        raise RuntimeError(f"traffic analytics counter capture failed: HTTP {status}")
+    payload = json.loads(body)
+    if not isinstance(payload, dict) or payload.get("scope") != "*" or payload.get("since") != "process_start":
+        raise RuntimeError("traffic analytics requires the global process-start scope")
+    analytics = payload.get("analytics")
+    if not isinstance(analytics, dict) or analytics.get("scope") != "*":
+        raise RuntimeError("traffic analytics response lacks the global queue view")
+    data = analytics.get("data")
+    if not isinstance(data, dict):
+        raise RuntimeError("traffic analytics response lacks the recorded counters")
+    values = {"queue_capacity": analytics.get("queue_capacity"),
+              "queue_dropped": analytics.get("queue_dropped"),
+              "recorded_requests": data.get("requests")}
+    if any(type(value) is not int or value < 0 for value in values.values()) or not values["queue_capacity"]:
+        raise RuntimeError("traffic analytics queue counters are absent or malformed")
+    return {"endpoint": "/api/traffic", "scope": "*", "since": "process_start",
+            "pid": rust.process.pid, "observed_monotonic": time.monotonic(), **values}
+
+
+def traffic_analytics_delta(before, after):
+    """Keep telemetry loss separate from failed HTTP requests and log rotation."""
+    for key in ["pid", "endpoint", "scope", "since", "queue_capacity"]:
+        if before[key] != after[key]:
+            raise RuntimeError(f"traffic analytics identity changed during capture: {key}")
+    result = {"queue_dropped_delta": after["queue_dropped"] - before["queue_dropped"],
+              "recorded_requests_delta": after["recorded_requests"] - before["recorded_requests"],
+              "interval_seconds": after["observed_monotonic"] - before["observed_monotonic"]}
+    if any(value < 0 for value in result.values()):
+        raise RuntimeError("traffic analytics counters or clock reset during capture")
+    return result

scripts/performance/transport_clients.py

@@ -0,0 +1,191 @@
+"""Strict H2/TLS and H3/QUIC clients for local RustProxy acceptance tests.
+
+The generated fixture certificate is explicitly trusted. Neither transport
+silently falls back to HTTP/1.1 or disables certificate verification.
+"""
+from __future__ import annotations
+
+import asyncio
+from contextlib import asynccontextmanager
+from dataclasses import dataclass, field
+from pathlib import Path
+import ssl
+
+import httpx
+from aioquic.asyncio import QuicConnectionProtocol, connect
+from aioquic.h3.connection import H3Connection, MessageError
+from aioquic.h3.events import DataReceived, HeadersReceived
+from aioquic.quic.configuration import QuicConfiguration
+from aioquic.quic.events import ConnectionTerminated, ProtocolNegotiated, StreamReset
+
+
+@dataclass
+class Response:
+    status: int
+    headers: dict[str, str]
+    body: bytes
+    protocol: str
+    alpn: str | None
+    stream_id: int | None = None
+
+
+class H2Client:
+    def __init__(self, client: httpx.AsyncClient, port: int):
+        self.client, self.port = client, port
+
+    async def request(self, method, path, headers=None, body=None):
+        async def chunks():
+            for offset in range(0, len(body), 16384):
+                yield body[offset:offset + 16384]
+                await asyncio.sleep(0)
+
+        response = await self.client.request(
+            method, f"https://127.0.0.1:{self.port}{path}",
+            headers=headers or {}, content=chunks() if body is not None else None,
+        )
+        network = response.extensions.get("network_stream")
+        tls = network.get_extra_info("ssl_object") if network else None
+        alpn = tls.selected_alpn_protocol() if tls else None
+        if response.http_version != "HTTP/2" or alpn != "h2":
+            raise AssertionError(f"H2 required, got {response.http_version}, ALPN={alpn!r}")
+        return Response(response.status_code, dict(response.headers), response.content,
+                        "HTTP/2", alpn, response.extensions.get("stream_id"))
+
+
+@dataclass
+class Pending:
+    future: asyncio.Future
+    status: int = 0
+    headers: dict[str, str] = field(default_factory=dict)
+    body: bytearray = field(default_factory=bytearray)
+
+
+class HeadAwareH3Connection(H3Connection):
+    """Supply request-method context missing from aioquic 1.7.0's length check.
+
+    RFC 9110 section 8.6 permits HEAD's Content-Length to describe a GET body.
+    Still reject any actual HEAD DATA; keep aioquic's exact length check for all
+    other requests. This changes only the test client, never the target server.
+    """
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.head_streams = set()
+
+    def _check_content_length(self, stream):
+        if stream.stream_id in self.head_streams:
+            if stream.content_length:
+                raise MessageError("HEAD response contained DATA")
+            return
+        super()._check_content_length(stream)
+
+
+def validate_head_adapter():
+    """Keep adapter checks separate from the real-server acceptance totals."""
+    from types import SimpleNamespace
+
+    connection = HeadAwareH3Connection.__new__(HeadAwareH3Connection)
+    connection.head_streams = {0}
+    for stream_id, declared, received, valid in [(0, 1040, 0, True), (0, 1040, 1, False),
+                                                  (4, 1040, 1040, True), (4, 1040, 0, False)]:
+        stream = SimpleNamespace(stream_id=stream_id, expected_content_length=declared,
+                                 content_length=received)
+        try:
+            connection._check_content_length(stream)
+            accepted = True
+        except MessageError:
+            accepted = False
+        assert accepted == valid, "H3 fixture HEAD adapter changed ordinary framing validation"
+    return {"cases": 4, "passed": 4, "kind": "test-client self-test, not RustProxy coverage"}
+
+
+class H3Client(QuicConnectionProtocol):
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.http = HeadAwareH3Connection(self._quic)
+        self.pending: dict[int, Pending] = {}
+        self.alpn = None
+        self.authority = None
+
+    def quic_event_received(self, event):
+        if isinstance(event, ProtocolNegotiated):
+            self.alpn = event.alpn_protocol
+        if isinstance(event, ConnectionTerminated):
+            for item in self.pending.values():
+                if not item.future.done():
+                    item.future.set_exception(ConnectionError(
+                        f"QUIC closed code={event.error_code}: {event.reason_phrase}"))
+        if isinstance(event, StreamReset):
+            item = self.pending.get(event.stream_id)
+            if item is not None and not item.future.done():
+                item.future.set_exception(ConnectionError(
+                    f"H3 stream reset code={event.error_code}"))
+        for http_event in self.http.handle_event(event):
+            if not isinstance(http_event, (HeadersReceived, DataReceived)):
+                continue
+            item = self.pending.get(http_event.stream_id)
+            if item is None or item.future.done():
+                continue
+            if isinstance(http_event, HeadersReceived):
+                for name, value in http_event.headers:
+                    if name == b":status":
+                        item.status = int(value)
+                    elif not name.startswith(b":"):
+                        item.headers[name.decode("ascii").lower()] = value.decode("latin-1")
+            else:
+                item.body.extend(http_event.data)
+                if len(item.body) > 4 * 1024 * 1024:
+                    item.future.set_exception(ValueError("fixture response exceeded 4 MiB bound"))
+                    continue
+            if http_event.stream_ended:
+                item.future.set_result(Response(item.status, item.headers, bytes(item.body),
+                                                "HTTP/3", self.alpn, http_event.stream_id))
+
+    async def request(self, method, path, headers=None, body=None):
+        if self.alpn != "h3":
+            raise AssertionError(f"H3 required, got ALPN={self.alpn!r}")
+        stream_id = self._quic.get_next_available_stream_id()
+        item = Pending(asyncio.get_running_loop().create_future())
+        self.pending[stream_id] = item
+        if method == "HEAD":
+            self.http.head_streams.add(stream_id)
+        fields = [(b":method", method.encode("ascii")), (b":scheme", b"https"),
+                  (b":authority", self.authority.encode("ascii")),
+                  (b":path", path.encode("ascii"))]
+        fields += [(name.lower().encode("ascii"), value.encode("latin-1"))
+                   for name, value in (headers or {}).items()]
+        try:
+            self.http.send_headers(stream_id, fields, end_stream=body is None)
+            self.transmit()
+            if body is not None:
+                if not body:
+                    self.http.send_data(stream_id, b"", end_stream=True)
+                    self.transmit()
+                for offset in range(0, len(body), 16384):
+                    end = min(offset + 16384, len(body))
+                    self.http.send_data(stream_id, body[offset:end], end_stream=end == len(body))
+                    self.transmit()
+                    await asyncio.sleep(0)
+            return await asyncio.wait_for(item.future, timeout=10)
+        finally:
+            self.pending.pop(stream_id, None)
+            self.http.head_streams.discard(stream_id)
+
+
+@asynccontextmanager
+async def client(protocol: str, port: int, certificate: Path):
+    if protocol == "h2":
+        context = ssl.create_default_context(cafile=str(certificate))
+        async with httpx.AsyncClient(http2=True, verify=context, trust_env=False,
+                                     timeout=10, limits=httpx.Limits(max_connections=1)) as session:
+            yield H2Client(session, port)
+    elif protocol == "h3":
+        config = QuicConfiguration(is_client=True, alpn_protocols=["h3"],
+                                   server_name="localhost", idle_timeout=10)
+        config.verify_mode = ssl.CERT_REQUIRED
+        config.load_verify_locations(cafile=str(certificate))
+        async with connect("127.0.0.1", port, configuration=config,
+                           create_protocol=H3Client) as session:
+            session.authority = f"127.0.0.1:{port}"
+            yield session
+    else:
+        raise ValueError(f"unsupported fixture protocol {protocol!r}")

scripts/performance/transport_regression.py

@@ -0,0 +1,281 @@
+#!/usr/bin/env python3
+"""Real, bounded RustProxy H2/TLS, H3/QUIC and H1 WebSocket acceptance.
+
+Install Python dependencies aioquic, httpx[http2] and websockets. Example:
+  python3 scripts/performance/transport_regression.py --binary before=/abs/binary \
+    --binary after=/abs/binary --out /abs/results --python-deps /abs/optional/pydeps
+
+Only fresh temporary databases and loopback listeners are used. Certificates
+are generated by RustProxy and verified by clients; no protocol fallback is used.
+This is an interoperability/functional suite, not a throughput benchmark.
+"""
+from __future__ import annotations
+
+import argparse
+import asyncio
+import copy
+import hashlib
+import importlib.metadata
+import json
+from pathlib import Path
+import platform
+import socket
+import sys
+import time
+import traceback
+
+from common import Fixture, seal_results, sha256, write_json
+from peer import Peer
+
+
+def response_details(response):
+    return {"status": response.status, "protocol": response.protocol,
+            "alpn": response.alpn, "stream_id": response.stream_id,
+            "bytes": len(response.body), "sha256": hashlib.sha256(response.body).hexdigest(),
+            "cache": response.headers.get("x-cache"),
+            "cache_reason": response.headers.get("x-cache-reason")}
+
+
+def tls_port(fixture):
+    while True:
+        port = fixture.port()
+        with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as udp:
+            try:
+                udp.bind(("127.0.0.1", port))
+                return port
+            except OSError:
+                continue
+
+
+def transport_config(rust, reverse_tls, static_tls, websocket_port, websocket_origin):
+    config = copy.deepcopy(rust.base)
+    for source, port in [("reverse", reverse_tls), ("static", static_tls)]:
+        identifier = f"{source}-tls"
+        config["listeners"]["endpoints"].append({
+            "id": identifier, "name": identifier, "scope": "internal", "enabled": True,
+            "binding": {"address": f"127.0.0.1:{port}", "tls": True, "http2": True,
+                        "http3": {"max_concurrent_connections": 16,
+                                  "max_concurrent_bidistream": 16,
+                                  "max_concurrent_unistream": 16, "max_idle_timeout": 10}}})
+        vhost = next(v for v in config["vhosts"] if v["id"] == source)
+        vhost["frontend"]["listener_ids"].append(identifier)
+        vhost["http2_enabled"] = True
+        vhost["http3_enabled"] = True
+    config["listeners"]["endpoints"].append({
+        "id": "websocket", "name": "WebSocket", "scope": "internal", "enabled": True,
+        "binding": {"address": f"127.0.0.1:{websocket_port}", "tls": False,
+                    "http2": False, "http3": None}})
+    config["vhosts"].append({
+        "id": "websocket", "frontend": {"hosts": ["127.0.0.1", "localhost"],
+                                        "listener_ids": ["websocket"]},
+        "websockets": True,
+        "backends": [{"id": "echo", "url": f"http://127.0.0.1:{websocket_origin}",
+                      "allow_private": True}],
+        "load_balancing": {"health_check": {"mode": "off"}}, "routes": []})
+    config["cache"].update({"enabled": False, "expose_status": True, "expose_hits": True,
+                            "expose_cache_status": True, "expose_only_hits": False})
+    config["compression"] = {"gzip": False, "br": False, "zstd": False, "deflate": False}
+    config["web_services"][0]["cache_control"] = "public, max-age=60"
+    return config
+
+
+async def http_cases(check, session, protocol, source, cache, fixture):
+    prefix = f"{protocol}.{source}.cache_{'on' if cache else 'off'}"
+    expected = (fixture.site / "small.html").read_bytes()
+    path = f"/transport-{protocol}.html"
+    first = await session.request("GET", path)
+    check(prefix + ".get", first.status == 200 and first.body == expected,
+          **response_details(first))
+    second = await session.request("GET", path)
+    check(prefix + ".cache_delivery", second.status == 200 and second.body == expected
+          and ((first.headers.get("x-cache") == "MISS" and second.headers.get("x-cache") == "HIT")
+               if cache else second.headers.get("x-cache") not in ("MISS", "HIT")),
+          first_cache=first.headers.get("x-cache"), **response_details(second))
+    head = await session.request("HEAD", path)
+    check(prefix + ".head", head.status == 200 and head.body == b""
+          and head.headers.get("content-length") == str(len(expected)), **response_details(head))
+    etag = second.headers.get("etag")
+    check(prefix + ".validator", bool(etag), etag=etag)
+    ranged = await session.request("GET", path, {"range": "bytes=11-40", "if-range": etag or ""})
+    check(prefix + ".range", ranged.status == 206 and ranged.body == expected[11:41]
+          and ranged.headers.get("content-range") == f"bytes 11-40/{len(expected)}",
+          **response_details(ranged))
+    mismatch = await session.request("GET", path, {"range": "bytes=11-40", "if-range": '"different"'})
+    check(prefix + ".if_range_mismatch", mismatch.status == 200 and mismatch.body == expected,
+          **response_details(mismatch))
+    conditional = await session.request("GET", path, {"if-none-match": etag or ""})
+    check(prefix + ".conditional_304", conditional.status == 304 and conditional.body == b"",
+          **response_details(conditional))
+    invalid = await session.request("GET", path, {"range": "bytes=999999-"})
+    check(prefix + ".range_416", invalid.status == 416 and invalid.body == b""
+          and invalid.headers.get("content-range") == f"bytes */{len(expected)}",
+          **response_details(invalid))
+    for name in ["large.bin", "huge.bin"]:
+        expected_file = (fixture.site / name).read_bytes()
+        response = await session.request("GET", "/" + name)
+        check(prefix + ".get." + name, response.status == 200 and response.body == expected_file,
+              **response_details(response))
+    parallel = await asyncio.gather(*(session.request("GET", "/large.bin") for _ in range(4)))
+    expected_large = (fixture.site / "large.bin").read_bytes()
+    check(prefix + ".multiplex_four_streams", all(r.status == 200 and r.body == expected_large for r in parallel)
+          and len({r.stream_id for r in parallel}) == 4,
+          stream_ids=[r.stream_id for r in parallel], response_bytes=[len(r.body) for r in parallel])
+    if source == "reverse":
+        for payload in [b"", (bytes(range(256)) * 512) + b"streamed-end"]:
+            response = await session.request("POST", "/echo?transport=preserved",
+                                             {"content-type": "application/octet-stream"}, payload)
+            data = json.loads(response.body) if response.status == 200 else {}
+            check(prefix + f".post_stream_{len(payload)}", response.status == 200
+                  and data.get("method") == "POST" and data.get("body_bytes") == len(payload)
+                  and data.get("body_sha256") == hashlib.sha256(payload).hexdigest()
+                  and data.get("path") == "/echo?transport=preserved",
+                  upload_bytes=len(payload), **response_details(response))
+    else:
+        response = await session.request("POST", path, body=b"static-method-gate")
+        check(prefix + ".post_static_method_gate", response.status == 405, **response_details(response))
+
+
+async def websocket_cases(check, port, cache):
+    import websockets
+
+    prefix = f"h1.websocket.cache_{'on' if cache else 'off'}"
+    async with websockets.connect(f"ws://127.0.0.1:{port}/echo", open_timeout=5,
+                                   close_timeout=3, ping_interval=None, compression=None) as session:
+        text = "RustProxy echo: Zażółć gęślą jaźń"
+        await session.send(text)
+        returned = await asyncio.wait_for(session.recv(), timeout=5)
+        check(prefix + ".text_echo", returned == text)
+        binary = bytes(range(256)) * 256
+        await session.send(binary)
+        returned = await asyncio.wait_for(session.recv(), timeout=5)
+        check(prefix + ".binary_echo", returned == binary, bytes=len(returned))
+        pong = await session.ping(b"rustproxy-transport")
+        await asyncio.wait_for(pong, timeout=5)
+        check(prefix + ".ping_pong", True)
+        await session.close(code=1000, reason="fixture complete")
+        check(prefix + ".normal_close", session.close_code == 1000, close_code=session.close_code)
+
+
+async def run_variant(binary, label, output, workers):
+    import websockets
+    from transport_clients import client
+
+    cases = []
+    started = time.monotonic()
+
+    def check(case, passed, **details):
+        cases.append({"case": case, "passed": bool(passed), **details})
+        print(f"{label} {'PASS' if passed else 'FAIL'} {case}", flush=True)
+        write_json(output / "checks.json", cases)
+
+    try:
+        with Fixture(output, workers=workers) as fixture:
+            plain = (fixture.site / "small.html").read_bytes()
+            for name in ["transport-h2.html", "transport-h3.html", "transport-shared.html"]:
+                (fixture.site / name).write_bytes(plain)
+            peer = Peer(fixture.origin_port, fixture.site)
+            websocket_origin, websocket_port = fixture.port(), fixture.port()
+
+            async def echo(session, _path=None):
+                async for message in session:
+                    await session.send(message)
+
+            try:
+                async with websockets.serve(echo, "127.0.0.1", websocket_origin,
+                                            ping_interval=None, compression=None, close_timeout=3):
+                    rust = fixture.rust(binary, label)
+                    reverse_tls, static_tls = tls_port(fixture), tls_port(fixture)
+                    config = transport_config(rust, reverse_tls, static_tls, websocket_port, websocket_origin)
+                    rust.apply(config, "transport-listeners")
+                    fixture.stop(rust.process)
+                    rust.start("transport")
+                    certificate = rust.data / "certs" / "default.pem"
+                    write_json(output / "transport.json", {
+                        "certificate_sha256": sha256(certificate), "certificate_verification": True,
+                        "h2_alpn_required": "h2", "h3_alpn_required": "h3",
+                        "reverse_tls_tcp_udp": reverse_tls, "static_tls_tcp_udp": static_tls,
+                        "websocket_h1_tcp": websocket_port, "fallback_allowed": False})
+                    for cache in [False, True]:
+                        if cache:
+                            config["cache"]["enabled"] = True
+                            rust.apply(config, "transport-cache-on")
+                        for protocol in ["h2", "h3"]:
+                            for source, port in [("reverse", reverse_tls), ("static", static_tls)]:
+                                group = f"{protocol}.{source}.cache_{'on' if cache else 'off'}"
+                                try:
+                                    async with client(protocol, port, certificate) as session:
+                                        await http_cases(check, session, protocol, source, cache, fixture)
+                                except Exception as exc:
+                                    check(group + ".runner", False, error=str(exc), traceback=traceback.format_exc())
+                        try:
+                            await websocket_cases(check, websocket_port, cache)
+                        except Exception as exc:
+                            check(f"h1.websocket.cache_{'on' if cache else 'off'}.runner", False,
+                                  error=str(exc), traceback=traceback.format_exc())
+                    for source, port in [("reverse", reverse_tls), ("static", static_tls)]:
+                        try:
+                            async with client("h2", port, certificate) as h2:
+                                warm = await h2.request("GET", "/transport-shared.html")
+                            async with client("h3", port, certificate) as h3:
+                                hit = await h3.request("GET", "/transport-shared.html")
+                            check(f"cache_cross_protocol.{source}.h2_to_h3",
+                                  warm.status == 200 and hit.status == 200 and hit.body == plain
+                                  and hit.headers.get("x-cache") == "HIT",
+                                  first_cache=warm.headers.get("x-cache"), **response_details(hit))
+                        except Exception as exc:
+                            check(f"cache_cross_protocol.{source}.runner", False,
+                                  error=str(exc), traceback=traceback.format_exc())
+                    check("runtime.still_alive", rust.process.poll() is None)
+            finally:
+                peer.close()
+    except Exception as exc:
+        check("runtime.runner", False, error=str(exc), traceback=traceback.format_exc())
+    report = {"label": label, "simulation": False, "seconds": time.monotonic() - started,
+              "passed": sum(case["passed"] for case in cases),
+              "failed": sum(not case["passed"] for case in cases), "cases": cases}
+    write_json(output / "report.json", report)
+    print(f"{label}: {report['passed']} passed, {report['failed']} failed", flush=True)
+    return report
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--binary", action="append", required=True, metavar="LABEL=/ABS/PATH")
+    parser.add_argument("--out", type=Path, required=True)
+    parser.add_argument("--workers", type=int, default=4)
+    parser.add_argument("--python-deps", type=Path)
+    args = parser.parse_args()
+    if args.python_deps:
+        sys.path.insert(0, str(args.python_deps.resolve()))
+    variants = {}
+    for value in args.binary:
+        label, separator, path = value.partition("=")
+        if not separator or not label.replace("-", "").replace("_", "").isalnum() or label in variants:
+            parser.error("--binary must be unique SIMPLE_LABEL=/ABS/PATH")
+        variants[label] = Path(path).resolve()
+    args.out = args.out.resolve()
+    args.out.mkdir(parents=True, exist_ok=True)
+    from transport_clients import validate_head_adapter
+    write_json(args.out / "environment.json", {
+        "kind": "real RustProxy transport acceptance", "simulation": False,
+        "python": sys.version, "platform": platform.platform(),
+        "dependencies": {name: importlib.metadata.version(name) for name in ["aioquic", "httpx", "h2", "websockets"]},
+        "aioquic_head_adapter": validate_head_adapter(),
+        "started_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
+        "not_covered": ["production certificates", "TLS upstreams", "H3 upstreams", "H2/H3 WebSocket extended CONNECT",
+                        "network loss or migration", "24-hour soak", "real-browser navigation"]})
+    reports = {}
+    for label, binary in variants.items():
+        reports[label] = asyncio.run(run_variant(binary, label, args.out / label, args.workers))
+    write_json(args.out / "comparison.json", {
+        "variants": {label: {key: report[key] for key in ["passed", "failed", "seconds"]}
+                     for label, report in reports.items()},
+        "cases": [{"case": case, **{label: next((row["passed"] for row in report["cases"] if row["case"] == case), "not_run")
+                  for label, report in reports.items()}}
+                  for case in sorted({row["case"] for report in reports.values() for row in report["cases"]})]})
+    seal_results(args.out)
+    return 1 if any(report["failed"] for report in reports.values()) else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/performance/wrk_report.lua

@@ -0,0 +1,14 @@
+-- No response hook: benchmarks must not be limited by Lua body processing.
+-- Known payload/status/headers are verified separately before measurement.
+done = function(summary, latency, requests)
+  io.write("RUSTPROXY_BENCH_JSON {")
+  io.write(string.format('"requests":%d,"duration_us":%d,"bytes":%d,',
+    summary.requests, summary.duration, summary.bytes))
+  io.write(string.format('"p50_us":%.3f,"p95_us":%.3f,"p99_us":%.3f,',
+    latency:percentile(50), latency:percentile(95), latency:percentile(99)))
+  io.write(string.format('"latency_mean_us":%.3f,"latency_max_us":%.3f,', latency.mean, latency.max))
+  io.write(string.format('"errors_connect":%d,"errors_read":%d,"errors_write":%d,"errors_timeout":%d,"errors_status":%d',
+    summary.errors.connect, summary.errors.read, summary.errors.write,
+    summary.errors.timeout, summary.errors.status))
+  io.write("}\n")
+end