v0.30.0

docs/MEMORY_STRESS.md

@@ -0,0 +1,96 @@
+# Long-running memory verification (RustProxy 0.30.0)
+
+This procedure supplements static source checks. A synthetic Python model and a
+local Python HTTP peer **do not execute RustProxy** and cannot establish its
+production RSS, heap growth or absence of leaks.
+
+## Audit coverage
+
+| Area | Source-level finding | Action in the same 0.30.0 version |
+| --- | --- | --- |
+| Selective cache PURGE | Per-key generation metadata could retain every distinct purged key indefinitely after its object was removed | Reset stale per-key metadata at the existing global generation barrier; drop an abnormally large map allocation; regression for 512 different purges |
+| HOT/COLD cache | Body/entry limits, bounded transient leases, limited cold queue, revalidation slots and background cleanup permits | Kept existing behavior |
+| Log output (JSONL/file/Logstash/Loki) | Serialized the same entry again when Loki and a JSONL sink ran simultaneously; sink queues and per-vhost open files were already bounded | Reuse serialized JSONL content for Loki; keep redaction and sink-specific formatting |
+| Historical log discovery | Prior audit already capped directory inventory while scanning, at 256 per-vhost and 64 gzip entries | Retained existing 0.30.0 limits |
+| Traffic analytics | Top lists limited to 32 entries per dimension, at most 64 vhosts and a 2048-item input queue, but repeated labels incurred transient `String` allocations | Borrow existing labels, allocate only when admitting/replacing a top-list entry |
+| Local directory listing | Two simultaneously live vectors for directory metadata and a copied HTML response | Filter directory metadata in place; move output bytes into response; keep all filenames and HEAD Content-Length behavior |
+| TCP/UDP streams and HTTP/H2/H3 | Connection/session permits, finite queues, idle deadlines and completed task draining | No change |
+| Upstream clients, DNS, health probes | DNS worker limit, timeouts, pool recycling, bounded health checks | No change |
+| Compression, transforms, PHP-FPM | Streaming response paths, transform memory and concurrency budgets, FastCGI streaming | No change |
+| Rate limits, ACME, Docker, background tasks | Limited client buckets with expiry and shrink, cancelled/limited background activities | No change |
+
+This is **not** a proof that every runtime allocation is released. The directory
+listing feature intentionally generates the complete visible directory as HTML;
+very large directories may still need substantial temporary memory. A high RSS
+after a load spike may also be allocator reuse instead of an active leak. Verify
+with real-runtime measurements rather than assuming either explanation.
+
+## Source/synthetic verification
+
+```sh
+python3 tools/checks/memory_contracts.py
+python3 tools/checks/high_traffic_memory_model.py --http-requests 10000
+./dev.sh check
+./dev.sh test
+```
+
+The offline test replays 300,000 distinct PURGE identities over six periods.
+It verifies the old cumulative model retains 50k/100k/150k/200k/250k/300k
+metadata keys whereas the revised epoch-based model holds one key after each
+operation. This is a **cardinality model**, not a Rust heap measurement. It also
+sends 10,000 HTTP GETs across 24 concurrent persistent connections and 1,000
+paths to a **Python test peer** to validate the load driver (the default quick
+`./dev.sh test` run sends 2,000 of those requests). The Rust regression
+for 512 distinct purges is executed only if `cargo test` runs successfully.
+
+## Real RustProxy staging traffic and RSS
+
+On a Linux staging machine with Rust/Cargo installed, configure a vhost and a
+known reachable backend/local Web Service. Enable whichever logging targets you
+need to test (JSONL file, Logstash, Loki) in the normal dashboard. Start the
+proxy and determine the **RustProxy** PID, not the load generator's PID.
+
+```sh
+./dev.sh check && ./dev.sh test
+# In another terminal: run the configured instance, e.g. ./dev.sh run
+pid=$(pgrep -n -x rust-proxy)
+
+# Warm / repeated URL: cache HITs, persistent clients, normal JSON access logs.
+python3 tools/dev/memory_load.py \
+  --url http://127.0.0.1:8080/ \
+  --host-header example.test \
+  --pid "$pid" --requests 400000 --concurrency 128 \
+  --cooldown 120 --output /tmp/rustproxy-memory-repeat.json
+
+# High cardinality: many URL paths, cache misses, large analytics churn.
+# Pick a staging path prefix accepted by your origin; status codes will be
+# recorded even when the requested URLs return 404.
+python3 tools/dev/memory_load.py \
+  --url http://127.0.0.1:8080/ \
+  --host-header example.test \
+  --pid "$pid" --requests 400000 --concurrency 128 \
+  --unique-paths 100000 --cooldown 120 \
+  --output /tmp/rustproxy-memory-distinct.json
+```
+
+Run at least three **successive** traffic/cooldown cycles on the same process.
+Test separately with access JSONL only, JSONL+Logstash and JSONL+Loki using
+reachable staging services; add streaming large response bodies, directory
+listings, TCP/UDP and HTTP/3 using appropriate specialized clients. Send
+controlled cache PURGE requests through your authenticated cache invalidation
+API and ensure previously purged cache entries never reappear.
+
+The tool samples `/proc/<pid>/status` (RSS, high-water mark, threads) and
+`/proc/<pid>/fd` at a configurable interval. Output includes response codes,
+requests/second, received bytes, time series and start/peak/end RSS. Check
+steady-state RSS after each cooldown, open file descriptors, timeouts, dropped
+logs, HIT/MISS behavior, and whether growth **continues** across periods of
+similar load after caches and connections have stabilized. In addition, inspect
+cache/transformation memory accounting and capture a Rust heap/allocation
+profile on the actual instance if RSS rises repeatedly.
+
+Do not run these load tests against an active production service without a
+maintenance or capacity plan. The load driver does not create vhosts, backend
+services, log collectors or a RustProxy executable, and it cannot reproduce
+all TCP, UDP, H2 and H3 protocol behavior. None of these real-instance
+acceptance steps were performed in the source-only audit environment.