RustProxy/docs/MEMORY_STRESS.md

Long-running memory verification (RustProxy 0.30.0)

This procedure supplements static source checks. A synthetic Python model and a local Python HTTP peer do not execute RustProxy and cannot establish its production RSS, heap growth or absence of leaks.

Audit coverage

Area	Source-level finding	Action in the same 0.30.0 version
Selective cache PURGE	Per-key generation metadata could retain every distinct purged key indefinitely after its object was removed	Reset stale per-key metadata at the existing global generation barrier; drop an abnormally large map allocation; regression for 512 different purges
HOT/COLD cache	Body/entry limits, bounded transient leases, limited cold queue, revalidation slots and background cleanup permits	Kept existing behavior
Log output (JSONL/file/Logstash/Loki)	Serialized the same entry again when Loki and a JSONL sink ran simultaneously; sink queues and per-vhost open files were already bounded	Reuse serialized JSONL content for Loki; keep redaction and sink-specific formatting
Historical log discovery	Prior audit already capped directory inventory while scanning, at 256 per-vhost and 64 gzip entries	Retained existing 0.30.0 limits
Traffic analytics	Top lists limited to 32 entries per dimension, at most 64 vhosts and a 2048-item input queue, but repeated labels incurred transient String allocations	Borrow existing labels, allocate only when admitting/replacing a top-list entry
Local directory listing	Two simultaneously live vectors for directory metadata and a copied HTML response	Filter directory metadata in place; move output bytes into response; keep all filenames and HEAD Content-Length behavior
TCP/UDP streams and HTTP/H2/H3	Connection/session permits, finite queues, idle deadlines and completed task draining	No change
Upstream clients, DNS, health probes	DNS worker limit, timeouts, pool recycling, bounded health checks	No change
Compression, transforms, PHP-FPM	Streaming response paths, transform memory and concurrency budgets, FastCGI streaming	No change
Rate limits, ACME, Docker, background tasks	Limited client buckets with expiry and shrink, cancelled/limited background activities	No change

This is not a proof that every runtime allocation is released. The directory listing feature intentionally generates the complete visible directory as HTML; very large directories may still need substantial temporary memory. A high RSS after a load spike may also be allocator reuse instead of an active leak. Verify with real-runtime measurements rather than assuming either explanation.

Source/synthetic verification

python3 tools/checks/memory_contracts.py
python3 tools/checks/high_traffic_memory_model.py --http-requests 10000
./dev.sh check
./dev.sh test

The offline test replays 300,000 distinct PURGE identities over six periods. It verifies the old cumulative model retains 50k/100k/150k/200k/250k/300k metadata keys whereas the revised epoch-based model holds one key after each operation. This is a cardinality model, not a Rust heap measurement. It also sends 10,000 HTTP GETs across 24 concurrent persistent connections and 1,000 paths to a Python test peer to validate the load driver (the default quick ./dev.sh test run sends 2,000 of those requests). The Rust regression for 512 distinct purges is executed only if cargo test runs successfully.

Real RustProxy staging traffic and RSS

On a Linux staging machine with Rust/Cargo installed, configure a vhost and a known reachable backend/local Web Service. Enable whichever logging targets you need to test (JSONL file, Logstash, Loki) in the normal dashboard. Start the proxy and determine the RustProxy PID, not the load generator's PID.

./dev.sh check && ./dev.sh test
# In another terminal: run the configured instance, e.g. ./dev.sh run
pid=$(pgrep -n -x rust-proxy)

# Warm / repeated URL: cache HITs, persistent clients, normal JSON access logs.
python3 tools/dev/memory_load.py \
  --url http://127.0.0.1:8080/ \
  --host-header example.test \
  --pid "$pid" --requests 400000 --concurrency 128 \
  --cooldown 120 --output /tmp/rustproxy-memory-repeat.json

# High cardinality: many URL paths, cache misses, large analytics churn.
# Pick a staging path prefix accepted by your origin; status codes will be
# recorded even when the requested URLs return 404.
python3 tools/dev/memory_load.py \
  --url http://127.0.0.1:8080/ \
  --host-header example.test \
  --pid "$pid" --requests 400000 --concurrency 128 \
  --unique-paths 100000 --cooldown 120 \
  --output /tmp/rustproxy-memory-distinct.json

Run at least three successive traffic/cooldown cycles on the same process. Test separately with access JSONL only, JSONL+Logstash and JSONL+Loki using reachable staging services; add streaming large response bodies, directory listings, TCP/UDP and HTTP/3 using appropriate specialized clients. Send controlled cache PURGE requests through your authenticated cache invalidation API and ensure previously purged cache entries never reappear.

The tool samples /proc/<pid>/status (RSS, high-water mark, threads) and /proc/<pid>/fd at a configurable interval. Output includes response codes, requests/second, received bytes, time series and start/peak/end RSS. Check steady-state RSS after each cooldown, open file descriptors, timeouts, dropped logs, HIT/MISS behavior, and whether growth continues across periods of similar load after caches and connections have stabilized. In addition, inspect cache/transformation memory accounting and capture a Rust heap/allocation profile on the actual instance if RSS rises repeatedly.

Do not run these load tests against an active production service without a maintenance or capacity plan. The load driver does not create vhosts, backend services, log collectors or a RustProxy executable, and it cannot reproduce all TCP, UDP, H2 and H3 protocol behavior. None of these real-instance acceptance steps were performed in the source-only audit environment.