RustProxy/docs/CACHE_HOTPATH_0.34.0.md

RustProxy 0.34.0 HOT HIT follow-up (source candidate)

What changed

The existing optimized HOT cache no longer removes/reinserts full entries at HIT time. This follow-up further shortens the critical section without changing cache policy or accepting an approximate eviction algorithm:

1. Store each published response's immutable headers in Arc<HeaderMap>. Snapshot the Arc and the shared Bytes body while locked, release the HOT mutex, then clone the headers and build the per-request response. Header cloning is not eliminated, because per-request Age, Cache-Status and transformation annotations must not mutate the stored representation.
2. Preserve exact per-HIT sequence, hit count and LRU ordering. Bypass the Vary index traversal for the common direct-key/no-Vary representation; continue selecting keyed variants for other representations.
3. Replace the per-domain flat pair-key map with a nested ordered map so a known (scope, domain) is updated using borrowed keys. Preserve the 4096 distinct-domain limit and the dashboard wire format. Update cached-byte accounting without allocating String keys.
4. When there are no active BAN rules, a cached response does not acquire the BAN mutex or clone any BAN rules. Active rules still perform expiration, matching and removal. Publication/removal of BAN rules updates an atomic count; BAN and PURGE security checks remain enabled.
5. Remove repeated full HOT-entry sweep() scans. The existing retention_deadline() explicitly returns u64::MAX: entries beyond their freshness TTL are retained for conditional revalidation and max-stale. Capacity/idle eviction and cold-tier GC remain responsible for memory/disk retention. HTTP freshness checks and Age are unchanged.
6. Avoid allocating the static cache-reason text on each repeat occurrence: previously registered reasons update by borrowed lookup. Recorded dashboard fields, counters and reason labels are unchanged.

No new configuration, cache key, COLD on-disk format, migration, panel toggle or feature flag was added. Cache query bypass, transformed representations, origin validators, TTL, hot-to-cold demotion, backend selection and sendfile behavior are intentionally unchanged.

Tests and verification

Native Rust test file: src/cache/tests/hot_hit_034_tests.rs.

• Multi-threaded HOT access checks monotonic, unique hit counters and bounded LRU/index growth.
• Transformed response HIT retains its ETag/extension; changes to the response header map never mutate cached headers. Memory accounting and per-domain saved-byte counters remain exact.
• Exact LRU eviction, selective BAN, stale revalidation retention and conditional client max-stale are covered.
• The manual ignored in-process lookup microbenchmark measures the whole get_hot() response construction, not network RPS.

./dev.sh check
CARGO_INCREMENTAL=0 ./dev.sh test
CARGO_INCREMENTAL=0 cargo build --release --locked
python3 scripts/tests/cache_hot_034_static_test.py
cargo test --release --locked --lib manual_hot_lookup_microbenchmark -- --ignored --nocapture

For real HTTP RPS use the supplied process regression and same-host nginx benchmark (scripts/performance/README.md) with identical configs. The standard nginx comparator benchmark explicitly disables the cache; it cannot measure this HOT-specific improvement. Compare two compiled RustProxy 0.34.0 binaries with the same cache-enabled, cache-warmed vhost and fixed no-query URL. Check X-Cache: HIT, body SHA-256, exact Cache-Status, transform marker, Age, ETag, p95/p99, CPU/request, memory and warm-up/eviction counts while measuring.

Remaining performance bottlenecks

The HOT store still uses a shared mutex to provide exact LRU, per-key generation checks and publication guarantees. Per-response HeaderMap cloning is still required and now occurs outside this lock. Sharding the store or making LRU eventually consistent would require a separate, extensively tested cache architecture revision: doing that without compilation and live soak would risk invalidation, accounting, BAN and tier-demotion regressions.

Acceptance gate open: the changed Rust source was not compiled in this environment, new Rust tests were not run and no HOT HIT RPS improvement has been measured. Source-only guards, SQL and upgrade tests do not establish a safe production build.