# RustProxy 0.34.0 DEV — 1 KiB lock contention follow-up

## Goal and baseline

**Priority 1: reverse proxy, HTTP/1.1 keep-alive, uncompressed 1,040-byte
responses, cache OFF.** Aiming for approximately 100,000 RPS and at least 80%
of nginx under the same pinned-CPU benchmark. Priority 2: local static WWW,
measured separately. No feature removals and **no version bump**.

The user supplied a **new executable** (SHA-256
`1a84c58b20945bba9fdd1bc1a44d9e9d6ac7659d714343a46a1a8511a78331fa`,
`rust-proxy 0.34.0`), apparently built from the prior DEV revision. It passed
151 actual black-box HTTP tests with 0 failures and 1 skipped optional zstd
decoder. This runtime coverage belongs to the **supplied executable only**,
not to the freshly edited source below. Neither `cargo` nor `rustc` is
available in the editing environment; the new revision has NOT been compiled.

## Modified source

### `Cache::record` and `Cache::record_reason`

Before, every HIT, MISS and BYPASS acquired the same `Store` mutex used by the
HOT object store, exact LRU and cache inventory. With cache disabled, a 1 KiB
request still contested that mutex to increment `cache_disabled`.

`src/cache/stats.rs` now holds bounded per-(scope,domain) statistics separately:

- An `ArcSwap`-published immutable, borrowed-lookup BTreeMap of domain handles.
  **The normal path does not lock or allocate**, even for BYPASS. New domains
  alone use a registration mutex and a copy-on-first-visit registry update.
- Atomic counts for HIT, MISS, BYPASS, stored objects and saved body bytes.
  The common static reasons (`cache_disabled`, `cache_hit`, `cache_miss`,
  `request_cookie`) use direct atomic counters. Other reason names use another
  immutable published index and atomic counters; they are registered once.
- Hard cardinality remains **4096 (scope, domain) pairs**. When full, new
  unregistered domains are skipped as before. Scope purge preserves stats.
  Only a revision-bearing global purge resets the per-domain registry.
- The dashboard still exposes the same `DomainCounters`, reason strings,
  saved byte counts, ratio and per-vhost/object views. Counter snapshots are
  quiescent-exact, but under concurrent requests individual fields can reflect
  slightly different instants, consistent with other atomic metrics.
- HOT object and COLD representation/LRU access, barriers, memory accounting,
  cache statuses, hit policy, retry and backend security paths are unchanged.

Trade-offs: a first-visit registration copies the bounded domain index and
allocates one counter set; this is not an adaptive, unbounded cache. The hot
path's ArcSwap guard and lookup still cost CPU. Measure that cost in profiles;
compare under high-cardinality traffic to identify any excessive registration
cost before production acceptance.

### Traffic analytics queue saturation

Previously `Exchange::drop` cloned the vhost, route, method, path, IP, user
agent and country before trying to enqueue the `Sample`. If the bounded
2,048-element MPSC queue was full, all those clones were discarded.

`Analytics::push_lazy` now acquires the same bounded Tokio queue permit with
`try_reserve` first and creates the sample only after a successful reservation.
Rejections increment `queue_dropped` exactly as before. Neither the analytics
worker, counters/top lists, reset behavior nor request log path is removed.
This improves allocation costs **on overflow** but does not turn dropped
analytics into delivered samples; monitor the existing drop counter.

### Per-vhost duration histogram atomics

Previously `VhostMetrics::complete` incremented every cumulative latency bucket
whose threshold covered the observation and the `+Inf` bucket. For an ordinary
short response this meant **11 shared atomic RMWs** for the histogram alone,
on top of existing exchange/status/byte/latency metrics.

It now increments **exactly one exclusive bucket** chosen from the same 10
latency boundaries. Both the regular vhost and Docker-vhost Prometheus exporters
compute the original cumulative `le` buckets, `+Inf` and `_count` from the 11
exclusive buckets **at scrape time**. The metric names, labels, boundaries,
body-lifetime timing and measurement collection remain intact. The cumulative
snapshot may be concurrent with request completions, as before. Added boundary
unit tests for exact thresholds and long-duration `+Inf`.

This is source-only: neither the throughput benefit nor compiled Rust behavior
has been verified in the editing environment. Review exporter parity and
under-load histogram correctness on the newly compiled executable.

## Verification and untested claims

- PASS: source preflight, cache/HOT/LB/RPS/histogram/static source guards and <=500-line
  production Rust modules.
- PASS: SQLite migration + preset constraints, upgrade/lock and source
  replacement tests.
- PASS: compiled user-supplied binary: 151 real HTTP checks, 0 failed,
  1 optional zstd skip. This binary predates the source changes above.
- NOT RUN: Rust compile, Rust unit tests, native regressions or RPS after
  rebuilding the new source. No numerical speedup attributed to this patch.
- Current diagnostic A/B on the user-supplied binary may use
  `scripts/performance/ab_1kb_dev.py` and must remain labelled *old vs already
  supplied binary*, **not** old vs newly edited source. The `wrk` 30 s x 7
  pinned-CPU benchmark remains the production acceptance gate.

## Repeat on a compiler-enabled benchmark host

```bash
./dev.sh doctor
./dev.sh check
CARGO_INCREMENTAL=0 ./dev.sh test
CARGO_INCREMENTAL=0 cargo build --release --locked
python3 scripts/performance/regression.py \
  --binary previous=/absolute/path/to/supplied-rust-proxy \
  --binary edited="$(realpath target/release/rust-proxy)" \
  --out /tmp/rustproxy034-counter-http
python3 scripts/performance/lb_regression.py \
  --binary previous=/absolute/path/to/supplied-rust-proxy \
  --binary edited="$(realpath target/release/rust-proxy)" \
  --out /tmp/rustproxy034-counter-lb
python3 scripts/performance/transport_regression.py \
  --binary previous=/absolute/path/to/supplied-rust-proxy \
  --binary edited="$(realpath target/release/rust-proxy)" \
  --out /tmp/rustproxy034-counter-transport
python3 scripts/performance/benchmark.py \
  --binary previous=/absolute/path/to/supplied-rust-proxy \
  --binary edited="$(realpath target/release/rust-proxy)" \
  --nginx /usr/sbin/nginx --wrk /absolute/path/to/wrk \
  --out /tmp/rustproxy034-counter-benchmark \
  --modes proxy --sizes small --concurrency 16,64,128 \
  --duration 30 --warmup 5 --repeats 7
```

Explicitly set `max_idle_per_host=64` on **both** before/after vhosts, with the
same TLS/H1/H2/health/cache/logging/settings. Verify 1,040 bytes and SHA-256,
no failures, latency percentiles, per-request CPU, RSS, file descriptors,
backend connects and all log/traffic drop counters. Test concurrent global
purges and per-vhost purges during traffic and verify the stats reset behavior
and cache HIT/MISS/BYPASS/reason distribution. Run long soaks and both cold/HOT
cache tests. The 100k RPS / nginx 80% goal is still **OPEN**.

## Build fix after first compiler feedback (OPT2-FIX1)

The first user build of OPT2 failed with six Rust `E0753` diagnostics because
`src/cache/stats.rs` starts with `//!` *inner module* documentation but is
injected via `include!("stats.rs")` after other items in `src/cache/mod.rs`.
The six documentation lines are now ordinary `//` comments; there is no
behavior change. Three unnecessary `mut` bindings in
`src/cache/dashboard.rs` were also removed (warnings only). Version remains
0.34.0; no performance or feature semantics have changed in this fix.

The compiler-enabled host should rerun `./dev.sh check`,
`CARGO_INCREMENTAL=0 ./dev.sh test` and an optimized release build, then
performance benchmarks. This editing environment still lacks `cargo` and
`rustc`, so the fix has been statically checked, **not compiled** here.