v0.34.3

docs/TROUBLESHOOTING.md

@@ -9,6 +9,7 @@ Start with a reproducible request, its hostname/path and the exact listener. Com
 | Domain returns the wrong site or local 404 | DNS A/AAAA, selected listener ID, exact vs wildcard host precedence, Vhost enabled state, redirect/404 host ownership. |
 | HTTP works, HTTPS does not | Trusted certificate, hostname/SAN coverage, TLS listener, port 443 firewall, certificate assignment and whether restart was needed. |
 | HTTPS/H2 works but H3 fails | UDP firewall/NAT, H3 toggle on TLS listener, browser QUIC support and actual transport negotiation. |
+| H2 runs at link speed but H3 is slower | Compare the same origin and client with browser H3 vs H2, keeping TLS/cache/compression consistent. Inspect QUIC loss/cwnd/MTU and VM UDP offload, not only total CPU. |
 | `502` / `503` / `504` to origin | Backend IP/port, pool eligibility/health, private-address policy, upstream HTTPS verification, protocol `auto`/`h2`, timeout and container/PHP process availability. |
 | WebSocket handshake fails or disconnects | Vhost WebSockets flag, backend Upgrade support, TLS/frontend protocol path, WebSocket idle limits and auth redirect. |
 | `403` / `405` / `429` | Applicable IP/ACL/country policy, HTTP methods, rate limit, GeoIP availability, Basic/Tinyauth permissions. |
@@ -47,6 +48,27 @@ rust-proxy --data-dir /var/lib/rustproxy config validate config.json
 
 Use **Logs → Live Logs** or the `--livelogs` CLI mode to correlate request IDs, Vhost/route/backend and status. Inspect **Stored Logs** for previously saved JSONL and selected low-level protocol failures.
 
+## HTTP/3 throughput and CPU diagnostics (0.34.3 source candidate)
+
+Do not assume that low H3 throughput is a CPU bottleneck from the overall VM percentage alone. A 50% VM reading can mean one saturated vCPU on a 2-vCPU machine. Quinn uses UDP datagrams, per-packet QUIC/TLS, congestion control and OS/virtual NIC offloads; H1/H2 over TCP can behave differently under the same load. Quinn enables UDP GSO by default when the platform supports it, but some VM/network stacks fall back to per-datagram sends.
+
+1. Run **four matched tests** against the same OpenSpeedTest/large streaming endpoint: direct backend H1, proxied HTTPS H2, proxied HTTPS H3 download, proxied H3 upload. Record client speed, per-process CPU, **per-thread** CPU, and whether the problem persists with a larger number of simultaneous clients.
+2. On Linux, inspect the NIC and guest kernel counters before and after an H3 run (replace `eth0` with the actual VM interface):
+
+   ```bash
+   ip -s link show dev eth0
+   ethtool -k eth0 | grep -E 'generic-segmentation-offload|generic-receive-offload|udp-segmentation|rx-udp-gro-forwarding'
+   nstat -az | grep -E 'Udp(InErrors|RcvbufErrors|SndbufErrors)|Ip(InDiscards|OutDiscards)'
+   top -H -p "$(pidof rust-proxy)"
+   ```
+
+3. If available, use `perf top -p "$(pidof rust-proxy)"` briefly during the benchmark, and inspect `%sys` / `%soft` in `mpstat -P ALL 1`. Check whether guest CPU has `aes` instruction support with `lscpu`.
+4. For the 0.34.3 source candidate, set `RUST_LOG=rust_proxy::http3=debug` in a **test** deployment and inspect `HTTP/3 bulk transfer transport snapshot` logs: `quic_lost_packets`, `quic_congestion_events`, `quic_rtt_ms`, `quic_cwnd_bytes`, `quic_path_mtu`, `quic_udp_tx_datagrams` and `quic_udp_tx_ios`. Those Quinn counters are **connection-wide cumulative snapshots**, not stream-specific deltas; compare differences for one isolated connection or across successive samples.
+
+5. Run `ss -uapm | grep -A1 rust-proxy` during a transfer; check `skmem rb`, `tb` and whether dropped-datagram `d` increments. RustProxy 0.34.3 defaults to 4 MiB per direction and logs the actual SO_RCVBUF/SO_SNDBUF effective values on bind. If Linux clamps them, inspect `sysctl net.core.rmem_max net.core.wmem_max`; saving new socket buffer targets in the panel requires a listener restart. See [HTTP transport](TRANSPORT.md#quic-udp-socket-buffers-0343).
+
+A rising loss/retransmission count or shrinking congestion window with low CPU suggests packet pacing, network path, UDP buffer/offload or VM issues. With negligible loss and one saturated worker thread, investigate QUIC encryption, UDP send/receive and application streaming costs using `perf` before changing window sizes. HTTP/3 overhead can also affect large file transfers, uploads and long-lived H3 streams; speedtest is only one way to expose it. No throughput improvement is claimed without a production-equivalent QUIC benchmark.
+
 ## Rollout gate for RustProxy 0.34.0
 
 The latest source optimizations were **not all compiled or performance-tested** in the evidence supplied with this release. Before deploying: