RustProxy/docs/TROUBLESHOOTING.md

Troubleshooting and validation checklist

Start with a reproducible request, its hostname/path and the exact listener. Compare Vhost, Backend, Cache, Logs and the service journal before changing multiple settings at once.

Quick symptoms

Symptom	First checks
Domain returns the wrong site or local 404	DNS A/AAAA, selected listener ID, exact vs wildcard host precedence, Vhost enabled state, redirect/404 host ownership.
HTTP works, HTTPS does not	Trusted certificate, hostname/SAN coverage, TLS listener, port 443 firewall, certificate assignment and whether restart was needed.
HTTPS/H2 works but H3 fails	UDP firewall/NAT, H3 toggle on TLS listener, browser QUIC support and actual transport negotiation.
H2 runs at link speed but H3 is slower	Compare the same origin and client with browser H3 vs H2, keeping TLS/cache/compression consistent. Inspect QUIC loss/cwnd/MTU and VM UDP offload, not only total CPU.
502 / 503 / 504 to origin	Backend IP/port, pool eligibility/health, private-address policy, upstream HTTPS verification, protocol auto/h2, timeout and container/PHP process availability.
WebSocket handshake fails or disconnects	Vhost WebSockets flag, backend Upgrade support, TLS/frontend protocol path, WebSocket idle limits and auth redirect.
403 / 405 / 429	Applicable IP/ACL/country policy, HTTP methods, rate limit, GeoIP availability, Basic/Tinyauth permissions.
Cache always BYPASS or MISS	Global + Vhost enablement, cookies/login, origin Cache-Control, Set-Cookie, Vary, TTL, size and exclusion rules.
Content remains stale	Cache Age/resident age, origin validator, policy changes, matching scheme/Host/query and selected purge scope.
Static file not found	Document root, process read permissions, index files, symlink policy, try_files and rewrite ordering.
PHP returns errors or another site's PHP settings	FPM endpoint/socket permission, distinct pool/INI settings, pool restart, PHP PATH_INFO, application logs.
Docker Vhost remains WAITING / 503	First agent poll, token/certificate pin, published host TCP binding, backend IP, container state and health.
Stream TCP/UDP unreachable	Actual listen address/port, protocol and firewall, backend IP, Stream health/session limits; HTTP Vhost settings do not apply.
Administration cannot be accessed remotely	Admin binds loopback by default. Use SSH tunnel or configure separate admin HTTPS and intentional bind address.
Prometheus exporter unavailable	Dedicated listener, CIDR and token, configured path, startup/restart state and monitoring server route.
Logs missing or search incomplete	Check sink enabled/path/permissions, Live vs Stored distinction, log rotation and bounded scan/truncation notices.
Certificate renewal fails	ACME runtime, HTTP-01 TCP 80, DNS-01 token/zone, expired credentials, Certbot files and external key ACL after rotation.

Useful commands

Local service status for installed Linux/systemd deployments:

systemctl status rustproxy
journalctl -u rustproxy -n 200 --no-pager

Inspect a specific HTTP frontend with the correct Host (adapt hostname, port and path):

curl -v -H 'Host: app.example.com' http://127.0.0.1:8080/health

For HTTPS certificate/routing testing use the real site DNS or a deliberate curl --resolve mapping; avoid permanently disabling certificate verification as a workaround. The local CLI can check configuration without saving:

rust-proxy --data-dir /var/lib/rustproxy config get
rust-proxy --data-dir /var/lib/rustproxy config validate config.json

Use Logs → Live Logs or the --livelogs CLI mode to correlate request IDs, Vhost/route/backend and status. Inspect Stored Logs for previously saved JSONL and selected low-level protocol failures.

HTTP/3 throughput and CPU diagnostics (0.34.3 source candidate)

Do not assume that low H3 throughput is a CPU bottleneck from the overall VM percentage alone. A 50% VM reading can mean one saturated vCPU on a 2-vCPU machine. Quinn uses UDP datagrams, per-packet QUIC/TLS, congestion control and OS/virtual NIC offloads; H1/H2 over TCP can behave differently under the same load. Quinn enables UDP GSO by default when the platform supports it, but some VM/network stacks fall back to per-datagram sends.

1. Run four matched tests against the same OpenSpeedTest/large streaming endpoint: direct backend H1, proxied HTTPS H2, proxied HTTPS H3 download, proxied H3 upload. Record client speed, per-process CPU, per-thread CPU, and whether the problem persists with a larger number of simultaneous clients.

2. On Linux, inspect the NIC and guest kernel counters before and after an H3 run (replace eth0 with the actual VM interface):

   ip -s link show dev eth0
   ethtool -k eth0 | grep -E 'generic-segmentation-offload|generic-receive-offload|udp-segmentation|rx-udp-gro-forwarding'
   nstat -az | grep -E 'Udp(InErrors|RcvbufErrors|SndbufErrors)|Ip(InDiscards|OutDiscards)'
   top -H -p "$(pidof rust-proxy)"
   

3. If available, use perf top -p "$(pidof rust-proxy)" briefly during the benchmark, and inspect %sys / %soft in mpstat -P ALL 1. Check whether guest CPU has aes instruction support with lscpu.

4. For the 0.34.3 source candidate, set RUST_LOG=rust_proxy::http3=debug in a test deployment and inspect HTTP/3 bulk transfer transport snapshot logs: quic_lost_packets, quic_congestion_events, quic_rtt_ms, quic_cwnd_bytes, quic_path_mtu, quic_udp_tx_datagrams and quic_udp_tx_ios. Those Quinn counters are connection-wide cumulative snapshots, not stream-specific deltas; compare differences for one isolated connection or across successive samples.

5. Run ss -uapm | grep -A1 rust-proxy during a transfer; check skmem rb, tb and whether dropped-datagram d increments. RustProxy 0.34.3 defaults to 4 MiB per direction and logs the actual SO_RCVBUF/SO_SNDBUF effective values on bind. If Linux clamps them, inspect sysctl net.core.rmem_max net.core.wmem_max; saving new socket buffer targets in the panel requires a listener restart. See HTTP transport.

A rising loss/retransmission count or shrinking congestion window with low CPU suggests packet pacing, network path, UDP buffer/offload or VM issues. With negligible loss and one saturated worker thread, investigate QUIC encryption, UDP send/receive and application streaming costs using perf before changing window sizes. HTTP/3 overhead can also affect large file transfers, uploads and long-lived H3 streams; speedtest is only one way to expose it. No throughput improvement is claimed without a production-equivalent QUIC benchmark.

Rollout gate for RustProxy 0.34.0

The latest source optimizations were not all compiled or performance-tested in the evidence supplied with this release. Before deploying:

1. Build using the declared Rust toolchain and run the project's available regression suite on the target platform.
2. Verify HTTP/1.1, HTTPS, HTTP/2, HTTP/3 (if enabled), WebSockets and multi-backend failover with a real origin.
3. Test private origin constraints, ACL/auth/CSRF, rate limits and protected cache content.
4. Test Web Server static files, PHP-FPM, cache revalidation/invalidation and optional sendfile=auto only if enabled.
5. Exercise certificate renewal/reload and backup restore on isolated staging.
6. Compare performance only under matched configurations with controlled repeatable traffic; do not label 0.34.0 as faster merely because its source reduces internal work.

For persistent problems, use the feature-specific guide in Documentation and preserve a redacted error example with environment/version and request ID.