RustProxy/docs/TROUBLESHOOTING.md

Troubleshooting and validation checklist

Start with a reproducible request, its hostname/path and the exact listener. Compare Vhost, Backend, Cache, Logs and the service journal before changing multiple settings at once.

Quick symptoms

Symptom	First checks
Domain returns the wrong site or local 404	DNS A/AAAA, selected listener ID, exact vs wildcard host precedence, Vhost enabled state, redirect/404 host ownership.
HTTP works, HTTPS does not	Trusted certificate, hostname/SAN coverage, TLS listener, port 443 firewall, certificate assignment and whether restart was needed.
HTTPS/H2 works but H3 fails	UDP firewall/NAT, H3 toggle on TLS listener, browser QUIC support and actual transport negotiation.
H2 runs at link speed but H3 is slower	Compare the same origin and client with browser H3 vs H2, keeping TLS/cache/compression consistent. Inspect QUIC loss/cwnd/MTU and VM UDP offload, not only total CPU.
Ordinary 502 / 503 / 504 to origin	Backend IP/port, pool eligibility/health, private-address policy, upstream HTTPS verification, protocol auto/h2, timeout and container/PHP process availability. A locally generated ordinary 502 has the generic Bad gateway page.
502 Tinyauth connection error	The authentication backend cannot be reached. Check Tinyauth process, DNS, verifier URL/port, network firewall, TLS/certificate, private backend allowlist and the RustProxy service journal. Do not route the verification URL back through the protected Vhost.
502 Tinyauth verification error	Tinyauth returned an unexpected status/body or identity headers were too large. Verify /api/auth/nginx, proxy header injection and the installed Tinyauth version; check the server journal for details.
504 Tinyauth timeout	The Tinyauth verification deadline was reached. Check service load/latency and connect_timeout_seconds / timeout_seconds (Vhost Authentication tab).
502 <provider> connection error / 502 <provider> verification error / 504 <provider> timeout	All delegated providers use the same error_auth_provider.html template. The name identifies the failing provider; the status identifies network, response-validation, or timeout failure. Check the internal URL, DNS/TLS, allowlist, deadline and RustProxy journal; origin access stays denied.
Authelia redirects repeatedly	Check login_url HTTPS origin, session.cookies domains/authelia_url mapping, real proxy forwarding headers and original application host. Unrelated registered domains require matching per-domain cookie/portal settings.
authentik redirects repeatedly or /outpost.goauthentik.io is 404	Check its proxy provider is Forward auth (single application) for the exact external application URL, assigned to the correct outpost. Use the outpost internal origin in the Vhost; RustProxy reserves this path on each protected host. Ensure the outpost can reach authentik and browser accepts HTTPS cookies.
WebSocket handshake fails or disconnects	Vhost WebSockets flag, backend Upgrade support, TLS/frontend protocol path, WebSocket idle limits and auth redirect.
403 / 405 / 429	Applicable IP/ACL/country policy, HTTP methods, rate limit, GeoIP availability, Basic/Tinyauth permissions.
Cache always BYPASS or MISS	Global + Vhost enablement, cookies/login, origin Cache-Control, Set-Cookie, Vary, TTL, size and exclusion rules.
Content remains stale	Cache Age/resident age, origin validator, policy changes, matching scheme/Host/query and selected purge scope.
Static file not found	Document root, process read permissions, index files, symlink policy, try_files and rewrite ordering.
PHP returns errors or another site's PHP settings	FPM endpoint/socket permission, distinct pool/INI settings, pool restart, PHP PATH_INFO, application logs.
Docker Vhost remains WAITING / 503	First agent poll, token/certificate pin, published host TCP binding, backend IP, container state and health.
Stream TCP/UDP unreachable	Actual listen address/port, protocol and firewall, backend IP, Stream health/session limits; HTTP Vhost settings do not apply.
Administration cannot be accessed remotely	Admin binds loopback by default. Use SSH tunnel or configure separate admin HTTPS and intentional bind address.
Prometheus exporter unavailable	Dedicated listener, CIDR and token, configured path, startup/restart state and monitoring server route.
Logs missing or search incomplete	Check sink enabled/path/permissions, Live vs Stored distinction, log rotation and bounded scan/truncation notices.
Certificate renewal fails	ACME runtime, HTTP-01 TCP 80, DNS-01 token/zone, expired credentials, Certbot files and external key ACL after rotation.

Useful commands

Local service status for installed Linux/systemd deployments:

systemctl status rustproxy
journalctl -u rustproxy -n 200 --no-pager

Inspect a specific HTTP frontend with the correct Host (adapt hostname, port and path):

curl -v -H 'Host: app.example.com' http://127.0.0.1:8080/health

For HTTPS certificate/routing testing use the real site DNS or a deliberate curl --resolve mapping; avoid permanently disabling certificate verification as a workaround. The local CLI can check configuration without saving:

rust-proxy --data-dir /var/lib/rustproxy config get
rust-proxy --data-dir /var/lib/rustproxy config validate config.json

Use Logs → Live Logs or the --livelogs CLI mode to correlate request IDs, Vhost/route/backend and status. Inspect Stored Logs for previously saved JSONL and selected low-level protocol failures.

HTTP/3 throughput and CPU diagnostics (0.34.3 source candidate)

Do not assume that low H3 throughput is a CPU bottleneck from the overall VM percentage alone. A 50% VM reading can mean one saturated vCPU on a 2-vCPU machine. Quinn uses UDP datagrams, per-packet QUIC/TLS, congestion control and OS/virtual NIC offloads; H1/H2 over TCP can behave differently under the same load. Quinn enables UDP GSO by default when the platform supports it, but some VM/network stacks fall back to per-datagram sends.

1. Run four matched tests against the same OpenSpeedTest/large streaming endpoint: direct backend H1, proxied HTTPS H2, proxied HTTPS H3 download, proxied H3 upload. Record client speed, per-process CPU, per-thread CPU, and whether the problem persists with a larger number of simultaneous clients.

2. On Linux, inspect the NIC and guest kernel counters before and after an H3 run (replace eth0 with the actual VM interface):

   ip -s link show dev eth0
   ethtool -k eth0 | grep -E 'generic-segmentation-offload|generic-receive-offload|udp-segmentation|rx-udp-gro-forwarding'
   nstat -az | grep -E 'Udp(InErrors|RcvbufErrors|SndbufErrors)|Ip(InDiscards|OutDiscards)'
   top -H -p "$(pidof rust-proxy)"
   

3. If available, use perf top -p "$(pidof rust-proxy)" briefly during the benchmark, and inspect %sys / %soft in mpstat -P ALL 1. Check whether guest CPU has aes instruction support with lscpu.

4. For the 0.34.3 source candidate, set RUST_LOG=rust_proxy::http3=debug in a test deployment and inspect HTTP/3 bulk transfer transport snapshot logs: quic_lost_packets, quic_congestion_events, quic_rtt_ms, quic_cwnd_bytes, quic_path_mtu, quic_udp_tx_datagrams and quic_udp_tx_ios. Those Quinn counters are connection-wide cumulative snapshots, not stream-specific deltas; compare differences for one isolated connection or across successive samples.

5. Run ss -uapm | grep -A1 rust-proxy during a transfer; check skmem rb, tb and whether dropped-datagram d increments. RustProxy 0.34.3 defaults to 4 MiB per direction and logs the actual SO_RCVBUF/SO_SNDBUF effective values on bind. If Linux clamps them, inspect sysctl net.core.rmem_max net.core.wmem_max; saving new socket buffer targets in the panel requires a listener restart. See HTTP transport.

A rising loss/retransmission count or shrinking congestion window with low CPU suggests packet pacing, network path, UDP buffer/offload or VM issues. With negligible loss and one saturated worker thread, investigate QUIC encryption, UDP send/receive and application streaming costs using perf before changing window sizes. HTTP/3 overhead can also affect large file transfers, uploads and long-lived H3 streams; speedtest is only one way to expose it. No throughput improvement is claimed without a production-equivalent QUIC benchmark.

Rollout gate for RustProxy 0.34.0

The latest source optimizations were not all compiled or performance-tested in the evidence supplied with this release. Before deploying:

1. Build using the declared Rust toolchain and run the project's available regression suite on the target platform.
2. Verify HTTP/1.1, HTTPS, HTTP/2, HTTP/3 (if enabled), WebSockets and multi-backend failover with a real origin.
3. Test private origin constraints, ACL/auth/CSRF, rate limits and protected cache content.
4. Test Web Server static files, PHP-FPM, cache revalidation/invalidation and optional sendfile=auto only if enabled.
5. Exercise certificate renewal/reload and backup restore on isolated staging.
6. Compare performance only under matched configurations with controlled repeatable traffic; do not label 0.34.0 as faster merely because its source reduces internal work.

For persistent problems, use the feature-specific guide in Documentation and preserve a redacted error example with environment/version and request ID.

OIDC central login (0.34.16)

• Login does not start / 502: verify that the HTTPS Issuer URL is the exact issuer from /.well-known/openid-configuration, including any path or trailing slash. RustProxy must resolve that address and connect to it with a trusted certificate, or you must configure an explicit internal backend origin for server-side OIDC requests. Discovery, authorization, token, UserInfo and JWKS endpoints must have the same HTTPS origin as the issuer.
• Redirect URI mismatch: register the exact https://APP-HOST/_rustproxy/oidc/callback value with the identity provider. The callback hostname must be explicitly listed in the protected Vhost hosts and be accessed over HTTPS. Don't protect the IdP's own login Vhost using this OIDC setting.
• Invalid code, nonce or signature: ensure that the provider issues an id_token using RS256 or ES256 and that a confidential client uses client_secret_basic with authorization code and PKCE S256. Repeated callback requests or overwritten browser transaction cookies are refused. A single browser login flow per protected host avoids cookie conflicts.
• No identity on the backend: trust only requests through RustProxy. A verified login injects Remote-Sub, Remote-Issuer, Remote-User, and optional profile headers; the application must explicitly consume them. An OIDC session cookie on one app/domain is not a login cookie for another.
• User removed at the provider but session still works: local application sessions are independent and expire after the configured session_ttl_seconds (one hour by default); visit /_rustproxy/oidc/logout on the application origin to clear the local session. RustProxy does not implement back-channel logout or token revocation.