RustProxy/docs/TROUBLESHOOTING.md

# Troubleshooting and validation checklist

Start with a reproducible request, its hostname/path and the exact listener. Compare **Vhost**, **Backend**, **Cache**, **Logs** and the service journal before changing multiple settings at once.

## Quick symptoms

| Symptom | First checks |
|---|---|
| Domain returns the wrong site or local 404 | DNS A/AAAA, selected listener ID, exact vs wildcard host precedence, Vhost enabled state, redirect/404 host ownership. |
| HTTP works, HTTPS does not | Trusted certificate, hostname/SAN coverage, TLS listener, port 443 firewall, certificate assignment and whether restart was needed. |
| HTTPS/H2 works but H3 fails | UDP firewall/NAT, H3 toggle on TLS listener, browser QUIC support and actual transport negotiation. |
| H2 runs at link speed but H3 is slower | Compare the same origin and client with browser H3 vs H2, keeping TLS/cache/compression consistent. Inspect QUIC loss/cwnd/MTU and VM UDP offload, not only total CPU. |
| Ordinary `502` / `503` / `504` to origin | Backend IP/port, pool eligibility/health, private-address policy, upstream HTTPS verification, protocol `auto`/`h2`, timeout and container/PHP process availability. A locally generated ordinary 502 has the generic **Bad gateway** page. |
| `502 Tinyauth connection error` | The authentication backend cannot be reached. Check Tinyauth process, DNS, verifier URL/port, network firewall, TLS/certificate, private backend allowlist and the RustProxy service journal. Do not route the verification URL back through the protected Vhost. |
| `502 Tinyauth verification error` | Tinyauth returned an unexpected status/body or identity headers were too large. Verify `/api/auth/nginx`, proxy header injection and the installed Tinyauth version; check the server journal for details. |
| `504 Tinyauth timeout` | The Tinyauth verification deadline was reached. Check service load/latency and `connect_timeout_seconds` / `timeout_seconds` (Vhost Authentication tab). |
| `502 <provider> connection error` / `502 <provider> verification error` / `504 <provider> timeout` | All delegated providers use the same `error_auth_provider.html` template. The name identifies the failing provider; the status identifies network, response-validation, or timeout failure. Check the internal URL, DNS/TLS, allowlist, deadline and RustProxy journal; origin access stays denied. |
| Authelia redirects repeatedly | Check `login_url` HTTPS origin, `session.cookies` domains/`authelia_url` mapping, real proxy forwarding headers and original application host. Unrelated registered domains require matching per-domain cookie/portal settings. |
| authentik redirects repeatedly or `/outpost.goauthentik.io` is 404 | Check its proxy provider is **Forward auth (single application)** for the exact external application URL, assigned to the correct outpost. Use the outpost internal origin in the Vhost; RustProxy reserves this path on each protected host. Ensure the outpost can reach authentik and browser accepts HTTPS cookies. |
| WebSocket handshake fails or disconnects | Vhost WebSockets flag, backend Upgrade support, TLS/frontend protocol path, WebSocket idle limits and auth redirect. |
| `403` / `405` / `429` | Applicable IP/ACL/country policy, HTTP methods, rate limit, GeoIP availability, Basic/Tinyauth permissions. |
| Cache always BYPASS or MISS | Global + Vhost enablement, cookies/login, origin `Cache-Control`, `Set-Cookie`, `Vary`, TTL, size and exclusion rules. |
| Content remains stale | Cache Age/resident age, origin validator, policy changes, matching scheme/Host/query and selected purge scope. |
| Static file not found | Document root, process read permissions, index files, symlink policy, `try_files` and rewrite ordering. |
| PHP returns errors or another site's PHP settings | FPM endpoint/socket permission, distinct pool/INI settings, pool restart, PHP `PATH_INFO`, application logs. |
| Docker Vhost remains WAITING / 503 | First agent poll, token/certificate pin, published host TCP binding, backend IP, container state and health. |
| Stream TCP/UDP unreachable | Actual listen address/port, protocol and firewall, backend IP, Stream health/session limits; HTTP Vhost settings do not apply. |
| Administration cannot be accessed remotely | Admin binds loopback by default. Use SSH tunnel or configure **separate admin HTTPS** and intentional bind address. |
| Prometheus exporter unavailable | Dedicated listener, CIDR and token, configured path, startup/restart state and monitoring server route. |
| Logs missing or search incomplete | Check sink enabled/path/permissions, Live vs Stored distinction, log rotation and bounded scan/truncation notices. |
| Certificate renewal fails | ACME runtime, HTTP-01 TCP 80, DNS-01 token/zone, expired credentials, Certbot files and external key ACL after rotation. |

## Useful commands

Local service status for installed Linux/systemd deployments:

```bash
systemctl status rustproxy
journalctl -u rustproxy -n 200 --no-pager
```

Inspect a specific HTTP frontend with the correct Host (adapt hostname, port and path):

```bash
curl -v -H 'Host: app.example.com' http://127.0.0.1:8080/health
```

For HTTPS certificate/routing testing use the real site DNS or a deliberate `curl --resolve` mapping; avoid permanently disabling certificate verification as a workaround. The local CLI can check configuration without saving:

```bash
rust-proxy --data-dir /var/lib/rustproxy config get
rust-proxy --data-dir /var/lib/rustproxy config validate config.json
```

Use **Logs → Live Logs** or the `--livelogs` CLI mode to correlate request IDs, Vhost/route/backend and status. Inspect **Stored Logs** for previously saved JSONL and selected low-level protocol failures.

## HTTP/3 throughput and CPU diagnostics
Do not assume that low H3 throughput is a CPU bottleneck from the overall VM percentage alone. A 50% VM reading can mean one saturated vCPU on a 2-vCPU machine. Quinn uses UDP datagrams, per-packet QUIC/TLS, congestion control and OS/virtual NIC offloads; H1/H2 over TCP can behave differently under the same load. Quinn enables UDP GSO by default when the platform supports it, but some VM/network stacks fall back to per-datagram sends.

1. Run **four matched tests** against the same OpenSpeedTest/large streaming endpoint: direct backend H1, proxied HTTPS H2, proxied HTTPS H3 download, proxied H3 upload. Record client speed, per-process CPU, **per-thread** CPU, and whether the problem persists with a larger number of simultaneous clients.
2. On Linux, inspect the NIC and guest kernel counters before and after an H3 run (replace `eth0` with the actual VM interface):

   ```bash
   ip -s link show dev eth0
   ethtool -k eth0 | grep -E 'generic-segmentation-offload|generic-receive-offload|udp-segmentation|rx-udp-gro-forwarding'
   nstat -az | grep -E 'Udp(InErrors|RcvbufErrors|SndbufErrors)|Ip(InDiscards|OutDiscards)'
   top -H -p "$(pidof rust-proxy)"
   ```

3. If available, use `perf top -p "$(pidof rust-proxy)"` briefly during the benchmark, and inspect `%sys` / `%soft` in `mpstat -P ALL 1`. Check whether guest CPU has `aes` instruction support with `lscpu`.
4. Set `RUST_LOG=rust_proxy::http3=debug` in a **test** deployment and inspect `HTTP/3 bulk transfer transport snapshot` logs: `quic_lost_packets`, `quic_congestion_events`, `quic_rtt_ms`, `quic_cwnd_bytes`, `quic_path_mtu`, `quic_udp_tx_datagrams` and `quic_udp_tx_ios`. Those Quinn counters are **connection-wide cumulative snapshots**, not stream-specific deltas; compare differences for one isolated connection or across successive samples.

5. Run `ss -uapm | grep -A1 rust-proxy` during a transfer; check `skmem rb`, `tb` and whether dropped-datagram `d` increments. RustProxy defaults to 4 MiB per direction and logs the actual SO_RCVBUF/SO_SNDBUF effective values on bind. If Linux clamps them, inspect `sysctl net.core.rmem_max net.core.wmem_max`; saving new socket buffer targets in the panel requires a listener restart. See [HTTP transport](TRANSPORT.md#quic-udp-socket-buffers).

A rising loss/retransmission count or shrinking congestion window with low CPU suggests packet pacing, network path, UDP buffer/offload or VM issues. With negligible loss and one saturated worker thread, investigate QUIC encryption, UDP send/receive and application streaming costs using `perf` before changing window sizes. HTTP/3 overhead can also affect large file transfers, uploads and long-lived H3 streams; speedtest is only one way to expose it. No throughput improvement is claimed without a production-equivalent QUIC benchmark.

## Release rollout checklist

1. Verify HTTP/1.1, HTTPS, HTTP/2, HTTP/3 (if enabled), WebSockets and multi-backend failover against a real origin.
2. Test private origin constraints, ACL/authentication, rate limits and protected cache content.
3. Test Web Server static files, PHP-FPM, cache revalidation/invalidation and `sendfile=auto` if enabled.
4. Exercise certificate renewal/reload and a backup **restore** on staging.
5. Compare performance only with matched configurations and repeatable traffic; see [Performance](PERFORMANCE.md).

For persistent problems, use the feature-specific guide in [Documentation](README.md) and keep a redacted error example with the version and request ID.

## OIDC central login
- **Login does not start / 502:** verify that the HTTPS Issuer URL is the exact `issuer` from `/.well-known/openid-configuration`, including any path or trailing slash. RustProxy must resolve that address and connect to it with a trusted certificate, or you must configure an explicit internal backend origin for server-side OIDC requests. Discovery, authorization, token, UserInfo and JWKS endpoints must have the same HTTPS origin as the issuer.
- **Redirect URI mismatch:** register the exact `https://APP-HOST/_rustproxy/oidc/callback` value with the identity provider. The callback hostname must be explicitly listed in the protected Vhost hosts and be accessed over HTTPS. Don't protect the IdP's own login Vhost using this OIDC setting.
- **Invalid code, nonce or signature:** ensure that the provider issues an `id_token` using RS256 or ES256 and that a confidential client uses `client_secret_basic` with authorization code and PKCE S256. Repeated callback requests or overwritten browser transaction cookies are refused. A single browser login flow per protected host avoids cookie conflicts.
- **No identity on the backend:** trust only requests through RustProxy. A verified login injects `Remote-Sub`, `Remote-Issuer`, `Remote-User`, and optional profile headers; the application must explicitly consume them. An OIDC session cookie on one app/domain is not a login cookie for another.
- **User removed at the provider but session still works:** local application sessions are independent and expire after the configured `session_ttl_seconds` (one hour by default); visit `/_rustproxy/oidc/logout` on the application origin to clear the local session. RustProxy does not implement back-channel logout or token revocation.