# Dedicated Prometheus endpoint

RustProxy exposes Prometheus text exposition 0.0.4 on a dedicated read-only
listener. It is independent from administrator browser sessions and does not
export raw client IPs, request paths, query strings, full User-Agent strings, cookies,
credentials, backend URLs or ACME account secrets.

## Configuration

```json
{
  "prometheus": {
    "enabled": false,
    "listen": "127.0.0.1:9191",
    "path": "/metrics",
    "allowed_cidrs": ["127.0.0.0/8", "::1/128"],
    "bearer_token_env": null,
    "allow_trusted_network_without_token": false,
    "include_system": true,
    "include_vhosts": true,
    "include_traffic": true,
    "include_backends": true,
    "include_docker_vhosts": true,
    "include_cache_domains": true,
    "include_streams": true,
    "include_logging": true
  }
}
```

The listener is plain HTTP. Keep it on loopback, a private monitoring network or
behind a protected tunnel. `allowed_cidrs` is always evaluated against the actual
transport peer (`PeerAddr`); `Forwarded` and `X-Forwarded-For` never grant metrics
access.

### Bearer token

Set `bearer_token_env` to an environment-variable name such as
`RUST_PROXY_METRICS_TOKEN`. RustProxy stores only the variable name. The value must
contain 32..4096 printable non-whitespace ASCII characters and must be present at
startup.

### Trusted network without a token

`allow_trusted_network_without_token=true` explicitly allows a peer already matched
by `allowed_cidrs` to scrape without a bearer token. This can be used for a private
Prometheus VLAN. An explicit IPv4/IPv6 `/0` is rejected in this mode. Keep the CIDR
list narrow; enabling this option is an authentication decision, not a forwarded-IP
feature.

A non-loopback exporter requires either a bearer-token variable or this explicit
trusted-network bypass. Changing enabled/listen/token-variable requires restart;
path, CIDRs, collector switches and the trusted-network bypass apply live.

Only `GET` on the exact configured path without a query string is accepted. A
wrong path is `404`, wrong method `405`, disallowed peer `403`, invalid/missing
required bearer `401`, and exporter concurrency exhaustion `429`.

## Web Server and PHP-FPM metrics (0.33.0)

Prometheus already exports core Web Server/PHP response counters by configured
`web_service` ID. The 0.33.0 additions below are enabled with the existing
Prometheus exporter (no new listener, credentials, options or open routes).

| Family | Type | Labels | Meaning |
|---|---|---|---|
| `rust_proxy_web_server_requests_total` | counter | `web_service` | Header-ready WWW responses, including PHP |
| `rust_proxy_web_server_responses_total` | counter | `web_service`, `class` | HTTP 2xx/3xx/4xx/5xx |
| `rust_proxy_web_server_response_header_duration_microseconds_total` | counter | `web_service` | Cumulative time until headers |
| `rust_proxy_web_server_requests_active` | gauge | `web_service` | Requests still awaiting headers |
| `rust_proxy_web_server_request_kind_total` | counter | `web_service`, `kind` | `static`, `php`, `other` handler |
| `rust_proxy_web_server_methods_total` | counter | `web_service`, `method` | `GET`, `POST`, `PUT`, `DELETE`, `other` |
| `rust_proxy_web_server_status_code_total` | counter | `web_service`, `status` | Selected 200/206/304/403/404/500/502/503 |
| `rust_proxy_web_server_response_body_bytes_total` | counter | `web_service` | Body bytes delivered before final compression |
| `rust_proxy_web_server_compression_total` | counter | `web_service`, `encoding` | Downstream gzip/br/zstd/deflate operations |
| `rust_proxy_php_requests_total` | counter | `web_service` | PHP response headers processed |
| `rust_proxy_php_responses_total` | counter | `web_service`, `class` | PHP response status classes |
| `rust_proxy_php_response_header_duration_microseconds_total` | counter | `web_service` | Cumulative PHP time until headers |
| `rust_proxy_php_response_body_bytes_total` | counter | `web_service` | PHP body bytes emitted before compression |
| `rust_proxy_php_concurrency_available` | gauge | `web_service` | Free RustProxy PHP request permits |
| `rust_proxy_php_concurrency_limit` | gauge | `web_service` | Configured concurrent PHP request limit |
| `rust_proxy_php_concurrency_in_use` | gauge | `web_service` | Permits currently held by PHP requests |
| `rust_proxy_php_failures_total` | counter | `web_service`, `reason` | Fixed FastCGI/PHP failure classes |

PHP `reason` is one of `capacity`, `connect_timeout`, `connect_error`,
`request_timeout`, `request_io`, `response_timeout`, `response_io`,
`invalid_response`, `body_io`. PHP concurrency is the admission semaphore in
RustProxy, not external PHP worker health or PHP-FPM child count. Status-code
labels are deliberately restricted to a stable, useful subset; arbitrary code,
request paths, IP addresses and untrusted method strings are not labels.
The exported byte counters are measured **before the final dynamic response
encoder**, so treat them as payload sizes rather than measured network transfer.
Precompressed static file delivery is not counted in the downstream compression
family. All time-to-headers metrics exclude streamed transfer duration.

Useful Grafana/PromQL examples:

```promql
sum(rate(rust_proxy_web_server_requests_total[5m]))
sum by (class) (rate(rust_proxy_web_server_responses_total[5m]))
sum by (kind) (rate(rust_proxy_web_server_request_kind_total[5m]))
100 * sum(rust_proxy_php_concurrency_in_use) / clamp_min(sum(rust_proxy_php_concurrency_limit), 1)
1e-3 * sum(rate(rust_proxy_web_server_response_header_duration_microseconds_total[5m])) / clamp_min(sum(rate(rust_proxy_web_server_requests_total[5m])), 1e-9)
sum by (reason) (rate(rust_proxy_php_failures_total[5m]))
```

Import `grafana/rustproxy-overview.json` and use the **Web service** variable
for the new Web Server, PHP-FPM, per-service and correlation sections.

## Coverage audit in 0.26.0

The exporter represents all scalar fields currently sampled by `diagnostics::View` when available. The per-interface `network_interfaces` list added in 0.18.6 is intentionally dashboard-only so kernel-created interface names cannot create an uncontrolled Prometheus label set. RustProxy also exports the bounded operational data already available from traffic metrics, cache, vhosts, backend pools, streams, logging, GeoIP and ACME renewal observations.

| Area | Exported diagnostics |
|---|---|
| Global traffic | requests/completions/active/rejects/errors, status classes, request/response bytes, exchange lifetime, accepted/active/rejected connections |
| Protocols | HTTP/1.1, HTTP/2 and HTTP/3 requests, H2/H3 connections/rejects, TLS connections/handshake failures, WebSocket tunnels/bytes/errors/lifetime, compression counters |
| Admin | request/status counters and time to control-plane response headers |
| Process/host | diagnostic source, runtime platform info, sampled timestamp, system/process CPU, logical CPUs, RAM total/used, system swap total/used, system uptime, RSS total/anon/file/shmem, virtual memory/limit, VmData, process swap, FDs/limit, threads, process start time, load 1/5/15, network totals/rates, process disk I/O, cgroup memory current/max |
| GeoIP | loaded state plus IPv4/IPv6 range counts |
| Cache | hot entries/bytes, global hit ratio and counters, cold entries/bytes and movement/error counters, fill bytes, cold queue bytes/items, active cold read/write bytes, combined transient bytes, configured-domain hot/cold occupancy, hit ratio and decision data |
| Vhosts | desired enable/domain/path/cache settings, per-vhost traffic/status/error/byte counters and cumulative exchange-duration histogram |
| Traffic analytics | independent live collector (`include_traffic`): capture-state gauges, per-vhost request/error/body-byte/duration totals; approximate bounded top countries; fixed User-Agent families; fixed HTTP method families; status classes; cache outcomes; serving source (`web_server`, `reverse_proxy`, `local_response`, `unresolved`); and currently configured route IDs |
| Built-in Web Server | configured Web Service count/info, PHP and directory-listing flags, vhost/route source metadata, plus actual request traffic through the shared vhost counters and Traffic serving-source metrics |
| Redirect/404 hosts | desired enable state, domain/rule counts and local-response status |
| Backends | pool readiness/policy info, bounded backend status, health/drain/backup, active work and configured request-concurrency/queue limits, requests/errors/bytes, weight, separate active/passive failure state, active/passive successes and health-failure totals, unconditional HTTP-5xx/timeout/connect/I/O/slow-response observation counters, DNS/TLS/header-timeout/read-timeout/send-timeout diagnostics, capacity rejection/queue-timeout counters, retry attempt/success totals, upstream-client generation/current-generation requests/generation age/rotation totals, plus `passive_http_5xx_total`, `passive_timeouts_total`, `passive_connect_errors_total`, `passive_io_errors_total` and `passive_slow_total` aliases, health transitions, last status, current/EWMA latency, last-probe/failure/ejection/recovery timestamps and traffic rate for ordinary vhosts |
| Docker vhosts | separate `rust_proxy_docker_vhost_*` and `rust_proxy_docker_backend_*` families: configured agents/assignments/domains, vhost traffic/status/duration plus backend readiness, health, requests/errors/bytes, latency/EWMA, separate active/passive failure state, unconditional observation counters and matching passive-health aliases, retry attempt/success totals, upstream-client generation/current-generation requests/generation age/rotation totals, transitions and probe results |
| Streams | desired enable state, active/session/reject/error counters, bytes/packets and max connections |
| Logging | structured sink queue capacity, sent/error/drop counters plus the global live-log contention drop counter |
| TLS/ACME | configured certificate count, auto-renew flag, bounded issuer metadata, latest ACME job state, renewal check/expiry timestamps, failure count and status |
| Inventory | listeners/protocol flags, vhost/redirect/404/stream/ACL counts, request capacity and available permits, config revision and restart-required state |

Values unavailable on the running OS are omitted rather than fabricated. Series
labels are bounded by configured IDs/domains, finite states, fixed protocol/status/UA/method/source
families, or the finite two-letter country space. Per-request URLs, raw client IPs, full
User-Agent values and arbitrary Host input are never labels.

This is complete for **currently implemented bounded diagnostic views**. It does
not intentionally expose secrets, raw configuration documents, access-log entries
or unbounded per-request values merely to make them visible to Prometheus.

## Prometheus examples

Loopback/private CIDR scrape without a token when the explicit trusted option is
used:

```yaml
scrape_configs:
  - job_name: rustproxy
    scrape_interval: 15s
    metrics_path: /metrics
    static_configs:
      - targets: ['127.0.0.1:9191']
```

With bearer authentication:

```yaml
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/secrets/rustproxy-token
```

Vhost rate:

```promql
sum by (vhost) (rate(rust_proxy_vhost_requests_total[5m]))
```

Web Server traffic rate:

```promql
sum by (vhost) (rate(rust_proxy_traffic_source_requests_total{source="web_server"}[5m]))
```

Top country snapshot from bounded Traffic analytics:

```promql
sum by (country) (rust_proxy_traffic_country_requests)
```

Country series are populated when **Settings -> Logging -> Capture country** is enabled; User-Agent family series follow **Capture User-Agent**. The exporter never turns those request-path captures on by itself.
Current capture state is exposed as `rust_proxy_traffic_country_capture_enabled` and `rust_proxy_traffic_user_agent_capture_enabled`. Configured-route snapshots use `rust_proxy_traffic_route_requests`; raw URL paths are never exported as labels.

Fixed User-Agent families:

```promql
sum by (family) (rust_proxy_traffic_user_agent_family_requests_total)
```

Vhost p95 exchange lifetime:

```promql
histogram_quantile(0.95,
  sum by (vhost, le) (rate(rust_proxy_vhost_exchange_duration_seconds_bucket[5m])))
```

Docker-vhost request rate (kept separate from ordinary vhosts):

```promql
sum by (agent, vhost, service) (rate(rust_proxy_docker_vhost_requests_total[5m]))
```

Docker backend availability and latency:

```promql
rust_proxy_docker_backend_up
rust_proxy_docker_backend_ewma_latency_seconds
```

Backend client lifecycle and retry diagnostics:

```promql
rust_proxy_backend_upstream_client_generation_requests
rust_proxy_backend_upstream_client_generation_age_seconds
rate(rust_proxy_backend_upstream_client_rotations_total[5m])
rate(rust_proxy_backend_retry_attempts_total[5m])
rate(rust_proxy_backend_retry_successes_total[5m])
```

The same fields are exported for Docker-discovered backends under the `rust_proxy_docker_backend_*` prefix.

Certificate expiry window:

```promql
rust_proxy_certificate_renewal_not_after_timestamp_seconds - time()
```

The supplied dashboard is `grafana/rustproxy-overview.json`; Debian packages also
install it under `/usr/share/rustproxy/grafana/`. Its dedicated **Docker vhosts** row
uses only the separate Docker metric families and provides Docker-agent/service
filters so multi-agent installations can be inspected independently from ordinary
vhosts/backends.

Reference: https://prometheus.io/docs/instrumenting/exposition_formats/

## Stream pool metrics (0.32.0)

The existing global `include_streams` controls all Stream Prometheus families. Each configured Stream may set nullable `metrics_enabled` (default true); each backend may set its own nullable `metrics_enabled` (inherit from Stream). Detailed backend counters are emitted only when both flags are enabled. Basic per-Stream counters, backend active/drain and numeric health-state gauges remain observable with global Stream export enabled; routing and health monitoring never depend on these flags. Backend series use only configured `stream`, `backend` and `protocol` labels, never client IP or arbitrary UDP payload. `rust_proxy_stream_backend_health_state` encodes unverified=0, checking=1, healthy=2, unhealthy=3, disabled=4. Optional detailed families include bytes/packets, assignments, errors/rejects, sessions and health successes/failures. All counters reset on process startup.

For v0.32.0 Stream pools, backend-level `metrics_enabled` overrides the Stream default; missing/null values inherit it. Detailed backend accounting defaults off, including for legacy `upstream` configurations. Global `include_streams` independently gates all Stream Prometheus export.