242 lines
14 KiB
Markdown
242 lines
14 KiB
Markdown
# Dedicated Prometheus endpoint
|
|
|
|
RustProxy exposes Prometheus text exposition 0.0.4 on a dedicated read-only
|
|
listener. It is independent from administrator browser sessions and does not
|
|
export raw client IPs, request paths, query strings, full User-Agent strings, cookies,
|
|
credentials, backend URLs or ACME account secrets.
|
|
|
|
## Configuration
|
|
|
|
```json
|
|
{
|
|
"prometheus": {
|
|
"enabled": false,
|
|
"listen": "127.0.0.1:9191",
|
|
"path": "/metrics",
|
|
"allowed_cidrs": ["127.0.0.0/8", "::1/128"],
|
|
"bearer_token_env": null,
|
|
"allow_trusted_network_without_token": false,
|
|
"include_system": true,
|
|
"include_vhosts": true,
|
|
"include_traffic": true,
|
|
"include_backends": true,
|
|
"include_docker_vhosts": true,
|
|
"include_cache_domains": true,
|
|
"include_streams": true,
|
|
"include_logging": true
|
|
}
|
|
}
|
|
```
|
|
|
|
The listener is plain HTTP. Keep it on loopback, a private monitoring network or
|
|
behind a protected tunnel. `allowed_cidrs` is always evaluated against the actual
|
|
transport peer (`PeerAddr`); `Forwarded` and `X-Forwarded-For` never grant metrics
|
|
access.
|
|
|
|
### Bearer token
|
|
|
|
Set `bearer_token_env` to an environment-variable name such as
|
|
`RUST_PROXY_METRICS_TOKEN`. RustProxy stores only the variable name. The value must
|
|
contain 32..4096 printable non-whitespace ASCII characters and must be present at
|
|
startup.
|
|
|
|
### Trusted network without a token
|
|
|
|
`allow_trusted_network_without_token=true` explicitly allows a peer already matched
|
|
by `allowed_cidrs` to scrape without a bearer token. This can be used for a private
|
|
Prometheus VLAN. An explicit IPv4/IPv6 `/0` is rejected in this mode. Keep the CIDR
|
|
list narrow; enabling this option is an authentication decision, not a forwarded-IP
|
|
feature.
|
|
|
|
A non-loopback exporter requires either a bearer-token variable or this explicit
|
|
trusted-network bypass. Changing enabled/listen/token-variable requires restart;
|
|
path, CIDRs, collector switches and the trusted-network bypass apply live.
|
|
|
|
Only `GET` on the exact configured path without a query string is accepted. A
|
|
wrong path is `404`, wrong method `405`, disallowed peer `403`, invalid/missing
|
|
required bearer `401`, and exporter concurrency exhaustion `429`.
|
|
|
|
## Web Server and PHP-FPM metrics (0.33.0)
|
|
|
|
Prometheus already exports core Web Server/PHP response counters by configured
|
|
`web_service` ID. The 0.33.0 additions below are enabled with the existing
|
|
Prometheus exporter (no new listener, credentials, options or open routes).
|
|
|
|
| Family | Type | Labels | Meaning |
|
|
|---|---|---|---|
|
|
| `rust_proxy_web_server_requests_total` | counter | `web_service` | Header-ready WWW responses, including PHP |
|
|
| `rust_proxy_web_server_responses_total` | counter | `web_service`, `class` | HTTP 2xx/3xx/4xx/5xx |
|
|
| `rust_proxy_web_server_response_header_duration_microseconds_total` | counter | `web_service` | Cumulative time until headers |
|
|
| `rust_proxy_web_server_requests_active` | gauge | `web_service` | Requests still awaiting headers |
|
|
| `rust_proxy_web_server_request_kind_total` | counter | `web_service`, `kind` | `static`, `php`, `other` handler |
|
|
| `rust_proxy_web_server_methods_total` | counter | `web_service`, `method` | `GET`, `POST`, `PUT`, `DELETE`, `other` |
|
|
| `rust_proxy_web_server_status_code_total` | counter | `web_service`, `status` | Selected 200/206/304/403/404/500/502/503 |
|
|
| `rust_proxy_web_server_response_body_bytes_total` | counter | `web_service` | Body bytes delivered before final compression |
|
|
| `rust_proxy_web_server_compression_total` | counter | `web_service`, `encoding` | Downstream gzip/br/zstd/deflate operations |
|
|
| `rust_proxy_php_requests_total` | counter | `web_service` | PHP response headers processed |
|
|
| `rust_proxy_php_responses_total` | counter | `web_service`, `class` | PHP response status classes |
|
|
| `rust_proxy_php_response_header_duration_microseconds_total` | counter | `web_service` | Cumulative PHP time until headers |
|
|
| `rust_proxy_php_response_body_bytes_total` | counter | `web_service` | PHP body bytes emitted before compression |
|
|
| `rust_proxy_php_concurrency_available` | gauge | `web_service` | Free RustProxy PHP request permits |
|
|
| `rust_proxy_php_concurrency_limit` | gauge | `web_service` | Configured concurrent PHP request limit |
|
|
| `rust_proxy_php_concurrency_in_use` | gauge | `web_service` | Permits currently held by PHP requests |
|
|
| `rust_proxy_php_failures_total` | counter | `web_service`, `reason` | Fixed FastCGI/PHP failure classes |
|
|
|
|
PHP `reason` is one of `capacity`, `connect_timeout`, `connect_error`,
|
|
`request_timeout`, `request_io`, `response_timeout`, `response_io`,
|
|
`invalid_response`, `body_io`. PHP concurrency is the admission semaphore in
|
|
RustProxy, not external PHP worker health or PHP-FPM child count. Status-code
|
|
labels are deliberately restricted to a stable, useful subset; arbitrary code,
|
|
request paths, IP addresses and untrusted method strings are not labels.
|
|
The exported byte counters are measured **before the final dynamic response
|
|
encoder**, so treat them as payload sizes rather than measured network transfer.
|
|
Precompressed static file delivery is not counted in the downstream compression
|
|
family. All time-to-headers metrics exclude streamed transfer duration.
|
|
|
|
Useful Grafana/PromQL examples:
|
|
|
|
```promql
|
|
sum(rate(rust_proxy_web_server_requests_total[5m]))
|
|
sum by (class) (rate(rust_proxy_web_server_responses_total[5m]))
|
|
sum by (kind) (rate(rust_proxy_web_server_request_kind_total[5m]))
|
|
100 * sum(rust_proxy_php_concurrency_in_use) / clamp_min(sum(rust_proxy_php_concurrency_limit), 1)
|
|
1e-3 * sum(rate(rust_proxy_web_server_response_header_duration_microseconds_total[5m])) / clamp_min(sum(rate(rust_proxy_web_server_requests_total[5m])), 1e-9)
|
|
sum by (reason) (rate(rust_proxy_php_failures_total[5m]))
|
|
```
|
|
|
|
Import `grafana/rustproxy-overview.json` and use the **Web service** variable
|
|
for the new Web Server, PHP-FPM, per-service and correlation sections.
|
|
|
|
## Coverage audit in 0.26.0
|
|
|
|
The exporter represents all scalar fields currently sampled by `diagnostics::View` when available. The per-interface `network_interfaces` list added in 0.18.6 is intentionally dashboard-only so kernel-created interface names cannot create an uncontrolled Prometheus label set. RustProxy also exports the bounded operational data already available from traffic metrics, cache, vhosts, backend pools, streams, logging, GeoIP and ACME renewal observations.
|
|
|
|
| Area | Exported diagnostics |
|
|
|---|---|
|
|
| Global traffic | requests/completions/active/rejects/errors, status classes, request/response bytes, exchange lifetime, accepted/active/rejected connections |
|
|
| Protocols | HTTP/1.1, HTTP/2 and HTTP/3 requests, H2/H3 connections/rejects, TLS connections/handshake failures, WebSocket tunnels/bytes/errors/lifetime, compression counters |
|
|
| Admin | request/status counters and time to control-plane response headers |
|
|
| Process/host | diagnostic source, runtime platform info, sampled timestamp, system/process CPU, logical CPUs, RAM total/used, system swap total/used, system uptime, RSS total/anon/file/shmem, virtual memory/limit, VmData, process swap, FDs/limit, threads, process start time, load 1/5/15, network totals/rates, process disk I/O, cgroup memory current/max |
|
|
| GeoIP | loaded state plus IPv4/IPv6 range counts |
|
|
| Cache | hot entries/bytes, global hit ratio and counters, cold entries/bytes and movement/error counters, fill bytes, cold queue bytes/items, active cold read/write bytes, combined transient bytes, configured-domain hot/cold occupancy, hit ratio and decision data |
|
|
| Vhosts | desired enable/domain/path/cache settings, per-vhost traffic/status/error/byte counters and cumulative exchange-duration histogram |
|
|
| Traffic analytics | independent live collector (`include_traffic`): capture-state gauges, per-vhost request/error/body-byte/duration totals; approximate bounded top countries; fixed User-Agent families; fixed HTTP method families; status classes; cache outcomes; serving source (`web_server`, `reverse_proxy`, `local_response`, `unresolved`); and currently configured route IDs |
|
|
| Built-in Web Server | configured Web Service count/info, PHP and directory-listing flags, vhost/route source metadata, plus actual request traffic through the shared vhost counters and Traffic serving-source metrics |
|
|
| Redirect/404 hosts | desired enable state, domain/rule counts and local-response status |
|
|
| Backends | pool readiness/policy info, bounded backend status, health/drain/backup, active work and configured request-concurrency/queue limits, requests/errors/bytes, weight, separate active/passive failure state, active/passive successes and health-failure totals, unconditional HTTP-5xx/timeout/connect/I/O/slow-response observation counters, DNS/TLS/header-timeout/read-timeout/send-timeout diagnostics, capacity rejection/queue-timeout counters, retry attempt/success totals, upstream-client generation/current-generation requests/generation age/rotation totals, plus `passive_http_5xx_total`, `passive_timeouts_total`, `passive_connect_errors_total`, `passive_io_errors_total` and `passive_slow_total` aliases, health transitions, last status, current/EWMA latency, last-probe/failure/ejection/recovery timestamps and traffic rate for ordinary vhosts |
|
|
| Docker vhosts | separate `rust_proxy_docker_vhost_*` and `rust_proxy_docker_backend_*` families: configured agents/assignments/domains, vhost traffic/status/duration plus backend readiness, health, requests/errors/bytes, latency/EWMA, separate active/passive failure state, unconditional observation counters and matching passive-health aliases, retry attempt/success totals, upstream-client generation/current-generation requests/generation age/rotation totals, transitions and probe results |
|
|
| Streams | desired enable state, active/session/reject/error counters, bytes/packets and max connections |
|
|
| Logging | structured sink queue capacity, sent/error/drop counters plus the global live-log contention drop counter |
|
|
| TLS/ACME | configured certificate count, auto-renew flag, bounded issuer metadata, latest ACME job state, renewal check/expiry timestamps, failure count and status |
|
|
| Inventory | listeners/protocol flags, vhost/redirect/404/stream/ACL counts, request capacity and available permits, config revision and restart-required state |
|
|
|
|
Values unavailable on the running OS are omitted rather than fabricated. Series
|
|
labels are bounded by configured IDs/domains, finite states, fixed protocol/status/UA/method/source
|
|
families, or the finite two-letter country space. Per-request URLs, raw client IPs, full
|
|
User-Agent values and arbitrary Host input are never labels.
|
|
|
|
This is complete for **currently implemented bounded diagnostic views**. It does
|
|
not intentionally expose secrets, raw configuration documents, access-log entries
|
|
or unbounded per-request values merely to make them visible to Prometheus.
|
|
|
|
## Prometheus examples
|
|
|
|
Loopback/private CIDR scrape without a token when the explicit trusted option is
|
|
used:
|
|
|
|
```yaml
|
|
scrape_configs:
|
|
- job_name: rustproxy
|
|
scrape_interval: 15s
|
|
metrics_path: /metrics
|
|
static_configs:
|
|
- targets: ['127.0.0.1:9191']
|
|
```
|
|
|
|
With bearer authentication:
|
|
|
|
```yaml
|
|
authorization:
|
|
type: Bearer
|
|
credentials_file: /etc/prometheus/secrets/rustproxy-token
|
|
```
|
|
|
|
Vhost rate:
|
|
|
|
```promql
|
|
sum by (vhost) (rate(rust_proxy_vhost_requests_total[5m]))
|
|
```
|
|
|
|
Web Server traffic rate:
|
|
|
|
```promql
|
|
sum by (vhost) (rate(rust_proxy_traffic_source_requests_total{source="web_server"}[5m]))
|
|
```
|
|
|
|
Top country snapshot from bounded Traffic analytics:
|
|
|
|
```promql
|
|
sum by (country) (rust_proxy_traffic_country_requests)
|
|
```
|
|
|
|
Country series are populated when **Settings -> Logging -> Capture country** is enabled; User-Agent family series follow **Capture User-Agent**. The exporter never turns those request-path captures on by itself.
|
|
Current capture state is exposed as `rust_proxy_traffic_country_capture_enabled` and `rust_proxy_traffic_user_agent_capture_enabled`. Configured-route snapshots use `rust_proxy_traffic_route_requests`; raw URL paths are never exported as labels.
|
|
|
|
Fixed User-Agent families:
|
|
|
|
```promql
|
|
sum by (family) (rust_proxy_traffic_user_agent_family_requests_total)
|
|
```
|
|
|
|
Vhost p95 exchange lifetime:
|
|
|
|
```promql
|
|
histogram_quantile(0.95,
|
|
sum by (vhost, le) (rate(rust_proxy_vhost_exchange_duration_seconds_bucket[5m])))
|
|
```
|
|
|
|
Docker-vhost request rate (kept separate from ordinary vhosts):
|
|
|
|
```promql
|
|
sum by (agent, vhost, service) (rate(rust_proxy_docker_vhost_requests_total[5m]))
|
|
```
|
|
|
|
Docker backend availability and latency:
|
|
|
|
```promql
|
|
rust_proxy_docker_backend_up
|
|
rust_proxy_docker_backend_ewma_latency_seconds
|
|
```
|
|
|
|
Backend client lifecycle and retry diagnostics:
|
|
|
|
```promql
|
|
rust_proxy_backend_upstream_client_generation_requests
|
|
rust_proxy_backend_upstream_client_generation_age_seconds
|
|
rate(rust_proxy_backend_upstream_client_rotations_total[5m])
|
|
rate(rust_proxy_backend_retry_attempts_total[5m])
|
|
rate(rust_proxy_backend_retry_successes_total[5m])
|
|
```
|
|
|
|
The same fields are exported for Docker-discovered backends under the `rust_proxy_docker_backend_*` prefix.
|
|
|
|
Certificate expiry window:
|
|
|
|
```promql
|
|
rust_proxy_certificate_renewal_not_after_timestamp_seconds - time()
|
|
```
|
|
|
|
The supplied dashboard is `grafana/rustproxy-overview.json`; Debian packages also
|
|
install it under `/usr/share/rustproxy/grafana/`. Its dedicated **Docker vhosts** row
|
|
uses only the separate Docker metric families and provides Docker-agent/service
|
|
filters so multi-agent installations can be inspected independently from ordinary
|
|
vhosts/backends.
|
|
|
|
Reference: https://prometheus.io/docs/instrumenting/exposition_formats/
|
|
|
|
## Stream pool metrics (0.32.0)
|
|
|
|
The existing global `include_streams` controls all Stream Prometheus families. Each configured Stream may set nullable `metrics_enabled` (default true); each backend may set its own nullable `metrics_enabled` (inherit from Stream). Detailed backend counters are emitted only when both flags are enabled. Basic per-Stream counters, backend active/drain and numeric health-state gauges remain observable with global Stream export enabled; routing and health monitoring never depend on these flags. Backend series use only configured `stream`, `backend` and `protocol` labels, never client IP or arbitrary UDP payload. `rust_proxy_stream_backend_health_state` encodes unverified=0, checking=1, healthy=2, unhealthy=3, disabled=4. Optional detailed families include bytes/packets, assignments, errors/rejects, sessions and health successes/failures. All counters reset on process startup.
|
|
|
|
For v0.32.0 Stream pools, backend-level `metrics_enabled` overrides the Stream default; missing/null values inherit it. Detailed backend accounting defaults off, including for legacy `upstream` configurations. Global `include_streams` independently gates all Stream Prometheus export.
|