RustProxy/docs/MONITORING.md

Monitoring, metrics and health

RustProxy has operational views in the authenticated panel and an optional standalone Prometheus exporter. They measure different things: dashboard sampling, process-lifetime counters and stored logs are not interchangeable.

What to watch

View	Main information	Interpretation
Overview	Active connections, request rate, errors, process CPU/RAM, protocols, backend state, optional charts	Dashboard samples are not long-term billing data.
Vhosts → Backends	Member health, weights, H1/H2 modes, current work, last observed upstream response	A last_status from normal proxy traffic is not an active health probe result.
Web Server	Requests/status classes, static/PHP traffic, response-header time, response bytes, PHP concurrency permits	Response header duration excludes streamed body transfer. PHP permits are not FPM child processes.
Traffic	Per-Vhost totals and top countries/IPs/paths/methods/agents/cache results	Bounded in-memory analytics, approximate top lists and possible sample loss under heavy load.
Cache	HOT/COLD occupancy, hit/miss/bypass, object metadata, selected invalidation	HTTP Age differs from cache residence age.
Docker / Streams	Agent status, dynamic Vhost health, TCP/UDP sessions and traffic	An agent can be waiting for first observation; Streams are non-HTTP.
Logs	Live requests, stored JSONL/error/protocol records	A log sink can be disabled while live memory view remains available.

Overview allows account-specific card layout and history controls; see Dashboard. GeoIP/country analytics depend on the installed/update database and whether country capture is enabled.

Settings → System information shows the operating-system/hardware snapshot and Statistics source. The old dedicated process-information block is no longer displayed there; this is a panel presentation change only. Process CPU/RAM and related measurements continue to be available in the Overview and authenticated metrics APIs where supported. Settings → Appearance → Data refresh stores the per-account refresh intervals; switching off automatic refresh does not disable manual refresh.

Prometheus exporter

The exporter is disabled by default and uses a separate HTTP listener, normally 127.0.0.1:9191, at /metrics. It is not protected by the admin browser cookie; it has its own client-network and bearer-token policy. In Settings → Prometheus enable the endpoint and configure:

In 0.34.1, use the separate Prometheus token controls in Settings to generate, rotate or revoke a 32+ character Bearer credential. Its clear value appears only once; copy it directly into your monitoring secrets store. The database keeps only its hash, and rotation/revocation takes effect without reprinting the previous secret. The older environment-variable credential remains supported as a fallback only until a generated credential or explicit revocation entry exists. Setting a new token does not reconfigure the exporter listener or its enable switch.

• listen, path, allowed_cidrs and an optional environment-variable name bearer_token_env (the secret value is supplied separately to the service).
• allow_trusted_network_without_token only for a deliberately restricted monitoring subnet. /0 CIDRs are not accepted in this mode.
• Collector switches for process/system, Vhosts, traffic, backends, Docker, cache domains, Streams and logging.

A non-loopback exporter requires a valid generated or environment Bearer token at scrape time, or an explicit token-free trusted-network exception; otherwise requests return 401. The exporter uses plain HTTP; keep it on loopback, trusted network or a protected TLS tunnel. Forwarded/X-Forwarded-For headers cannot grant exporter access because network restrictions apply to the transport peer.

For legacy environment-token authentication, provide a 32–4096-character printable non-whitespace ASCII token in a service environment variable and set bearer_token_env to its name, e.g. RUST_PROXY_METRICS_TOKEN. The generated-token workflow above avoids this environment variable. Changing exporter activation/bind/token-variable usually needs restart; some collector/CIDR settings apply live. The exact allowed GET path must not have a query string.

Example scrape configuration, after choosing the correct private target:

scrape_configs:
  - job_name: rustproxy
    static_configs:
      - targets: ['127.0.0.1:9191']

When using a bearer token, configure Prometheus authorization or bearer_token_file securely rather than embedding a clear token in an exported dashboard. Review grafana/README.md and grafana/rustproxy-overview.json for chart examples; nagios/README.md contains external check scripts.

JSON metrics API for external dashboards

RustProxy also offers a plain, read-only JSON snapshot. It does not use SSE, WebSockets or persistent subscriptions: clients poll using ordinary GET (for example, every 2–5 seconds).

Endpoint	Where	Authentication
GET /api/metrics/json	Administration listener (default 127.0.0.1:9090)	Admin session or a General API ro/rw Bearer token from Settings → Security, restricted by allowed CIDRs
GET /api/metrics/summary	Administration listener, compact calculated metrics only	Same admin session/General API Bearer credentials and CIDR policy
GET /metrics.json	Dedicated exporter (default 127.0.0.1:9191, only when Prometheus is enabled)	Same allowed CIDRs and exporter Bearer token as /metrics; not an admin/API token
GET /metrics.summary.json	Dedicated exporter, compact calculated metrics only	Same exporter Bearer token/CIDR policy, if enabled

When the exporter path differs from /metrics, the JSON routes append .json and .summary.json to that path (e.g. /private/stats → /private/stats.json and /private/stats.summary.json). Exporter routes reject query strings. Every JSON response uses application/json and Cache-Control: no-store; do not expose these endpoints publicly.

For a dedicated integration, issue a General API token with read-only permission and a narrow allowed_cidrs set. Example from a permitted client:

curl -fsS -H "Authorization: Bearer $RUSTPROXY_API_TOKEN" \
  http://127.0.0.1:9090/api/metrics/json | jq '{vhosts: .vhosts | {total,enabled,disabled}, requests: .requests.total, cpu: .system.cpu_percent, ram_bytes: .system.process_rss_bytes}'

Summary: ready for dashboard cards

For the shortest response, poll /api/metrics/summary (or the standalone exporter's .summary.json sibling) instead of loading every Prometheus sample. The full /api/metrics/json includes the same calculated object at summary. The response groups all values into vhosts, backends, traffic, system, cache, web_server, streams and capacity, plus a collection timestamp.

curl -fsS -H "Authorization: Bearer $RUSTPROXY_API_TOKEN" \
  http://127.0.0.1:9090/api/metrics/summary | \
  jq '{working_vhosts: .vhosts.serving_estimate, backends: .backends.total, \
       backend_up: .backends.up, rps: .traffic.requests_per_second, \
       traffic_mb: .traffic.megabytes_total, cpu: .system.cpu_percent, \
       ram_percent: .system.memory_percent}'

• vhosts.enabled is the configured on/off flag. vhosts.serving_estimate counts enabled vhosts using a ready backend pool or an embedded Web Server. It is not an HTTP probe or guarantee of reachability; redirect hosts, not-found hosts and Docker assignments are counted separately. Only enabled vhosts with a compiled pool are included in backend_ready / backend_not_ready.
• backends.total counts live compiled pool members including Docker. configured_static counts saved static members. up, down, checking, draining, disabled, unknown, pools_ready and active_exchanges are independently calculated; traffic_bytes_per_second sums moving backend rates and does not represent all public HTTP traffic.
• traffic.requests_total, bytes_in_total, bytes_out_total and bytes_total are cumulative process counters; megabytes_total uses decimal MB (10⁶ B). requests_per_second, errors_per_second, bytes_in_per_second, bytes_out_per_second and bytes_per_second are rolling poll interval rates computed on the server from a shared bounded sample (minimum 1 s, maximum 30 s between usable samples). An initial poll, restart, counter reset or gap longer than 30 s yields null until a second valid poll. Very rapid simultaneous polls reuse the latest completed interval. These are not guaranteed exact one-second rolling averages; poll every 2–5 seconds. requests_per_second_average is a separate uptime average.
• traffic.error_percent is lifetime errors divided by completed requests; cache.hit_percent is lifetime HOT/COLD hit ratio from cache metrics. Zero-denominator percentages are null unless a cache ratio has a defined default. The traffic.network_rx_bytes_per_second and network_tx_bytes_per_second fields are host interface rates, not RustProxy-exclusive transfer; backend rate is separate.
• system.cpu_percent, process_cpu_percent, memory_percent, RAM and RSS reflect the last OS sample (system.sampled_at_ms). Unsupported values can be null. web_server.requests_total adds observed Web Server responses, while streams.active_sessions adds observed Layer-4 sessions.

The JSON is organized for simple chart consumption:

• summary contains the small, calculated metrics object described above.
• vhosts includes total, enabled, disabled, backend readiness and an array of per-Vhost counters.
• requests includes process-lifetime totals, active exchanges, errors, status classes, byte counters, latency, active connections and H1/H2/H3/WebSockets.
• system includes sampled host/process CPU, memory, swap, process I/O/FDs, network interfaces and throughput.
• backends, web_server, streams, docker, cache, traffic, transform, logging and capacity contain their corresponding operational summaries.
• prometheus.types and prometheus.samples include every numeric sample exported by the existing Prometheus text renderer, with label pairs as JSON objects. Histogram buckets/counts/sums are included. The normal Prometheus collector switches still control this detailed section.

Example field access:

const snapshot = await (await fetch('/api/metrics/json')).json();
const vhostsEnabled = snapshot.vhosts.enabled;
const requestTotal = snapshot.requests.total;
const cpuPercent = snapshot.system.cpu_percent; // can be null before sampling
const samples = snapshot.prometheus.samples.rust_proxy_vhost_requests_total ?? [];
// samples[0] -> { labels: { vhost: 'site1' }, value: 123 }

collected_at_ms is the JSON snapshot time; system.sampled_at_ms is the last OS collector sample time (the OS collector runs in the background). Raw counters represent cumulative totals, not instantaneous RPS. Use the server-computed interval rates from summary or calculate (new_total - old_total) / elapsed_seconds from raw counters in the client; reset the baseline after counter decreases, process restart or config reset. Absent/unsupported OS values are null; non-finite Prometheus values are also represented as null. Traffic breakdowns are approximate and can lose samples under load. The JSON excludes raw request paths, client IP addresses and raw User-Agent values, matching the bounded-label privacy model of Prometheus.

Metric privacy and limits

Prometheus series use bounded configured labels; arbitrary request URLs, raw client IPs, credentials, raw User-Agents and query strings are intentionally not exposed as unbounded labels. Dashboard traffic top lists can still contain sensitive operational metadata, so keep the panel authenticated.

Metrics reset on process replacement, and some per-Vhost/backend records can reset when identity or config changes. Distinguish counters, rates and gauges; watch for traffic sampling gaps rather than interpreting every missing point as zero. Before deployment, verify collector reachability, firewall rules, CPU/RAM overhead and alert thresholds against real traffic.