# Monitoring, metrics and health

RustProxy has operational views in the authenticated panel and an optional standalone Prometheus exporter. They measure different things: dashboard sampling, process-lifetime counters and stored logs are not interchangeable.

## What to watch

| View | Main information | Interpretation |
|---|---|---|
| **Overview** | Active connections, request rate, errors, process CPU/RAM, protocols, backend state, optional charts | Dashboard samples are not long-term billing data. |
| **Vhosts → Backends** | Member health, weights, H1/H2 modes, current work, last observed upstream response | A `last_status` from normal proxy traffic is not an active health probe result. |
| **Web Server** | Requests/status classes, static/PHP traffic, response-header time, response bytes, PHP concurrency permits | Response header duration excludes streamed body transfer. PHP permits are not FPM child processes. |
| **Traffic** | Per-Vhost totals and top countries/IPs/paths/methods/agents/cache results | Bounded in-memory analytics, approximate top lists and possible sample loss under heavy load. |
| **Cache** | HOT/COLD occupancy, hit/miss/bypass, object metadata, selected invalidation | HTTP `Age` differs from cache residence age. |
| **Docker / Streams** | Agent status, dynamic Vhost health, TCP/UDP sessions and traffic | An agent can be waiting for first observation; Streams are non-HTTP. |
| **Logs** | Live requests, stored JSONL/error/protocol records | A log sink can be disabled while live memory view remains available. |

Overview allows account-specific card layout and history controls; see [Dashboard](DASHBOARD.md). GeoIP/country analytics depend on the installed/update database and whether country capture is enabled.

**Settings → System information** shows the operating-system/hardware snapshot and **Statistics source**. The old dedicated process-information block is no longer displayed there; this is a panel presentation change only. Process CPU/RAM and related measurements continue to be available in the Overview and authenticated metrics APIs where supported. **Settings → Appearance → Data refresh** stores the per-account refresh intervals; switching off automatic refresh does not disable manual refresh.

## Prometheus exporter

The exporter is **disabled by default** and uses a separate HTTP listener, normally `127.0.0.1:9191`, at `/metrics`. It is not protected by the admin browser cookie; it has its **own** client-network and bearer-token policy. In **Settings → Prometheus** enable the endpoint and configure:

In 0.34.1, use the separate **Prometheus token** controls in Settings to generate, rotate or revoke a 32+ character Bearer credential. Its clear value appears only once; copy it directly into your monitoring secrets store. The database keeps only its hash, and rotation/revocation takes effect without reprinting the previous secret. The older environment-variable credential remains supported as a fallback *only until* a generated credential or explicit revocation entry exists. Setting a new token does not reconfigure the exporter listener or its enable switch.


- `listen`, `path`, `allowed_cidrs` and an optional environment-variable **name** `bearer_token_env` (the secret value is supplied separately to the service).
- `allow_trusted_network_without_token` only for a deliberately restricted monitoring subnet. `/0` CIDRs are not accepted in this mode.
- Collector switches for process/system, Vhosts, traffic, backends, Docker, cache domains, Streams and logging.

A non-loopback exporter requires a valid generated or environment Bearer token at scrape time, or an explicit token-free trusted-network exception; otherwise requests return 401. The exporter uses plain HTTP; keep it on loopback, trusted network or a protected TLS tunnel. `Forwarded`/`X-Forwarded-For` headers cannot grant exporter access because network restrictions apply to the transport peer.

For legacy environment-token authentication, provide a 32–4096-character printable non-whitespace ASCII token in a **service environment variable** and set `bearer_token_env` to its name, e.g. `RUST_PROXY_METRICS_TOKEN`. The generated-token workflow above avoids this environment variable. Changing exporter activation/bind/token-variable usually needs restart; some collector/CIDR settings apply live. The exact allowed GET path must not have a query string.

Example scrape configuration, after choosing the correct private target:

```yaml
scrape_configs:
  - job_name: rustproxy
    static_configs:
      - targets: ['127.0.0.1:9191']
```

When using a bearer token, configure Prometheus `authorization` or `bearer_token_file` securely rather than embedding a clear token in an exported dashboard. Review `grafana/README.md` and `grafana/rustproxy-overview.json` for chart examples; `nagios/README.md` contains external check scripts.

## JSON metrics API for external dashboards

RustProxy also offers a **plain, read-only JSON snapshot**. It does not use SSE,
WebSockets or persistent subscriptions: clients poll using ordinary `GET` (for
example, every 2–5 seconds).

| Endpoint | Where | Authentication |
|---|---|---|
| `GET /api/metrics/json` | Administration listener (default `127.0.0.1:9090`) | Admin session or a **General API** `ro`/`rw` Bearer token from **Settings → Security**, restricted by allowed CIDRs |
| `GET /api/metrics/summary` | Administration listener, compact calculated metrics only | **Same** admin session/General API Bearer credentials and CIDR policy |
| `GET /metrics.json` | Dedicated exporter (default `127.0.0.1:9191`, **only when Prometheus is enabled**) | Same allowed CIDRs and exporter Bearer token as `/metrics`; not an admin/API token |
| `GET /metrics.summary.json` | Dedicated exporter, compact calculated metrics only | **Same** exporter Bearer token/CIDR policy, if enabled |

When the exporter `path` differs from `/metrics`, the JSON routes append
`.json` and `.summary.json` to that path (e.g. `/private/stats` →
`/private/stats.json` and `/private/stats.summary.json`). Exporter routes
reject query strings. Every JSON response uses `application/json` and
`Cache-Control: no-store`; do not expose these endpoints publicly.

For a dedicated integration, issue a General API token with **read-only**
permission and a narrow `allowed_cidrs` set. Example from a permitted client:

```sh
curl -fsS -H "Authorization: Bearer $RUSTPROXY_API_TOKEN" \
  http://127.0.0.1:9090/api/metrics/json | jq '{vhosts: .vhosts | {total,enabled,disabled}, requests: .requests.total, cpu: .system.cpu_percent, ram_bytes: .system.process_rss_bytes}'
```

### Summary: ready for dashboard cards

For the shortest response, poll **`/api/metrics/summary`** (or the standalone
exporter's `.summary.json` sibling) instead of loading every Prometheus sample.
The full `/api/metrics/json` includes the **same calculated object** at `summary`.
The response groups all values into `vhosts`, `backends`, `traffic`, `system`,
`cache`, `web_server`, `streams` and `capacity`, plus a collection timestamp.

```sh
curl -fsS -H "Authorization: Bearer $RUSTPROXY_API_TOKEN" \
  http://127.0.0.1:9090/api/metrics/summary | \
  jq '{working_vhosts: .vhosts.serving_estimate, backends: .backends.total, \
       backend_up: .backends.up, rps: .traffic.requests_per_second, \
       traffic_mb: .traffic.megabytes_total, cpu: .system.cpu_percent, \
       ram_percent: .system.memory_percent}'
```

- `vhosts.enabled` is the configured on/off flag. `vhosts.serving_estimate`
  counts enabled vhosts using a ready backend pool or an embedded Web Server.
  It is **not** an HTTP probe or guarantee of reachability; redirect hosts,
  not-found hosts and Docker assignments are counted separately. Only enabled
  vhosts with a compiled pool are included in `backend_ready` / `backend_not_ready`.
- `backends.total` counts **live compiled pool members including Docker**.
  `configured_static` counts saved static members. `up`, `down`, `checking`,
  `draining`, `disabled`, `unknown`, `pools_ready` and `active_exchanges`
  are independently calculated; `traffic_bytes_per_second` sums moving
  backend rates and **does not** represent all public HTTP traffic.
- `traffic.requests_total`, `bytes_in_total`, `bytes_out_total` and `bytes_total`
  are cumulative process counters; `megabytes_total` uses decimal MB (10⁶ B).
  `requests_per_second`, `errors_per_second`, `bytes_in_per_second`,
  `bytes_out_per_second` and `bytes_per_second` are **rolling poll interval
  rates** computed on the server from a shared bounded sample (minimum 1 s,
  maximum 30 s between usable samples). An initial poll, restart, counter reset
  or gap longer than 30 s yields `null` until a second valid poll. Very rapid
  simultaneous polls reuse the latest completed interval. These are not
  guaranteed exact one-second rolling averages; poll every **2–5 seconds**.
  `requests_per_second_average` is a separate **uptime average**.
- `traffic.error_percent` is lifetime errors divided by completed requests;
  `cache.hit_percent` is lifetime HOT/COLD hit ratio from cache metrics.
  Zero-denominator percentages are `null` unless a cache ratio has a defined
  default. The `traffic.network_rx_bytes_per_second` and
  `network_tx_bytes_per_second` fields are **host** interface rates, not
  RustProxy-exclusive transfer; backend rate is separate.
- `system.cpu_percent`, `process_cpu_percent`, `memory_percent`, RAM and RSS
  reflect the last OS sample (`system.sampled_at_ms`). Unsupported values can
  be `null`. `web_server.requests_total` adds observed Web Server responses,
  while `streams.active_sessions` adds observed Layer-4 sessions.

The JSON is organized for simple chart consumption:

- `summary` contains the small, calculated metrics object described above.
- `vhosts` includes `total`, `enabled`, `disabled`, backend readiness and an array of per-Vhost counters.
- `requests` includes process-lifetime totals, active exchanges, errors, status
  classes, byte counters, latency, active connections and H1/H2/H3/WebSockets.
- `system` includes sampled host/process CPU, memory, swap, process I/O/FDs, network interfaces and throughput.
- `backends`, `web_server`, `streams`, `docker`, `cache`, `traffic`, `transform`,
  `logging` and `capacity` contain their corresponding operational summaries.
- `prometheus.types` and `prometheus.samples` include **every numeric sample**
  exported by the existing Prometheus text renderer, with label pairs as JSON
  objects. Histogram buckets/counts/sums are included. The normal Prometheus
  collector switches still control this detailed section.

Example field access:

```js
const snapshot = await (await fetch('/api/metrics/json')).json();
const vhostsEnabled = snapshot.vhosts.enabled;
const requestTotal = snapshot.requests.total;
const cpuPercent = snapshot.system.cpu_percent; // can be null before sampling
const samples = snapshot.prometheus.samples.rust_proxy_vhost_requests_total ?? [];
// samples[0] -> { labels: { vhost: 'site1' }, value: 123 }
```

`collected_at_ms` is the JSON snapshot time; `system.sampled_at_ms` is the
last OS collector sample time (the OS collector runs in the background).
Raw counters represent cumulative totals, **not instantaneous RPS**. Use the
server-computed interval rates from `summary` or calculate
`(new_total - old_total) / elapsed_seconds` from raw counters in the client;
reset the baseline after counter decreases, process restart or config reset.
Absent/unsupported OS values are `null`; non-finite Prometheus values are also
represented as `null`. Traffic breakdowns are approximate and can lose samples
under load. The JSON excludes raw request paths, client IP addresses and raw
User-Agent values, matching the bounded-label privacy model of Prometheus.

## Metric privacy and limits

Prometheus series use bounded configured labels; arbitrary request URLs, raw client IPs, credentials, raw User-Agents and query strings are intentionally not exposed as unbounded labels. Dashboard traffic top lists can still contain sensitive operational metadata, so keep the panel authenticated.

Metrics reset on process replacement, and some per-Vhost/backend records can reset when identity or config changes. Distinguish counters, rates and gauges; watch for traffic sampling gaps rather than interpreting every missing point as zero. Before deployment, verify collector reachability, firewall rules, CPU/RAM overhead and alert thresholds against real traffic.