161 lines
12 KiB
Markdown
161 lines
12 KiB
Markdown
# Monitoring, metrics and health
|
||
|
||
RustProxy has operational views in the authenticated panel and an optional standalone Prometheus exporter. They measure different things: dashboard sampling, process-lifetime counters and stored logs are not interchangeable.
|
||
|
||
## What to watch
|
||
|
||
| View | Main information | Interpretation |
|
||
|---|---|---|
|
||
| **Overview** | Active connections, request rate, errors, process CPU/RAM, protocols, backend state, optional charts | Dashboard samples are not long-term billing data. |
|
||
| **Vhosts → Backends** | Member health, weights, H1/H2 modes, current work, last observed upstream response | A `last_status` from normal proxy traffic is not an active health probe result. |
|
||
| **Web Server** | Requests/status classes, static/PHP traffic, response-header time, response bytes, PHP concurrency permits | Response header duration excludes streamed body transfer. PHP permits are not FPM child processes. |
|
||
| **Traffic** | Per-Vhost totals and top countries/IPs/paths/methods/agents/cache results | Bounded in-memory analytics, approximate top lists and possible sample loss under heavy load. |
|
||
| **Cache** | HOT/COLD occupancy, hit/miss/bypass, object metadata, selected invalidation | HTTP `Age` differs from cache residence age. |
|
||
| **Docker / Streams** | Agent status, dynamic Vhost health, TCP/UDP sessions and traffic | An agent can be waiting for first observation; Streams are non-HTTP. |
|
||
| **Logs** | Live requests, stored JSONL/error/protocol records | A log sink can be disabled while live memory view remains available. |
|
||
|
||
Overview allows account-specific card layout and history controls; see [Dashboard](DASHBOARD.md). GeoIP/country analytics depend on the installed/update database and whether country capture is enabled.
|
||
|
||
## Prometheus exporter
|
||
|
||
The exporter is **disabled by default** and uses a separate HTTP listener, normally `127.0.0.1:9191`, at `/metrics`. It is not protected by the admin browser cookie; it has its **own** client-network and bearer-token policy. In **Settings → Prometheus** enable the endpoint and configure:
|
||
|
||
In 0.34.1, use the separate **Prometheus token** controls in Settings to generate, rotate or revoke a 32+ character Bearer credential. Its clear value appears only once; copy it directly into your monitoring secrets store. The database keeps only its hash, and rotation/revocation takes effect without reprinting the previous secret. The older environment-variable credential remains supported as a fallback *only until* a generated credential or explicit revocation entry exists. Setting a new token does not reconfigure the exporter listener or its enable switch.
|
||
|
||
|
||
- `listen`, `path`, `allowed_cidrs` and an optional environment-variable **name** `bearer_token_env` (the secret value is supplied separately to the service).
|
||
- `allow_trusted_network_without_token` only for a deliberately restricted monitoring subnet. `/0` CIDRs are not accepted in this mode.
|
||
- Collector switches for process/system, Vhosts, traffic, backends, Docker, cache domains, Streams and logging.
|
||
|
||
A non-loopback exporter requires a valid generated or environment Bearer token at scrape time, or an explicit token-free trusted-network exception; otherwise requests return 401. The exporter uses plain HTTP; keep it on loopback, trusted network or a protected TLS tunnel. `Forwarded`/`X-Forwarded-For` headers cannot grant exporter access because network restrictions apply to the transport peer.
|
||
|
||
For legacy environment-token authentication, provide a 32–4096-character printable non-whitespace ASCII token in a **service environment variable** and set `bearer_token_env` to its name, e.g. `RUST_PROXY_METRICS_TOKEN`. The generated-token workflow above avoids this environment variable. Changing exporter activation/bind/token-variable usually needs restart; some collector/CIDR settings apply live. The exact allowed GET path must not have a query string.
|
||
|
||
Example scrape configuration, after choosing the correct private target:
|
||
|
||
```yaml
|
||
scrape_configs:
|
||
- job_name: rustproxy
|
||
static_configs:
|
||
- targets: ['127.0.0.1:9191']
|
||
```
|
||
|
||
When using a bearer token, configure Prometheus `authorization` or `bearer_token_file` securely rather than embedding a clear token in an exported dashboard. Review `grafana/README.md` and `grafana/rustproxy-overview.json` for chart examples; `nagios/README.md` contains external check scripts.
|
||
|
||
## JSON metrics API for external dashboards
|
||
|
||
RustProxy also offers a **plain, read-only JSON snapshot**. It does not use SSE,
|
||
WebSockets or persistent subscriptions: clients poll using ordinary `GET` (for
|
||
example, every 2–5 seconds).
|
||
|
||
| Endpoint | Where | Authentication |
|
||
|---|---|---|
|
||
| `GET /api/metrics/json` | Administration listener (default `127.0.0.1:9090`) | Admin session or a **General API** `ro`/`rw` Bearer token from **Settings → Security**, restricted by allowed CIDRs |
|
||
| `GET /api/metrics/summary` | Administration listener, compact calculated metrics only | **Same** admin session/General API Bearer credentials and CIDR policy |
|
||
| `GET /metrics.json` | Dedicated exporter (default `127.0.0.1:9191`, **only when Prometheus is enabled**) | Same allowed CIDRs and exporter Bearer token as `/metrics`; not an admin/API token |
|
||
| `GET /metrics.summary.json` | Dedicated exporter, compact calculated metrics only | **Same** exporter Bearer token/CIDR policy, if enabled |
|
||
|
||
When the exporter `path` differs from `/metrics`, the JSON routes append
|
||
`.json` and `.summary.json` to that path (e.g. `/private/stats` →
|
||
`/private/stats.json` and `/private/stats.summary.json`). Exporter routes
|
||
reject query strings. Every JSON response uses `application/json` and
|
||
`Cache-Control: no-store`; do not expose these endpoints publicly.
|
||
|
||
For a dedicated integration, issue a General API token with **read-only**
|
||
permission and a narrow `allowed_cidrs` set. Example from a permitted client:
|
||
|
||
```sh
|
||
curl -fsS -H "Authorization: Bearer $RUSTPROXY_API_TOKEN" \
|
||
http://127.0.0.1:9090/api/metrics/json | jq '{vhosts: .vhosts | {total,enabled,disabled}, requests: .requests.total, cpu: .system.cpu_percent, ram_bytes: .system.process_rss_bytes}'
|
||
```
|
||
|
||
### Summary: ready for dashboard cards
|
||
|
||
For the shortest response, poll **`/api/metrics/summary`** (or the standalone
|
||
exporter's `.summary.json` sibling) instead of loading every Prometheus sample.
|
||
The full `/api/metrics/json` includes the **same calculated object** at `summary`.
|
||
The response groups all values into `vhosts`, `backends`, `traffic`, `system`,
|
||
`cache`, `web_server`, `streams` and `capacity`, plus a collection timestamp.
|
||
|
||
```sh
|
||
curl -fsS -H "Authorization: Bearer $RUSTPROXY_API_TOKEN" \
|
||
http://127.0.0.1:9090/api/metrics/summary | \
|
||
jq '{working_vhosts: .vhosts.serving_estimate, backends: .backends.total, \
|
||
backend_up: .backends.up, rps: .traffic.requests_per_second, \
|
||
traffic_mb: .traffic.megabytes_total, cpu: .system.cpu_percent, \
|
||
ram_percent: .system.memory_percent}'
|
||
```
|
||
|
||
- `vhosts.enabled` is the configured on/off flag. `vhosts.serving_estimate`
|
||
counts enabled vhosts using a ready backend pool or an embedded Web Server.
|
||
It is **not** an HTTP probe or guarantee of reachability; redirect hosts,
|
||
not-found hosts and Docker assignments are counted separately. Only enabled
|
||
vhosts with a compiled pool are included in `backend_ready` / `backend_not_ready`.
|
||
- `backends.total` counts **live compiled pool members including Docker**.
|
||
`configured_static` counts saved static members. `up`, `down`, `checking`,
|
||
`draining`, `disabled`, `unknown`, `pools_ready` and `active_exchanges`
|
||
are independently calculated; `traffic_bytes_per_second` sums moving
|
||
backend rates and **does not** represent all public HTTP traffic.
|
||
- `traffic.requests_total`, `bytes_in_total`, `bytes_out_total` and `bytes_total`
|
||
are cumulative process counters; `megabytes_total` uses decimal MB (10⁶ B).
|
||
`requests_per_second`, `errors_per_second`, `bytes_in_per_second`,
|
||
`bytes_out_per_second` and `bytes_per_second` are **rolling poll interval
|
||
rates** computed on the server from a shared bounded sample (minimum 1 s,
|
||
maximum 30 s between usable samples). An initial poll, restart, counter reset
|
||
or gap longer than 30 s yields `null` until a second valid poll. Very rapid
|
||
simultaneous polls reuse the latest completed interval. These are not
|
||
guaranteed exact one-second rolling averages; poll every **2–5 seconds**.
|
||
`requests_per_second_average` is a separate **uptime average**.
|
||
- `traffic.error_percent` is lifetime errors divided by completed requests;
|
||
`cache.hit_percent` is lifetime HOT/COLD hit ratio from cache metrics.
|
||
Zero-denominator percentages are `null` unless a cache ratio has a defined
|
||
default. The `traffic.network_rx_bytes_per_second` and
|
||
`network_tx_bytes_per_second` fields are **host** interface rates, not
|
||
RustProxy-exclusive transfer; backend rate is separate.
|
||
- `system.cpu_percent`, `process_cpu_percent`, `memory_percent`, RAM and RSS
|
||
reflect the last OS sample (`system.sampled_at_ms`). Unsupported values can
|
||
be `null`. `web_server.requests_total` adds observed Web Server responses,
|
||
while `streams.active_sessions` adds observed Layer-4 sessions.
|
||
|
||
The JSON is organized for simple chart consumption:
|
||
|
||
- `summary` contains the small, calculated metrics object described above.
|
||
- `vhosts` includes `total`, `enabled`, `disabled`, backend readiness and an array of per-Vhost counters.
|
||
- `requests` includes process-lifetime totals, active exchanges, errors, status
|
||
classes, byte counters, latency, active connections and H1/H2/H3/WebSockets.
|
||
- `system` includes sampled host/process CPU, memory, swap, process I/O/FDs, network interfaces and throughput.
|
||
- `backends`, `web_server`, `streams`, `docker`, `cache`, `traffic`, `transform`,
|
||
`logging` and `capacity` contain their corresponding operational summaries.
|
||
- `prometheus.types` and `prometheus.samples` include **every numeric sample**
|
||
exported by the existing Prometheus text renderer, with label pairs as JSON
|
||
objects. Histogram buckets/counts/sums are included. The normal Prometheus
|
||
collector switches still control this detailed section.
|
||
|
||
Example field access:
|
||
|
||
```js
|
||
const snapshot = await (await fetch('/api/metrics/json')).json();
|
||
const vhostsEnabled = snapshot.vhosts.enabled;
|
||
const requestTotal = snapshot.requests.total;
|
||
const cpuPercent = snapshot.system.cpu_percent; // can be null before sampling
|
||
const samples = snapshot.prometheus.samples.rust_proxy_vhost_requests_total ?? [];
|
||
// samples[0] -> { labels: { vhost: 'site1' }, value: 123 }
|
||
```
|
||
|
||
`collected_at_ms` is the JSON snapshot time; `system.sampled_at_ms` is the
|
||
last OS collector sample time (the OS collector runs in the background).
|
||
Raw counters represent cumulative totals, **not instantaneous RPS**. Use the
|
||
server-computed interval rates from `summary` or calculate
|
||
`(new_total - old_total) / elapsed_seconds` from raw counters in the client;
|
||
reset the baseline after counter decreases, process restart or config reset.
|
||
Absent/unsupported OS values are `null`; non-finite Prometheus values are also
|
||
represented as `null`. Traffic breakdowns are approximate and can lose samples
|
||
under load. The JSON excludes raw request paths, client IP addresses and raw
|
||
User-Agent values, matching the bounded-label privacy model of Prometheus.
|
||
|
||
## Metric privacy and limits
|
||
|
||
Prometheus series use bounded configured labels; arbitrary request URLs, raw client IPs, credentials, raw User-Agents and query strings are intentionally not exposed as unbounded labels. Dashboard traffic top lists can still contain sensitive operational metadata, so keep the panel authenticated.
|
||
|
||
Metrics reset on process replacement, and some per-Vhost/backend records can reset when identity or config changes. Distinguish counters, rates and gauges; watch for traffic sampling gaps rather than interpreting every missing point as zero. Before deployment, verify collector reachability, firewall rules, CPU/RAM overhead and alert thresholds against real traffic.
|