83 lines
5.6 KiB
Markdown
83 lines
5.6 KiB
Markdown
# Stored log explorer
|
|
|
|
RustProxy 0.29.0 provides an authenticated, read-only **Logs** workspace at `/logs`. The top bar has one **Logs** menu with separate **Stored logs** and **Live Logs** actions; the two data paths remain independent.
|
|
|
|
## Separation from Live Logs
|
|
|
|
Live Logs uses the bounded in-memory request history and Tokio broadcast channel. It remains available even when file logging is disabled and is optimized for current routed request activity.
|
|
|
|
Stored Logs reads only paths derived from the active RustProxy logging configuration: the combined structured access log or per-vhost JSONL directory plus the optional error log. Existing files remain browsable after their writer is disabled. Low-level `protocol_error` records remain persistence-only and do not enter Live Logs.
|
|
|
|
## Search syntax
|
|
|
|
The dashboard uses one search field and a page-size selector. Search is case-insensitive and supports field clauses, negation, quoted values, wildcards and numeric comparisons.
|
|
|
|
Examples:
|
|
|
|
```text
|
|
req: GET AND !country: Poland
|
|
status:>=500 AND path:/api/*
|
|
error:"connection reset"
|
|
client_ip:192.0.2.* AND status:5xx
|
|
```
|
|
|
|
Rules:
|
|
|
|
- whitespace and explicit `AND` join clauses;
|
|
- `!field:value` excludes matching records;
|
|
- quotes keep spaces in one value;
|
|
- `*` acts as a wildcard;
|
|
- numeric fields accept `>`, `>=`, `<` and `<=`;
|
|
- `status:5xx` matches a status class;
|
|
- bare terms search all stored values;
|
|
- `req`/`request` aliases `method`, `ip`/`client` aliases `client_ip`, and `source` aliases the internal log-source ID;
|
|
- `OR` is not supported and is rejected explicitly.
|
|
|
|
The API continues to accept the older `source`, `event`, `status_class`, `client_ip` and `vhost` parameters for compatibility, but the dashboard no longer exposes those separate controls.
|
|
|
|
## JSON parsing and dynamic columns
|
|
|
|
JSON/JSONL lines are parsed into record objects. Canonical access-log fields are reconstructed from the active JSON field mapping where possible, so fields such as `client_ip` remain usable even when their output key was renamed.
|
|
|
|
The query response includes a bounded list of discovered leaf field names. The browser builds the column picker from those fields. The default concise view is time, event, status, client IP, vhost, record summary and source; any returned JSON field can be selected dynamically. Fields such as `matched_by`, `sni`, `authority`, `country`, `request_id` or custom mapped fields therefore appear only when the data actually contains them.
|
|
|
|
At least one column must remain visible. Stored-log column selection is saved per administrator account and accepts up to 64 bounded field names.
|
|
|
|
Selecting a row highlights it and expands parsed field/value pairs directly below the row. A nested **Raw JSON** section shows the normalized complete record.
|
|
|
|
## RAW error logs
|
|
|
|
RustProxy's built-in RAW error-log format is also normalized for browsing. The explorer extracts timestamp, level, target, message and recognizable `key=value` fields, while retaining the original line in `raw`.
|
|
|
|
RAW is not a lossless structured serialization, so extraction is best-effort. Fields that cannot be separated safely remain visible through the message/raw record rather than being invented.
|
|
|
|
## Vhost-aware protocol records
|
|
|
|
Ordinary completed requests use `event: "request"`. Low-level failures use `event: "protocol_error"`. When RustProxy can safely identify a configured vhost, the record is persisted through that vhost's existing sink. In per-vhost file mode an unresolved record uses `_unmatched.jsonl`.
|
|
|
|
Attribution priority remains routed vhost, HTTP `:authority`, HTTP `Host`, TLS/QUIC SNI, then unmatched. Matching reuses normal exact/wildcard frontend resolution and listener binding. Client Host/SNI text never becomes a filesystem path.
|
|
|
|
## Browser security and bounds
|
|
|
|
The dashboard no longer exposes the configured-file inventory. The Source selector is optional: `All configured files` is the default, while selecting one source narrows the same bounded query to that log only.
|
|
|
|
The browser still never supplies a filesystem path. The server derives sources from the active configuration and reads only regular files. On Unix it opens with `O_NOFOLLOW`; per-vhost enumeration rejects a symlinked directory and accepts only direct regular `.jsonl` children.
|
|
|
|
Logrotate-style gzip rotations are discovered next to configured combined access/error files and inside the configured per-vhost directory. Only regular non-symlink files matching the active filename rotation pattern (for example `access.jsonl.1.gz`) are eligible. The browser still refers to them by server-generated source IDs rather than paths.
|
|
|
|
Gzip reading is bounded before and after decompression. A compressed source larger than 8 MiB is not decompressed for a query, decompressed output is capped at 64 MiB, and only the bounded tail is parsed/searched. These limits prevent a gzip rotation from becoming an unbounded memory/decompression workload.
|
|
|
|
The reader remains bounded:
|
|
|
|
- up to 2 MiB of retained tail data from one source;
|
|
- up to 8 MiB across one query;
|
|
- up to 20,000 lines across a query;
|
|
- at most 32 sources in an all-source query;
|
|
- page size at most 500 and offset at most 10,000;
|
|
- search input at most 512 bytes and 32 parsed clauses;
|
|
- dynamic field discovery at most 64 leaf fields.
|
|
|
|
If more than 32 sources are eligible, the newest 32 are scanned and the response is marked truncated. File reads run in a blocking worker rather than the async proxy runtime.
|
|
|
|
The explorer is diagnostic rather than a retention/indexing system. RustProxy does not create rollovers or compress files itself; use operating-system log rotation for retention. Existing gzip rotations can be read as described above.
|