v0.26.2
This commit is contained in:
Mateusz Gruszczyński committed
1 parent
b9d137eee1
commit
32e6875980
51 files changed
+1255
-452
No files matched your search
+56
-65
@@ -1,87 +1,78 @@
|
||||
# Stored log explorer
|
||||
|
||||
RustProxy 0.26.1 provides an authenticated, read-only **Logs** workspace at `/logs`. It is a local file-backed explorer for RustProxy logging outputs and is intentionally separate from **Live Logs**.
|
||||
RustProxy 0.26.2 provides an authenticated, read-only **Logs** workspace at `/logs`. The top bar has one **Logs** menu with separate **Stored logs** and **Live Logs** actions; the two data paths remain independent.
|
||||
|
||||
## Separation from Live Logs
|
||||
|
||||
Live Logs uses the bounded in-memory request history and Tokio broadcast channel. It remains available even when file logging is disabled and is optimized for current routed request activity.
|
||||
|
||||
The stored-log explorer does not read that memory ring. It reads only files derived from the active RustProxy logging configuration:
|
||||
Stored Logs reads only paths derived from the active RustProxy logging configuration: the combined structured access log or per-vhost JSONL directory plus the optional error log. Existing files remain browsable after their writer is disabled. Low-level `protocol_error` records remain persistence-only and do not enter Live Logs.
|
||||
|
||||
- the combined structured access-log path, or JSONL files directly inside the configured per-vhost directory;
|
||||
- the configured optional error-log file.
|
||||
## Search syntax
|
||||
|
||||
Existing files remain browsable after their writer is disabled. Low-level `protocol_error` records are sent only through configured persistent structured-log sinks and are deliberately not inserted into Live Logs.
|
||||
The dashboard uses one search field and a page-size selector. Search is case-insensitive and supports field clauses, negation, quoted values, wildcards and numeric comparisons.
|
||||
|
||||
Examples:
|
||||
|
||||
```text
|
||||
req: GET AND !country: Poland
|
||||
status:>=500 AND path:/api/*
|
||||
error:"connection reset"
|
||||
client_ip:192.0.2.* AND status:5xx
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- whitespace and explicit `AND` join clauses;
|
||||
- `!field:value` excludes matching records;
|
||||
- quotes keep spaces in one value;
|
||||
- `*` acts as a wildcard;
|
||||
- numeric fields accept `>`, `>=`, `<` and `<=`;
|
||||
- `status:5xx` matches a status class;
|
||||
- bare terms search all stored values;
|
||||
- `req`/`request` aliases `method`, `ip`/`client` aliases `client_ip`, and `source` aliases the internal log-source ID;
|
||||
- `OR` is not supported and is rejected explicitly.
|
||||
|
||||
The API continues to accept the older `source`, `event`, `status_class`, `client_ip` and `vhost` parameters for compatibility, but the dashboard no longer exposes those separate controls.
|
||||
|
||||
## JSON parsing and dynamic columns
|
||||
|
||||
JSON/JSONL lines are parsed into record objects. Canonical access-log fields are reconstructed from the active JSON field mapping where possible, so fields such as `client_ip` remain usable even when their output key was renamed.
|
||||
|
||||
The query response includes a bounded list of discovered leaf field names. The browser builds the column picker from those fields. The default concise view is time, event, status, client IP, vhost, record summary and source; any returned JSON field can be selected dynamically. Fields such as `matched_by`, `sni`, `authority`, `country`, `request_id` or custom mapped fields therefore appear only when the data actually contains them.
|
||||
|
||||
At least one column must remain visible. Stored-log column selection is saved per administrator account and accepts up to 64 bounded field names.
|
||||
|
||||
Selecting a row highlights it and expands parsed field/value pairs directly below the row. A nested **Raw JSON** section shows the normalized complete record.
|
||||
|
||||
## RAW error logs
|
||||
|
||||
RustProxy's built-in RAW error-log format is also normalized for browsing. The explorer extracts timestamp, level, target, message and recognizable `key=value` fields, while retaining the original line in `raw`.
|
||||
|
||||
RAW is not a lossless structured serialization, so extraction is best-effort. Fields that cannot be separated safely remain visible through the message/raw record rather than being invented.
|
||||
|
||||
## Vhost-aware protocol records
|
||||
|
||||
Ordinary completed requests use `event: "request"`. Low-level failures use `event: "protocol_error"`. When RustProxy can safely identify a configured vhost, the record is persisted through that vhost's existing sink. In per-vhost file mode an unresolved record uses `_unmatched.jsonl`.
|
||||
|
||||
Attribution priority is:
|
||||
|
||||
1. vhost already selected by normal routing;
|
||||
2. HTTP `:authority`;
|
||||
3. HTTP `Host`;
|
||||
4. TLS/QUIC SNI;
|
||||
5. no safe match -> `_unmatched`.
|
||||
|
||||
All hostname matching reuses the normal exact/wildcard frontend resolver and listener binding rules. A listener or local port by itself is not sufficient to select a vhost.
|
||||
|
||||
A representative record is:
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "protocol_error",
|
||||
"reason": "tls_handshake_error",
|
||||
"protocol_error": "tls_handshake_error",
|
||||
"error": "...",
|
||||
"error_detail": "...",
|
||||
"client_ip": "192.0.2.10",
|
||||
"local_addr": "192.0.2.20:443",
|
||||
"transport": "tls",
|
||||
"vhost": "app",
|
||||
"sni": "app.example",
|
||||
"tls_sni": "app.example",
|
||||
"authority": null,
|
||||
"matched_by": "sni",
|
||||
"duration_ms": 0
|
||||
}
|
||||
```
|
||||
|
||||
`matched_by` is `router`, `authority`, `host`, `sni` or `none`. Compatibility fields from 0.26.0 remain present so existing consumers do not need an immediate migration.
|
||||
|
||||
SNI, authority and Host are lookup inputs only. RustProxy never derives a filename directly from client-provided identity text. Only the configured internal vhost identifier returned by the shared resolver can select a vhost sink. Invalid internal IDs fall back to `_unmatched` rather than being sanitized into a new path.
|
||||
|
||||
Diagnostic identity values are control-character stripped and bounded before persistence. Existing error-detail redaction and limits remain in force. Raw packets, request bodies and raw malformed request bytes are not captured.
|
||||
|
||||
## Logs Explorer UI
|
||||
|
||||
The workspace supports source, case-insensitive text, event class, HTTP status class, client IP and vhost filters. Text search scans stored values, including protocol diagnostics such as `reason`, `error`, `sni` and `authority`, and refreshes with a short debounce while typing.
|
||||
|
||||
The table provides account-scoped selectable columns. The default view keeps the original concise columns while optional protocol columns expose transport, match source, SNI and authority. At least one column must remain visible.
|
||||
|
||||
The table header stays visible while the result list scrolls. Selecting a row highlights it and expands the complete JSON record inline immediately below that row; selecting it again collapses the details.
|
||||
|
||||
The configured-file inventory is collapsed by default so installations with many per-vhost files do not lose most of the screen to source paths. The source selector still contains the configured sources, including `_unmatched.jsonl` when present.
|
||||
|
||||
For structured access logs, the explorer reconstructs canonical fields from the active JSON field mapping. In particular, `client_ip` remains available to the table and filter even when the output JSON key was renamed by a custom mapping.
|
||||
Attribution priority remains routed vhost, HTTP `:authority`, HTTP `Host`, TLS/QUIC SNI, then unmatched. Matching reuses normal exact/wildcard frontend resolution and listener binding. Client Host/SNI text never becomes a filesystem path.
|
||||
|
||||
## Browser security and bounds
|
||||
|
||||
The browser never supplies a filesystem path. `/api/file-logs` returns opaque source IDs derived from the current configuration, and `/api/file-logs/query` accepts only those IDs or `all`.
|
||||
The dashboard no longer exposes the configured-file inventory. The Source selector is optional: `All configured files` is the default, while selecting one source narrows the same bounded query to that log only.
|
||||
|
||||
The reader is bounded:
|
||||
The browser still never supplies a filesystem path. The server derives sources from the active configuration and reads only regular files. On Unix it opens with `O_NOFOLLOW`; per-vhost enumeration rejects a symlinked directory and accepts only direct regular `.jsonl` children.
|
||||
|
||||
- up to 2 MiB scanned from the tail of one source;
|
||||
The reader remains bounded:
|
||||
|
||||
- up to 2 MiB scanned from one source;
|
||||
- up to 8 MiB across one query;
|
||||
- up to 20,000 lines across the complete query, shared across selected sources;
|
||||
- at most 32 sources in an `all` query;
|
||||
- page size at most 500 and offset at most 10,000.
|
||||
- up to 20,000 lines across a query;
|
||||
- at most 32 sources in an all-source query;
|
||||
- page size at most 500 and offset at most 10,000;
|
||||
- search input at most 512 bytes and 32 parsed clauses;
|
||||
- dynamic field discovery at most 64 leaf fields.
|
||||
|
||||
If more than 32 configured files are eligible for an `all` query, the newest 32 are scanned and the response is marked truncated. A specific source can always be selected directly.
|
||||
If more than 32 sources are eligible, the newest 32 are scanned and the response is marked truncated. File reads run in a blocking worker rather than the async proxy runtime.
|
||||
|
||||
File reads run in a blocking worker rather than the async proxy runtime. Only regular files are accepted. On Unix the reader opens with `O_NOFOLLOW`; a symlink is not followed. Per-vhost enumeration also rejects a configured directory that is itself a symlink and only lists direct regular `.jsonl` children.
|
||||
|
||||
The endpoints are private administration routes and inherit the existing panel session/Host/Origin access controls. There is no public-proxy route to the file reader.
|
||||
|
||||
The explorer is diagnostic rather than a retention system. RustProxy does not add rollover, indexing or compression; use the operating-system log rotation policy for retention.
|
||||
The explorer is diagnostic rather than a retention/indexing system. RustProxy does not add rollover, indexing or compression; use operating-system log rotation for retention.
|
||||
Reference in new issue
Block a user