RustProxy/docs/FILE_LOGS.md

Stored log explorer

RustProxy provides an authenticated, read-only Logs workspace at /logs. The top bar has one Logs menu with separate Stored logs and Live Logs actions; the two data paths remain independent.

Separation from Live Logs

Live Logs uses the bounded in-memory request history and Tokio broadcast channel. It remains available even when file logging is disabled and is optimized for current routed request activity.

Stored Logs reads only paths derived from the active RustProxy logging configuration: the combined structured access log or per-vhost JSONL directory plus the optional error log. Existing files remain browsable after their writer is disabled. Low-level protocol_error records remain persistence-only and do not enter Live Logs.

Search syntax

The dashboard uses one search field and a page-size selector. Search is case-insensitive and supports field clauses, negation, quoted values, wildcards and numeric comparisons.

Examples:

req: GET AND !country: Poland
status:>=500 AND path:/api/*
error:"connection reset"
client_ip:192.0.2.* AND status:5xx

Rules:

• whitespace and explicit AND join clauses;
• !field:value excludes matching records;
• quotes keep spaces in one value;
• * acts as a wildcard;
• numeric fields accept >, >=, < and <=;
• status:5xx matches a status class;
• bare terms search all stored values;
• req/request aliases method, ip/client aliases client_ip, and source aliases the internal log-source ID;
• OR is not supported and is rejected explicitly.

The API continues to accept the older source, event, status_class, client_ip and vhost parameters for compatibility, but the dashboard no longer exposes those separate controls.

JSON parsing and dynamic columns

JSON/JSONL lines are parsed into record objects. Canonical access-log fields are reconstructed from the active JSON field mapping where possible, so fields such as client_ip remain usable even when their output key was renamed.

The query response includes a bounded list of discovered leaf field names. The browser builds the column picker from those fields. The default concise view is time, event, status, client IP, vhost, record summary and source; any returned JSON field can be selected dynamically. Fields such as matched_by, sni, authority, country, request_id or custom mapped fields therefore appear only when the data actually contains them.

At least one column must remain visible. Stored-log column selection is saved per administrator account and accepts up to 64 bounded field names.

Selecting a row highlights it and expands parsed field/value pairs directly below the row. A nested Raw JSON section shows the normalized complete record.

Sorting columns (0.31.8)

Click a Stored Logs column header (desktop/table layout) to sort by that column; click again to reverse direction. The selected field and direction are saved with the administrator's column preferences. Numeric fields sort numerically, text case-insensitively, and absent values come last. Both standard columns (time, record, source) and discovered nested JSON fields are supported.

Sorting takes place on the server across the entire bounded matching scan, before page offset/limit is applied. Thus page two continues the same sorted result instead of sorting only the current 100 displayed rows. Server bounds and truncation notices still apply; this is not an unbounded full-log historical index. The default remains newest first. The mobile card layout retains the selected sort order, set via the desktop table view.

Narrow-screen navigation (0.31.5)

At viewport widths up to 700px, /logs uses a vertically scrolling list of compact cards in place of a horizontally expanding results table. Each card shows every enabled column, with long values wrapped or shortened until expanded; tap the card (or focus and press Enter/Space) to inspect all parsed fields and the complete Raw JSON. Search, source, page size, column selection, pagination and repeated-value analysis stay available. Large raw text and aggregate tables scroll inside their own containers, never across the entire administration page. Tablet/desktop results keep the normal tabular layout.

Repeated-value analysis

The Repeated values tab (advanced parser) can count repeated values for any parsed field returned by the explorer. Typical security-investigation fields are user_agent, client_ip, host and path. Counts are calculated across the bounded matching scan before pagination, so changing the page does not change the count set. Minimum repetitions and top-result count are configurable within server limits. Values in the result table are directly copyable.

Aggregation does not remove the Stored Logs safety bounds: at most 4,096 distinct aggregate values are retained and at most 200 results are returned. The response marks an aggregate as truncated when a cardinality or result cap is reached.

RAW error logs

RustProxy's built-in RAW error-log format is also normalized for browsing. The explorer extracts timestamp, level, target, message and recognizable key=value fields, while retaining the original line in raw.

RAW is not a lossless structured serialization, so extraction is best-effort. Fields that cannot be separated safely remain visible through the message/raw record rather than being invented.

Vhost-aware protocol records

Ordinary completed requests use event: "request". Low-level failures use event: "protocol_error". When RustProxy can safely identify a configured vhost, the record is persisted through that vhost's existing sink. In per-vhost file mode an unresolved record uses _unmatched.jsonl.

Attribution priority remains routed vhost, HTTP :authority, HTTP Host, TLS/QUIC SNI, then unmatched. Matching reuses normal exact/wildcard frontend resolution and listener binding. Client Host/SNI text never becomes a filesystem path.

Browser security and bounds

The dashboard no longer exposes the configured-file inventory. The Source selector is optional: All configured files is the default, while selecting one source narrows the same bounded query to that log only.

The browser still never supplies a filesystem path. The server derives sources from the active configuration and reads only regular files. On Unix it opens with O_NOFOLLOW; per-vhost enumeration rejects a symlinked directory and accepts only direct regular .jsonl children.

Logrotate-style gzip rotations are discovered next to configured combined access/error files and inside the configured per-vhost directory. Only regular non-symlink files matching the active filename rotation pattern (for example access.jsonl.1.gz) are eligible. The browser still refers to them by server-generated source IDs rather than paths.

For per-vhost logging, the active vhost.jsonl and matching vhost.jsonl.*.gz files share one logical group ID. The panel enables Consolidate rotated logs per vhost by default, so selecting that vhost searches the active and compressed members together as one logical source. Disable the switch to inspect individual files. Physical source IDs and metadata remain separate internally, and the existing global source/byte/line scan limits still apply.

Gzip reading is bounded before and after decompression. A compressed source larger than 8 MiB is not decompressed for a query, decompressed output is capped at 64 MiB, and only the bounded tail is parsed/searched. These limits prevent a gzip rotation from becoming an unbounded memory/decompression workload.

Per-vhost directory discovery now retains only the newest 256 matching JSONL/gzip candidates while scanning, and a combined log’s gzip-rotation scan retains only the newest 64 matches. Both caps apply before sorting, so directory size no longer causes a proportional memory allocation for the inventory. The usual 32-source query budget still applies. Filesystem enumeration still takes time proportional to the number of directory entries.

The reader remains bounded:

• up to 2 MiB of retained tail data from one source;
• up to 8 MiB across one query;
• up to 20,000 lines across a query;
• at most 32 sources in an all-source query;
• page size at most 500 and offset at most 10,000;
• search input at most 512 bytes and 32 parsed clauses;
• dynamic field discovery at most 64 leaf fields.

If more than 32 sources are eligible, the newest 32 are scanned and the response is marked truncated. File reads run in a blocking worker rather than the async proxy runtime.

The explorer is diagnostic rather than a retention/indexing system. RustProxy does not create rollovers or compress files itself; use operating-system log rotation for retention. Existing gzip rotations can be read as described above.