v0.31.5
This commit is contained in:
Mateusz Gruszczyński committed
1 parent
d942c1a35d
commit
75104f87ba
103 files changed
+1344
-370
No files matched your search
+9
-1
@@ -1,6 +1,6 @@
|
||||
# Stored log explorer
|
||||
|
||||
RustProxy 0.29.1 provides an authenticated, read-only **Logs** workspace at `/logs`. The top bar has one **Logs** menu with separate **Stored logs** and **Live Logs** actions; the two data paths remain independent.
|
||||
RustProxy provides an authenticated, read-only **Logs** workspace at `/logs`. The top bar has one **Logs** menu with separate **Stored logs** and **Live Logs** actions; the two data paths remain independent.
|
||||
|
||||
## Separation from Live Logs
|
||||
|
||||
@@ -45,6 +45,12 @@ At least one column must remain visible. Stored-log column selection is saved pe
|
||||
|
||||
Selecting a row highlights it and expands parsed field/value pairs directly below the row. A nested **Raw JSON** section shows the normalized complete record.
|
||||
|
||||
## Repeated-value analysis
|
||||
|
||||
The **Repeated values** tab (advanced parser) can count repeated values for any parsed field returned by the explorer. Typical security-investigation fields are `user_agent`, `client_ip`, `host` and `path`. Counts are calculated across the bounded matching scan before pagination, so changing the page does not change the count set. Minimum repetitions and top-result count are configurable within server limits. Values in the result table are directly copyable.
|
||||
|
||||
Aggregation does not remove the Stored Logs safety bounds: at most 4,096 distinct aggregate values are retained and at most 200 results are returned. The response marks an aggregate as truncated when a cardinality or result cap is reached.
|
||||
|
||||
## RAW error logs
|
||||
|
||||
RustProxy's built-in RAW error-log format is also normalized for browsing. The explorer extracts timestamp, level, target, message and recognizable `key=value` fields, while retaining the original line in `raw`.
|
||||
@@ -65,6 +71,8 @@ The browser still never supplies a filesystem path. The server derives sources f
|
||||
|
||||
Logrotate-style gzip rotations are discovered next to configured combined access/error files and inside the configured per-vhost directory. Only regular non-symlink files matching the active filename rotation pattern (for example `access.jsonl.1.gz`) are eligible. The browser still refers to them by server-generated source IDs rather than paths.
|
||||
|
||||
For per-vhost logging, the active `vhost.jsonl` and matching `vhost.jsonl.*.gz` files share one logical group ID. The panel enables **Consolidate rotated logs per vhost** by default, so selecting that vhost searches the active and compressed members together as one logical source. Disable the switch to inspect individual files. Physical source IDs and metadata remain separate internally, and the existing global source/byte/line scan limits still apply.
|
||||
|
||||
Gzip reading is bounded before and after decompression. A compressed source larger than 8 MiB is not decompressed for a query, decompressed output is capped at 64 MiB, and only the bounded tail is parsed/searched. These limits prevent a gzip rotation from becoming an unbounded memory/decompression workload.
|
||||
|
||||
Per-vhost directory discovery now retains only the newest **256** matching JSONL/gzip candidates while scanning, and a combined log’s gzip-rotation scan retains only the newest **64** matches. Both caps apply before sorting, so directory size no longer causes a proportional memory allocation for the inventory. The usual 32-source query budget still applies. Filesystem enumeration still takes time proportional to the number of directory entries.
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
# RustProxy 0.31.5 implementation and verification
|
||||
|
||||
## Scope
|
||||
|
||||
0.31.5 adds an operator workflow from traffic/log inspection to a bounded User-Agent blocklist without changing SQLite schema 9 or existing saved configuration requirements.
|
||||
|
||||
## User-Agent blocking
|
||||
|
||||
`user_agent_blocking` is available globally and on each vhost. Both levels use the same bounded policy:
|
||||
|
||||
- disabled by default;
|
||||
- at most 128 patterns, 256 bytes per pattern and 16 KiB total pattern text;
|
||||
- case-insensitive ASCII matching;
|
||||
- a pattern without `*` matches a substring;
|
||||
- `*` is a wildcard and may be used at the start, middle or end;
|
||||
- a request with no `User-Agent` header is not denied by this rule;
|
||||
- global and per-vhost policies are cumulative.
|
||||
|
||||
The check runs after vhost lookup but before redirects, cache/origin processing and before an upstream is opened. A match returns HTTP 403. The global policy is edited under **Settings -> Security**; the vhost policy is edited under **Vhosts -> Security**.
|
||||
|
||||
This is an operator blocklist, not a browser identity/authentication mechanism. User-Agent values are client-controlled and can be changed trivially.
|
||||
|
||||
## Traffic copy workflow
|
||||
|
||||
Traffic dimension labels are rendered as copy controls. The copied value is the raw displayed dimension value rather than a formatted count, so IP addresses, User-Agent values, paths, hosts, methods, status values and other traffic dimensions can be pasted directly into investigation notes or configuration fields.
|
||||
|
||||
## Stored Logs repeated-value aggregation
|
||||
|
||||
`GET /api/file-logs/query` accepts optional `aggregate_field`, `aggregate_min_count` and `aggregate_limit` parameters. Aggregation is computed across matching records in the same bounded scan used by Stored Logs, before page offset/limit are applied.
|
||||
|
||||
The field may be a canonical field such as `user_agent`, `client_ip`, `host` or `path`, or another parsed/nested field accepted by the Stored Logs field resolver. Scalar strings, booleans and numbers are counted; bounded array scalar values are counted individually; objects are not flattened into invented text values.
|
||||
|
||||
Safety limits are retained: the scan remains limited to 8 MiB/20,000 lines/32 sources, aggregate output is at most 200 rows, and aggregate cardinality is capped at 4,096 distinct values. The response reports when aggregation was truncated by those limits.
|
||||
|
||||
## Consolidated per-vhost Stored Logs
|
||||
|
||||
Each per-vhost active JSONL file and matching `.jsonl.*.gz` rotation receives the same logical `group_id`, for example `vhost:example.com`. Individual physical files still have unique source IDs and metadata.
|
||||
|
||||
When `consolidate_vhosts=true` and `source=vhost:<id>`, the query scans all discovered members in that group subject to the existing global source/byte/line bounds. The panel enables this behavior by default and presents one logical vhost option with the member count and summed discovered size. Turning consolidation off exposes the physical files separately again.
|
||||
|
||||
Combined access/error logs keep their existing source behavior; this switch only changes per-vhost grouping.
|
||||
|
||||
## Compatibility and security
|
||||
|
||||
- Existing configuration files deserialize with `user_agent_blocking` disabled through serde defaults.
|
||||
- No SQLite migration or cache-format migration is introduced.
|
||||
- Stored Logs still accept server-generated source IDs only, never arbitrary filesystem paths.
|
||||
- Existing symlink rejection, gzip input/output caps and blocking-worker isolation remain in force.
|
||||
- User-Agent denial occurs before origin contact, so a blocked request cannot cause a new upstream connection.
|
||||
|
||||
## Verification status
|
||||
|
||||
Rust-independent checks cover language packs, JavaScript syntax/UI source contracts, OpenAPI/source-version consistency, Stored Logs grouping/aggregation source contracts and existing migration/release smoke tests. The source also contains Rust unit/regression tests for User-Agent matching and Stored Logs aggregation/grouping.
|
||||
|
||||
The validation environment used for this source package does not contain Rust/Cargo. Therefore `cargo fmt`, `cargo clippy`, Rust compilation and Rust unit/integration execution remain a release gate. Run `./dev.sh check` and `./dev.sh test` on the supported Rust toolchain before production deployment, then verify real HTTP 403 behavior, vhost/global precedence, clipboard actions and gzip consolidation against representative production logs.
|
||||
+2
-1
@@ -31,6 +31,7 @@ This is the canonical index for the current project documentation. The authentic
|
||||
- [0.26.3 implementation note](V0.26.3_IMPLEMENTATION.md) - global interface scaling, chart themes, square draft dock and Settings navigation polish.
|
||||
- [0.26.2 implementation note](V0.26.2_IMPLEMENTATION.md) - unified Logs navigation, field-query search, dynamic JSON columns and RAW parsing.
|
||||
- [0.26.1 implementation note](V0.26.1_IMPLEMENTATION.md) - historical vhost-aware protocol logging and Stored Logs UX milestone.
|
||||
- [0.31.5 User-Agent blocking and consolidated log analysis](IMPLEMENTATION_0.31.5.md) - global/per-vhost UA blocklists, Traffic copy actions, repeated-value aggregation and per-vhost gzip consolidation.
|
||||
- [0.31.4 administration public Origin and settings](IMPLEMENTATION_0.31.4.md) - opt-in HTTPS origin, one-field DNS allowlist, additional hosts in advanced settings, legacy deduplication, CSRF and remaining deployment checks.
|
||||
- [0.31.3 mobile session / invalid-Origin recovery](IMPLEMENTATION_0.31.3.md) - deterministic BFCache and visibility session coordination, 403 diagnostics, recovery and remaining acceptance.
|
||||
- [0.31.1 mobile panel bootstrap](IMPLEMENTATION_0.31.1.md) - panel script readiness, versioned assets, failure recovery and acceptance tests.
|
||||
@@ -83,7 +84,7 @@ This is the canonical index for the current project documentation. The authentic
|
||||
|
||||
## Historical or version-labelled context
|
||||
|
||||
Version-labelled `V*.md` files are retained as historical implementation/audit records. Their version number describes the audited milestone. The current release implementation note is [IMPLEMENTATION_0.31.4.md](IMPLEMENTATION_0.31.4.md); [IMPLEMENTATION_0.31.3.md](IMPLEMENTATION_0.31.3.md) is retained as the previous mobile recovery milestone; [IMPLEMENTATION_0.31.1.md](IMPLEMENTATION_0.31.1.md) is retained as the prior mobile milestone; [IMPLEMENTATION_0.31.0.md](IMPLEMENTATION_0.31.0.md) is retained as the previous milestone; [V0.29.1_IMPLEMENTATION.md](V0.29.1_IMPLEMENTATION.md) is retained as historical context, and [V0.29.0_IMPLEMENTATION.md](V0.29.0_IMPLEMENTATION.md) is retained as the prior redirect/health/cache/log milestone; [V0.28.2_IMPLEMENTATION.md](V0.28.2_IMPLEMENTATION.md) is retained as the prior certificate-hardening milestone; [V0.28.1_IMPLEMENTATION.md](V0.28.1_IMPLEMENTATION.md) is retained as the prior external-path milestone; [V0.28.0_IMPLEMENTATION.md](V0.28.0_IMPLEMENTATION.md) is retained as the prior GeoIP/listener/cache milestone, and [V0.27.3_IMPLEMENTATION.md](V0.27.3_IMPLEMENTATION.md) is retained as the prior HTTP/3/cache-maintenance milestone, [V0.27.2_IMPLEMENTATION.md](V0.27.2_IMPLEMENTATION.md) is retained as the prior managed-ACME repair, [V0.27.1_IMPLEMENTATION.md](V0.27.1_IMPLEMENTATION.md) as the prior managed-runtime/list-sorting milestone, and older version notes remain historical evidence. The latest retained audit is [V0.25.5_AUDIT.md](V0.25.5_AUDIT.md); older audit files remain available for traceability.
|
||||
Version-labelled `V*.md` files are retained as historical implementation/audit records. Their version number describes the audited milestone. The current release implementation note is [IMPLEMENTATION_0.31.5.md](IMPLEMENTATION_0.31.5.md); [IMPLEMENTATION_0.31.4.md](IMPLEMENTATION_0.31.4.md) is retained as the previous administration-Origin milestone; [IMPLEMENTATION_0.31.3.md](IMPLEMENTATION_0.31.3.md) is retained as the previous mobile recovery milestone; [IMPLEMENTATION_0.31.1.md](IMPLEMENTATION_0.31.1.md) is retained as the prior mobile milestone; [IMPLEMENTATION_0.31.0.md](IMPLEMENTATION_0.31.0.md) is retained as the previous milestone; [V0.29.1_IMPLEMENTATION.md](V0.29.1_IMPLEMENTATION.md) is retained as historical context, and [V0.29.0_IMPLEMENTATION.md](V0.29.0_IMPLEMENTATION.md) is retained as the prior redirect/health/cache/log milestone; [V0.28.2_IMPLEMENTATION.md](V0.28.2_IMPLEMENTATION.md) is retained as the prior certificate-hardening milestone; [V0.28.1_IMPLEMENTATION.md](V0.28.1_IMPLEMENTATION.md) is retained as the prior external-path milestone; [V0.28.0_IMPLEMENTATION.md](V0.28.0_IMPLEMENTATION.md) is retained as the prior GeoIP/listener/cache milestone, and [V0.27.3_IMPLEMENTATION.md](V0.27.3_IMPLEMENTATION.md) is retained as the prior HTTP/3/cache-maintenance milestone, [V0.27.2_IMPLEMENTATION.md](V0.27.2_IMPLEMENTATION.md) is retained as the prior managed-ACME repair, [V0.27.1_IMPLEMENTATION.md](V0.27.1_IMPLEMENTATION.md) as the prior managed-runtime/list-sorting milestone, and older version notes remain historical evidence. The latest retained audit is [V0.25.5_AUDIT.md](V0.25.5_AUDIT.md); older audit files remain available for traceability.
|
||||
|
||||
Older context documents such as [INTEGRATIONS_040](INTEGRATIONS_040.md), [MANAGEMENT](MANAGEMENT.md), [SECURITY_AUDIT](SECURITY_AUDIT.md), [ACME_RFC2136](ACME_RFC2136.md) and older version audits remain available for traceability but should not override the current functional/technical documents above.
|
||||
|
||||
|
||||
@@ -217,3 +217,9 @@ revision independently; the obsolete hide_mock_origin field is not accepted. X-M
|
||||
removed using ordinary response-header rules. Scope labels on listeners never
|
||||
enforce network restrictions. See ACME_RFC2136.md for private TSIG/account storage
|
||||
and the separate backup requirement for manual DNS-01 issuance.
|
||||
## User-Agent blocklists
|
||||
|
||||
RustProxy can apply an optional global User-Agent blocklist and an additional per-vhost blocklist. Matching is case-insensitive; plain text matches a substring and `*` is a wildcard. Policies are bounded and disabled by default. They are evaluated before opening an upstream; a match returns HTTP 403. Requests without a User-Agent are not denied by this rule.
|
||||
|
||||
User-Agent is client-controlled and easy to spoof, so this is an operational noise/blocking control rather than an authentication or identity boundary. Keep ACLs, authentication and rate limits as the authoritative controls.
|
||||
|
||||
+25
-1
@@ -1,4 +1,28 @@
|
||||
# Verification report - RustProxy 0.31.4
|
||||
# Verification report - RustProxy 0.31.5
|
||||
|
||||
## 0.31.5 follow-up - Stored Logs navigation polish
|
||||
|
||||
- Split Stored Logs into **Log entries** and **Repeated values** tabs, sharing source/search filters; the count query is only included while the analysis tab is active. Move the persisted consolidation toggle into the header. Verify via dedicated Node UI source regression. Rust compilation/browser acceptance remain outstanding.
|
||||
|
||||
## 0.31.5 follow-up - build-specific JS/CSS filenames
|
||||
|
||||
- `scripts/build/swagger_assets.py` creates a fresh random 64-bit hex identifier using Python `secrets` when Cargo invokes the build script. The identifier is embedded in the binary and substituted into dashboard, public login and Docs HTML. HTML retains its existing cache-header behavior. Matching fingerprint paths serve the existing embedded asset bytes without runtime disk access; unrecognized fingerprints return 404. Legacy URLs remain for compatibility. Package remains 0.31.5; rebuilding with unchanged source changes these asset URLs whenever the build script reruns.
|
||||
- Regression checks cover URL templates, version independence, build token generation, asset routing and anonymous allowlist. Source/UI checks run without Cargo; compilation and real browser/cache tests remain open.
|
||||
|
||||
## 0.31.5 follow-up - Stored Logs consolidation preference
|
||||
|
||||
- Stored Logs now restores the last selected consolidation mode from browser-local storage (`rp-file-logs-consolidate-vhosts`) on panel load; saves changes immediately; defaults to the existing checked mode if no preference is present or storage is blocked. Clearing browser storage restores the default. No API or SQLite change.
|
||||
- Verification: `security_logs_0315_ui_test.mjs` checks preference read/write integration. Rust compile and runtime acceptance remain open as described below.
|
||||
|
||||
## 0.31.5 - User-Agent blocking and consolidated log analysis (2026-10-11)
|
||||
|
||||
- **Scope:** global and per-vhost User-Agent blocklists, copyable Traffic values, Stored Logs repeated-value aggregation and optional per-vhost consolidation of active JSONL plus gzip rotations.
|
||||
- **Compatibility:** no SQLite migration; existing configurations load with User-Agent blocking disabled. Existing Stored Logs physical source IDs remain available when consolidation is disabled.
|
||||
- **Security:** User-Agent matching is bounded and runs before upstream contact; Stored Logs continue to accept only server-derived sources with existing symlink/gzip/scan limits. User-Agent is client-controlled and is not treated as an identity boundary.
|
||||
- **PASS — full Rust-independent release run:** `./dev.sh test` passed module/version/source contracts, SQLite schema-9 migration/upgrade smoke, security/dependency checks, the 300,000-PURGE memory model, 2,000 HTTP requests to the Python test peer, every bundled Node UI/simulation test (including `security_logs_0315_ui_test.mjs`), documentation and both language packs. It then exited **127** only at the missing Rust/Cargo guard; the Python peer is not a RustProxy binary.
|
||||
- **PASS — focused 0.31.5 regression:** `security_logs_0315_ui_test.mjs` verifies global/per-vhost UA wiring and pre-origin ordering, Traffic copy controls, Stored Logs aggregation/consolidation markers, OpenAPI and English/Polish locale keys.
|
||||
- **PASS — other available checks:** `go test ./...` in `docker-agent`; `bash -n dev.sh`; Node syntax checks for the modified UI files/test; OpenAPI JSON parsing; `docs_check.py`; `langs_check.py`; `check_source.py`; and `rust_module_contracts.py`.
|
||||
- **Rust release gate OPEN:** `./dev.sh check` exits **127** because this environment does not provide `rustc`, `cargo` or `rustfmt`; new Rust unit/regression tests are present but cannot be compiled/executed here.
|
||||
|
||||
## 0.31.4 reissue - origin protection and one-field administration public URL (2026-10-11)
|
||||
|
||||
|
||||
Reference in new issue
Block a user