diff --git a/AGENTS.md b/AGENTS.md index 1db637b..676868b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,7 +2,7 @@ Read README.md, ROADMAP.md and docs/TEST_REPORT.md before making changes. The project name is RustProxy; the Cargo package/binary remains `rust-proxy` for -compatibility. Current source version: 0.31.4. +compatibility. Current source version: 0.31.5. ## Delivery diff --git a/CHANGELOG.md b/CHANGELOG.md index 0cdebbe..de88f70 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,20 @@ > Operator-facing release history. Historical engineering notes are kept in > [`docs/CHANGELOG_TECHNICAL_ARCHIVE.md`](docs/CHANGELOG_TECHNICAL_ARCHIVE.md). +## 0.31.5 + +- Give repeated-value analysis its own Stored Logs tab and move the rotation-consolidation option to a compact top-right control. Preserve the browser-local consolidation choice. + +- Fingerprint panel, login and documentation JS/CSS asset filenames with a random 16-hex build identifier independent of the application version. Reject legacy JS/CSS URLs (404), retaining unversioned JSON/SVG; enforce exact fingerprints and preserve the existing HTML cache-header behavior. No SQLite/API changes. + +- Remember the Stored Logs consolidation toggle per browser with localStorage, preserving the existing default and avoiding API/database changes. + +- Add global and per-vhost User-Agent blocklists with bounded, case-insensitive substring/wildcard matching before upstream connection. Existing configurations remain valid and the feature is disabled by default. +- Make Traffic values copyable for faster investigation and blocklist preparation. +- Add repeated-value aggregation to Stored Logs for User-Agent, client IP and arbitrary parsed fields, with bounded result/cardinality limits. +- Let Stored Logs consolidate an active per-vhost JSONL file and its gzip rotations into one logical source. Preserve individual file metadata/source IDs and existing combined access/error behavior. +- Update English/Polish UI, OpenAPI and release documentation. SQLite schema 9 is unchanged; Rust/Cargo verification remains outstanding in this environment. + ## 0.31.4 - Correct the administration panel's handling of intermittent `auth.invalid_origin` with optional explicit HTTPS `administration.public_origin` when behind a trusted reverse proxy. Allow a bounded, HTTPS-first multi-hop forwarded-proto chain without trusting arbitrary forwarding headers. diff --git a/Cargo.lock b/Cargo.lock index f2fc5f2..ec475a2 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1881,7 +1881,7 @@ dependencies = [ [[package]] name = "rust-proxy" -version = "0.31.4" +version = "0.31.5" dependencies = [ "anyhow", "arc-swap", diff --git a/Cargo.toml b/Cargo.toml index f15717d..f12d48d 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "rust-proxy" -version = "0.31.4" +version = "0.31.5" edition = "2021" rust-version = "1.99" description = "RustProxy - independent reverse proxy with local administration" diff --git a/README.md b/README.md index 86af2b4..19d31b6 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,11 @@ -# RustProxy 0.31.4 +# RustProxy 0.31.5 **0.31.0 panel/cache additions:** browser-negotiated Zstd/Brotli/gzip for HTML, JS and CSS (Settings → Administration); optional HTTP BAN/PURGE restricted to explicit IP/CIDR allowlist (Settings → Cache, disabled by default); System information Refresh button beside the API badge. See [Settings](docs/SETTINGS.md) and [Cache](docs/CACHE.md). Independent Rust reverse proxy and static Web Server with a dashboard, SQLite configuration, streaming HTTP/TLS/WebSocket forwarding, raw TCP/UDP stream hosts, IP-bound vhosts and backend pools. -**0.31.4 administration Origin pinning and simplified one-field configuration (Cargo verification required).** - -**Reissued source package:** this package supersedes the earlier 0.31.4 archive under the same version number. Update/version checks cannot distinguish the two by version alone; rebuild/redeploy explicitly and verify the package SHA-256. +**0.31.5 User-Agent blocking and operator log-analysis workflow (Cargo verification required).** ## Layer-4 streams @@ -15,7 +13,16 @@ Raw TCP/UDP forwarding has its own **Streams** page in the primary navigation, a Named HTTP/HTTPS listener bind addresses use the same chip-style editor as vhost hostnames, including multi-address listeners. -## What changed in 0.31.4 +## What changed in 0.31.5 + +- Add bounded **User-Agent blocking** globally under Settings → Security and per vhost under Vhosts → Security. Matching is case-insensitive; plain text matches a substring and `*` is a wildcard. Global and per-vhost rules are cumulative and run before opening an upstream. +- Make Traffic table/chart values directly copyable, including User-Agent, IP, host/path and other displayed dimensions, to speed up blocklist creation and investigation. +- Add **Advanced parser · repeated values** in Stored Logs. It counts repeated values for `user_agent`, `client_ip`, `host`, `path` or any supported parsed field across the bounded matching scan before pagination. +- Add a default-enabled **Consolidate rotated logs per vhost** switch. Active `vhost.jsonl` and matching `vhost.jsonl.*.gz` rotations are queried as one logical vhost source while retaining individual source IDs internally. +- Preserve SQLite schema 9 and existing saved configurations through serde defaults. See [Stored Logs](docs/FILE_LOGS.md) and [0.31.5 implementation and tests](docs/IMPLEMENTATION_0.31.5.md). +- **Release gate remains open:** Rust/Cargo compilation and real proxy/browser acceptance are still required before production deployment. + +## Previously in 0.31.4 - Fix intermittent `auth.invalid_origin` on the **administration panel only** behind a trusted reverse proxy. The optional `administration.public_origin` pins the browser-facing HTTPS origin and allows a bounded `X-Forwarded-Proto: https, http` chain when the immediate proxy peer is trusted. Without a pin, ambiguous forwarded-proto values still return HTTP 403. - Set **only** `administration.public_origin`, for example `https://panel.example.test`, in Settings → Administration. Its DNS hostname is automatically included in the panel Host allowlist; it does **not** need to be repeated in `allowed_hosts`. @@ -24,6 +31,7 @@ Named HTTP/HTTPS listener bind addresses use the same chip-style editor as vhost - Restart RustProxy after modifying the setting. See [Administration HTTPS](docs/SETTINGS.md) and [0.31.4 implementation and tests](docs/IMPLEMENTATION_0.31.4.md). - **Release gate remains open:** Rust/Cargo compilation, resolved dependency lockfile and real mobile/reverse-proxy acceptance are still required before production deployment. + ## Previously in 0.31.3 - Correct mobile tab/BFCache recovery: share in-flight session validation across resume events and defer SSE/polling until verification succeeds. A failed check keeps a visible retry UI instead of reloading the page, and 401 still requires a fresh login. diff --git a/ROADMAP.md b/ROADMAP.md index ac4c9dc..47c9959 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,5 +1,14 @@ # RustProxy roadmap +## 0.31.5 - User-Agent blocking and consolidated log analysis + +- [x] Add bounded global and per-vhost User-Agent blocklists and enforce them before upstream contact. +- [x] Add one-click copy for Traffic values used during investigation. +- [x] Add repeated-value aggregation for arbitrary parsed Stored Logs fields. +- [x] Consolidate active and rotated gzip per-vhost logs into one logical source on demand. +- [x] Update OpenAPI, translations, documentation and Rust-independent regression checks. +- [ ] Resolve Cargo dependencies, compile/test Rust and perform real browser/proxy acceptance before production deployment. + ## 0.31.4 - Administration-only public HTTPS Origin with one-field configuration - [x] Add explicit HTTPS `administration.public_origin` and restrict its use to trusted reverse-proxy peers for panel requests. diff --git a/SHA256SUMS b/SHA256SUMS index 7e99d5b..11a804a 100644 --- a/SHA256SUMS +++ b/SHA256SUMS @@ -1,18 +1,18 @@ 61bf40a0572889dc8b0dda9eae4e93e44b0de9364b3c798106c85700ffb37c23 .dockerignore 4140a2956264b44f259e6a37fc00ba02097bbe68b09296e04d3d243c1287129b .env.example -9e6356475bf587091bab72246ec9e325b498f7e48a6220e9a18baa50d0a385a6 .gitignore -ce2dfe3ddaf72f4d9f47fbddde8cc3a6379c95797785bbafdbc9f505ab0c06d7 AGENTS.md -3868e2b5ac1e1e324e0647da39dd4ace965b102fdd06a26d720977cb7bcc4ba2 CHANGELOG.md -140089a442d329eb0819f17bbb8e9215687ede43d77fb74068c1b4e79b074547 Cargo.lock -d8e3b968089f11eac6098bb8f51598a9012de6ec2f84a0ef6899ac27cdf2a134 Cargo.toml +4a19a62f235c7f520b68411c32dc4930a8aa58c3d91d4a12fd83b8b9344db84d .gitignore +095ff293a2250958aa9877e5f6b299018b412fc26556a355e72c3f782244279f AGENTS.md +bc7ed26f595df6df1f037a43ea3f8eeb01dd7c4ddcd068950d54b8b69af6d985 CHANGELOG.md +9b943bc748d57182cb96b084f5d2e54d516310e37f5d0004607f4df24b398c14 Cargo.lock +20fe56f1a563ef54179f0050bb1a708dee595f8763261c953965bb3f95277f5b Cargo.toml 25c7f3c8162cbf5cc156439774118a81dae9fdc52b611ec7e40c230fc99a772c Dockerfile -6c1cddc4650a1bd9e31b3598baffb0d55435a2d3a5e31653c111aacd4ed96070 README.md -ea6203f84882ab99aa22b99dd26d0d7b8d7fd64639aa74230ee431b3edac25ae ROADMAP.md +fac9ad52b9cb6f506988f6be8de88a2a2e2f288319f2f01b7849a8e053d32359 README.md +aa94ea36f16a56ace163895b1a5fb01ba9a45ad44432cf45a2d884b89436df2e ROADMAP.md da8311e3db945290666324a2f24bf4521b365272d7ac3f862ded1be993eea646 UPDATE_CHECK.toml f645f218dd062bf6cb54a8ee634290562cad364a4cccf09f16e3c8303582af8c build.rs ea8d091843f86f4c66a5c76b20268b2feb397070851b6e05c5bc7d7a8d9157eb deploy/README.md 4834c278430ddf3bf842995735195368f1ac2c3b5c1d749df4f9238caf1e2b25 deploy/rustproxy.service -7e9d5cf9a0720ac00a87a177b41c44524ed6b04e819343ea16967945b3a413ae dev.sh +b395a672fff045c1700858c2af21ea75aadce41877b0126271a6977cd3c00e7c dev.sh 5f51efe621f45512a466fa7f2656867c6502a135ca040f4bfeec06a6919b39b6 docker-agent/README.md 7fee7d51a7deb67e73e53a0bdff3aa4630aea9bab9b23a0f9b8001b2169892b1 docker-agent/certificate.go 5e4f565c3c9de4dba897af2e801ea646a83395e72eae4717a6d502dff2fcec10 docker-agent/docker-agent.env.example @@ -37,12 +37,13 @@ c8e69e492179ce39958cc63615cf46c4ac08cfdb7f3483aaab66868a3abcd687 docs/CONTAINER befa50579e3475b22589f26380d85b3345b80e4fa947d49649a3b91a8dd8b8c2 docs/CUSTOM_LOCATIONS.md 16d8f59908c66e595d6003495cbf5c068c68b10aa9495eb8612c38b2de0e4a7c docs/DOCKER_DISCOVERY.md 5324b215075214a5bb9b388bc82d6491eacfc507df95856d1aecedd184e25b80 docs/ERROR_LOGGING.md -35b7784afa9a2f7d805f7cd1bd188f93050fc0ebaea61b30152075fc7a2549bb docs/FILE_LOGS.md +ac855e78d546b5abe3215b93c0510050044b98c7ef593ac2405614999b2493cb docs/FILE_LOGS.md 8dfc89a8b3512103b87864bd7c24a3dc3de4465edf8f9874e6bed1f4634dd930 docs/HTTP3.md 5301e321611d494fd97b04ab648df1f66a6a0d1f1f4c3475eab485e138d5605b docs/IMPLEMENTATION_0.31.0.md eb52e72325e1307d05cb04878b6d22dfedea491908c5f36110b32421765096c8 docs/IMPLEMENTATION_0.31.1.md 991ff3c09ca1e2bcf73b8fa2975405b955dd640c3f1e1246e75038c043c17183 docs/IMPLEMENTATION_0.31.3.md db2087608a1225c56e42c416d949b777fb15b1ed8b69a1e71fc2a6af6750a1e5 docs/IMPLEMENTATION_0.31.4.md +ab2633deb3ad3c6bad056970363d3227213f919915a2298d9549e8029b5e0c78 docs/IMPLEMENTATION_0.31.5.md a00ec3c1d3b28ba10677435128e031e5f18fe8c5cef2ac90eb0f6779cefeb434 docs/INSTALL.md 3b5c31c805e5aee0dc2100b81246a9b631b5abb996faa598ae94f87ba39305f3 docs/INTEGRATIONS_040.md 4bb56add544f15d54695cf797cd16a51c09024360cdeef803a88988f4a54b3a5 docs/LIVE_LOGS.md @@ -56,18 +57,18 @@ e7516646dfb107d6240370f4e7460b1189f2eb21f398740381d5d102847bbfe6 docs/NOT_FOUND e68d5a5cf0991de9308f78afb0264e02979f7463a5f395eb7a6e9ec2bd3a3d54 docs/PHP_FPM.md bc068a176056f5ea4d3b637071e644ed58a2140494c233ef22aa66e9700e3299 docs/PHP_FPM_ISOLATION.md 572c70fbd2f61fecd80a8ff9fa5de4a94c4f158de5e90da8e920f99b051241fa docs/PROMETHEUS.md -1abcf8b9abcff02cf40c0f3e5597af1e1bf3e6d48c2f7f4278a53c6ebad51f5b docs/README.md +4c6909970419f336735106d30af685010d63ab56b7721b794be219e62ab86060 docs/README.md 6e46a137b9c88dac8cb9e3fdca877430092392de9913d746278dbff8652460c2 docs/REDIRECTS.md 0dfd0d431c5aedc67abac1ce5de040ffa790ec9578fd7d6f91a987c333d73358 docs/REFERENCES.md d85c5567fc6a7714b96ad7cfe4ee9ff2085f84c4f31534a6484340d03a73e0f7 docs/RENEWAL.md a2bb837cdd3559d58158880c1f832badf64035376a3623050bb4e40c2bca104c docs/ROADMAP_HISTORY.md 759bf2987f2b602488829d1856fa33d57c2a3b23bc681ef5a931b9633af3a533 docs/SECRET_STORE_BACKUP.md -5dbdf45fddba198a28f511dbbd2f12bf7d2b88c41c3cbedbc20dfea3858779d3 docs/SECURITY.md +ec7e32fd19e6ba6ed1732f1f3dd92ac0860609e8c0bbd3c1998840cdfb24bf54 docs/SECURITY.md 620b69e566d27af04f0604f4ca3f88cf95cdfea7fbc6ae9dd92446d2b2b5d9be docs/SECURITY_AUDIT.md 007537ee971463072b2482dfa4fdcf1955f837ced0b711d4eef3b6267d16d4f6 docs/SECURITY_AUDIT_0.31.0.md 970025a6d1a37452b8bead6c9d2b3d4743ea602251dc446d575531d76c11ae83 docs/SETTINGS.md 3611defd206cde107e75cac006159856d12cd8f15df7ae2eb9ec3592ed2b4abd docs/STREAMS.md -d7cd61b534a2d38511d212cd9ee906b16b28d99d4a7ba5ed6ef7c1ab3611d2b8 docs/TEST_REPORT.md +686377f81a4169e08599d50f9decfb18a4a084ce4d808aa77357deb8824cf2ba docs/TEST_REPORT.md cfb452d8496839e832b5fbacdd4e52525b0ecc3112fbb62340c1e580bd5c1717 docs/TODO.md 751c9b13b47d0959a8575d6e82bdecba988b950a374ba0b084fbba08b4b65ebe docs/TRANSFORM.md 660687fc961f71aae2bdf71595aa47a336fa782f4e78ceee8c1d1e63e46393ee docs/TRANSPORT.md @@ -139,8 +140,8 @@ cb01fdef286f38b442dd9252c23c4f8e44eb72e9ecdd99adb926e8e9f6318bb1 integrations/r 96563a83e87cfb33e238c2a970ce36b1a1b51e91318d09b123c3755002ba2646 integrations/routeros-oci-build/build-routeros-oci.sh 3f267cafa16006e5456ade78ee2d5fae02cc7ce747839523068ffd8d7c98798a integrations/routeros-oci-build/docker-entrypoint.sh 71cde48daa538431dfac51e438b1ed23b061abfdb4e144ba704237d75ac6270c langs/README.md -61337803fabb183629fa4dc9ff038ee7ee41956c8304de3d0122d425d0b1f53a langs/en_US.json -949dc843e348ada1b1208407daa13463a0e7e16548411376382e18635578dcbb langs/pl_pl.json +e0451bcd6dc17d1c0bd5becb0a5a35617df21d2f851982a64831e9f62c13bc17 langs/en_US.json +10967c073e750d9f66f60bd0e2e89dbfb77bb4c5156ee694ffe6e86a7b76ccfa langs/pl_pl.json a99ad040be10fa028351e290b9363236b3513a1cadbdc9fb9e0260d517b9b0ae make_zip.py e830d37dd1f81a4b0aa7d535f5bba69e34e433b0fd99cc8ee0b296786c32379a migrations/0002_admin.sql 20112cc82192098b8d602f1c4f7ca08712e9e1366c845fabb92753187c370d35 migrations/0003_certificate_renewals.sql @@ -158,7 +159,7 @@ ee50f38c037765fab742e888d4b9fc4d874c31c785c6064d4eca3d2c4d24d944 nagios/check_r 6e036f736a2b40feb78eec17bf34871771524190f283591d1c89e48e6d2271e6 nagios/check_rustproxy_queues 7458b2d74495e6fb1c1edeea4d5a793705476431dadb60627187111f91441e13 rust-toolchain.toml 2145ad17ae7a823756371ceab6bdbeb332211df93193ffb4e486f7a80c4f29ae scripts/README.md -a1f55872201b5178f8af5e2beee6b3a68e37026831e41426ee62a8c15c1cf333 scripts/build/swagger_assets.py +31953ddf3fda1c9b47fb5de7f85b91a23782b4ccc9555b8423a965659a0a8786 scripts/build/swagger_assets.py e4d15aa6aa864d04b9b30aca33826b833632af02ca643f8b23e219a5ea1298dd scripts/deploy/deploy.py cc268d9d3dd125c9cbff2522fe3137a561fb4fcbd5ee461d70ef2e3a8241cdb1 scripts/deploy/deploy_core.py 8c7da13e95bcddd2bd1b4f1b22ade1c5ac1ecf54448601daa8416905d4239751 scripts/deploy/deploy_runtime.py @@ -166,7 +167,7 @@ a641bd35400c56f3b68b0a92e97b602d06090f03972ca8f01fa744e90dd97403 scripts/deploy 2cc658faaa610b21b48942c59b955525539ad4503fd60f2a49c7e626a22faaff scripts/migration/migrate_npm.py a459d95bbb0076f33ade8b273b813308167f9d8d45242b309a2d18320ca0093c scripts/migration/migrate_npm_api.py 31ccad3ec3c880119f3b7002f0c6652f8b43634edad8ab3763cb0e4497d0bb1f scripts/migration/migrate_npm_core.py -abf438c3cb5a9d0dcc1351a541d3d399f3e963fc793ceb1e4708beaf3f7523a9 scripts/resolve_dependencies_0312.sh +9afb0d704e1770c31913e968014adc35dbfdeb40906235b7868f22a4562e3b71 scripts/resolve_dependencies_0312.sh 20124173452b4514bad8850c1637569a9a191882478451e549bd16b432605fc1 scripts/runtime/acme_provider.py e69813fb1d3a4cdcfb7243964cc323fc21a603087bf2e84188fa42b9d3cee4b5 scripts/runtime/acme_rfc2136.py 22e880e98c7b9d700c7a4c02b9b1d543e5dbd1f4ecdca42813a1083165b32a19 scripts/version.py @@ -183,24 +184,25 @@ f8075a5ef9ae5651d72a33ac0f0fd5b44ad0ae30f6e2d2318412177e0ed837f5 src/acme/runti e38331a1b186b49fe9d04051de52b603e1fa028def37e4e2f52e386b4d7b29ac src/acme/tests.rs b6f38f3fd75d6685bbc882ed1077a38bf495cfcf01c397f69ed9b98be68b446f src/admin/access.rs 019f4aa33e3596bc9c11ef8cf5703bcfffe46c15288c0eba83c8dd9e38055fb2 src/admin/certificates.rs -1de4c7c116eeab2452ccccbaba3f2b7b63c6eb1c1e146ab870d19c5a5ca16f80 src/admin/documentation.rs -efa4a357d51d9b3c575e64593b55c8a30afbab5cd96a44cc39786207228757b6 src/admin/documentation/documents.rs +5e150d4ef0413a2f92db5b6631669933b84d4b4796633d01476c4b39188cd05c src/admin/documentation.rs +bc782a5f5908d82c5dc509ca572d0c38c87201726769b1903bc15a78fc8dd7b8 src/admin/documentation/documents.rs 67b901485d5858d5b5bf78a6260035db32241531d99481754c3533fdf404facb src/admin/documentation/tests.rs -002922c91aa07cbea17cb0ee11127bb6e7e015de108ee032761b34507dcfa06d src/admin/file_log_search.rs -2ebaf24e9d0d87383d2eea073ee2d3ac37962eeb503a0f1ee2c2de4f3f6d1b7a src/admin/file_logs.rs -d7aa68a78a4cb5a61342d92c1f3209af230d0f9ca3fea1115f67078372e6f37c src/admin/file_logs_io.rs -899e87a73bfe46c4078a46f9e1eb1c17b384472e8a30d2bab5486872b43fc809 src/admin/logging_api.rs +1ce779b84aa8f47bdb3c34f83593c1f2f06d2235f0319847096cfcd035434168 src/admin/file_log_search.rs +b3aad0db47d930ee177253cb5d5eb772272452bab4385a34566b111c239a6302 src/admin/file_logs.rs +c6b552cd5acd9e743c28b1a5564420c214c619fe891ab0561b7d4499aae8d6c0 src/admin/file_logs_io.rs +d5eb4019a4aa6a4fdcad9310dce9076dcb85ca4640540c64e924367ba2276234 src/admin/file_logs_query.rs +7de7da489a367658b700f00776fa43604d3554500b5fbef4da2764e402e68afa src/admin/logging_api.rs 58802528128f53e493f7339063433a747b605007a8842b5efc670132d897d6f7 src/admin/mod.rs -3028fd407ec2cbda7742746241113739136a27a5b81993acb2d6f99187086246 src/admin/openapi.json +9c65356d79438aabf0adcee6c6b1f0c4236425af205430e06a12685c1be40eb5 src/admin/openapi.json 0d1a5324e8ad32f5ab0861b722663f50f066e256bc5fd5dcced6a63f2bd399d1 src/admin/operations.rs -2b0ec6df0ed9e1bdc5c9e4ff1758b59dd14abc150f98365ac3edcf4672908e48 src/admin/panel_assets.rs +30895e05c5d5d0a26be29f667e0f6d72f16a96a5c8a98e09cd11d8ef2d7b4763 src/admin/panel_assets.rs 75bbedd54fdabb2b489642a642ee7c6533f69cdacb85e251763a56ba11174d5c src/admin/panel_content.rs 7c113cd9d8642a808c73ebee24e695f7a318995e39dc2b81e747d2b2246e277f src/admin/preferences.rs d9e90166d2199152585a5208feedfc1dd1395dc32ddbd2a094ae4a1aee45e115 src/admin/preferences_columns.rs -9baf3c0ec4c543bda51605930ac57d6425c5e03e06d29ff755a7ce3ff611aeaf src/admin/public.rs -c9f952bec99dfe1d5a60998a38bd177f82f54de567230413e0c322ee561eba7c src/admin/public/tests.rs +055642f1239346616788f3b1da52f814de5ba0649316352459b6d84b0710eeea src/admin/public.rs +7a00e4ab943f74d2b1bdcbfab7f9cd3f584e9f59536b4aca6c4c31f5f5508d12 src/admin/public/tests.rs 6125419b099cf4d330767bc07424d52a2b400f273d31084c058c43a72a6427b7 src/admin/refresh_preferences.rs -765c14e52bdc9e542d6ff651c250c24d33b77b7e46f1b5f19acf96cc2c99bdc8 src/admin/router.rs +cbcb7e73ec47b2d224d9072f5023b8ccec082e2b110ba090e84fb6fe05e92492 src/admin/router.rs a40adfbaa908da3ea726d7aa3d45394a70a6a55b6fc0063270ec4d2a2f2026a3 src/admin/session.rs e7355ef15e4377cd71e47d054c607079dd8e6711f1d86824f716b8003210a046 src/admin/status.rs e79d3dc0c4fde1ca4a69c204fb41cc5aa1db02aefdabff3435f27b5db0c6eb8d src/admin/status_live.rs @@ -208,19 +210,19 @@ c97881b17b7a42511868b0e48e2f25aab6fb5b613d538f5e1f1967508454e486 src/admin/stat cadd7dca29b9aad8c77c7bc23dd249ea37a15adfa4d636a8ef00273a91ce35dc src/admin/tail.rs 684307b74155ac7833acfb5e9e8eaccc92575d49b760928c30fc22e805a42a4c src/admin/tests.rs 18aa617a0bf60a3d8d2ad7dd059a369e7549ef46683d8f912e07f8fb091c26d1 src/admin/tests/part_01.rs -6f8c1e30babb2cc4cb7accf4bfa28338651a469b16341f6e7eaec881dd9abf2b src/admin/tests/part_02.rs +20abde2c1fa72d0712c2962479d66ae0131fb67ed12850c541306fe5ed1d2b84 src/admin/tests/part_02.rs 34cf08cafb63cf7a8a6336e5c81fc99abfce8669662e387a3702c3a532be8056 src/admin/tests/part_03.rs 2137a53991c9337988545c4472b4ffcb9a111cac9c460ebbec7e51c0be8f5630 src/admin/tests/part_04.rs -5f0062abc42a3b81c79088dc0824ab4c81a22c6ea765e77d2527d07d13bffd58 src/admin/tests/part_05.rs -311c4ad16b3ad5b295a339241afd4ca8ea0d3c7f54f609c27518b4de1b4a4b71 src/admin/tests/part_06.rs -5fa006413fd93529ac9aaa4ffb28bf6b0b88d0a55a0bf0c49576db4a06fe9fc3 src/admin/tests/part_07.rs +5bf2a7c2a9415b04eaba915a920efb5c60a1d6b6837a30481dd993c6ceb074da src/admin/tests/part_05.rs +0274038c2b18f1c9ab0d990c0d636cf6d4255454018463f49984898086f74bdb src/admin/tests/part_06.rs +402a219dbd70d6654bd1afc3d2ede2f5d3add96ba3f9867f7f51fba870efa404 src/admin/tests/part_07.rs 54ed2130e85239ffa4576417dd68fbb9349e396210b75f5da862897137d7e876 src/admin/tests/part_08.rs 225a29e2e560e42e06e19c3d41c0196c4a42e1e84caa7c83446cab62e3ac30dc src/admin/tests/part_09.rs bffa6611f3776e97d0065b510b63f083aabf7b3429fcc5cb26b1883fdb7c47dd src/admin/tests/part_10.rs 578f93cc5771b5858813945e5608b15c572bd3d7fd15bae291d292e2994ece25 src/admin/tests/part_11.rs 48fcd818c522fe80f2c7b2e218f87de6a7772ffd0a24e93502780b03ec78e335 src/admin/web/acme.js 446011c2c7afddf7734ceb268c75a3b4146dbe8e915b11986495e956b6c62c69 src/admin/web/app/config-history.js -9527dcafb9c14e420cb841a6984d373ba2136d07c45350e9368dd7a88e89af22 src/admin/web/app/core.js +7374ba5ee2c9e09677369542abec3a5b3d450df13aec705b0d8796a083e9d974 src/admin/web/app/core.js 23721bfbac850c4f77b116cc32bb9d2db6361ac4dd7e310a8c8070ab906997f6 src/admin/web/app/preferences.js 729ed152eb8471c67fb0e55e44fe0669c2708802ce83228df9cb4e53c14cee02 src/admin/web/app/refresh.js 984d092961bfaaa19f982e54c7d4be474e8417b208d814ac88bda63f1ab19f4c src/admin/web/app/runtime.js @@ -230,7 +232,7 @@ db56370db60e514a2241d231cd949bac2c7aa2f413a2c14035a877029a8d864f src/admin/web/ 50d1ebcbaf4d0b2968da23bb943fc3f67131c5d32df3f61d23ff37aad9da5b81 src/admin/web/demo.html 40cbf3a19827adbff179009b9189263a58a469319db25cf4adf0e4dc441c6532 src/admin/web/docker.js d4aef376cb44b2a13174dca1cb9383f64c04b0b93279f4f0b4805e76002548a1 src/admin/web/docs.css -60a4944a0a24b889296991b8af44b467f776fc85ec3597eb2e2940ef1f6fc713 src/admin/web/docs.html +aeffd403fa1823df7d9fa4c48280b4033b2d15e4cb9f66608caff89290de0a52 src/admin/web/docs.html 6adb291fcc98d516fd9745d9673db16e408854a32cbf41335ae956bda11d46de src/admin/web/docs.js c52c430b9a6e584012e6014709e1e16ae7e775b2a5aedff8eb767fa867c1cbdd src/admin/web/drafts.js 232b766f278ae4424705af351910677fcd7ac9a20bb8320d9a12b763c91219a2 src/admin/web/error_401.html @@ -240,22 +242,22 @@ f7d0955ab18c15e04369204e789f341d2e832f927c1ff20a3b2b9e35e7898f00 src/admin/web/ 386b9dc3aecebe7727f7cf7bbd2a02ab036a3abf01315975111a51559128df73 src/admin/web/error_503.html e6c4d91d1542d0a5b47bb8736f9539d43a6713df5a473b635519d5b7eac2eb9c src/admin/web/error_504.html 0a5fff35555153f59d4346933030a67b6e1a8768cee7a47cfa2fcf1a2c564f63 src/admin/web/favicon.svg -af6e7a2e890df2fb7385a374bcb3fba52fb5a53dbe18adc1fb7865660fcba842 src/admin/web/file-logs.js +4645af5dda304f839cd3af79bd5c52001f3092a5545aeafd7b37c2499fed24b7 src/admin/web/file-logs.js b14e7b3578322f309627f6fe965cc16abcf85da3437c9070be6caecae1908743 src/admin/web/host404.js -7994b44abf769243ff3a21d248d3eb79f26736c7730bd94978c29be08b3f07ef src/admin/web/index.html +c3178df6f9bb4bbb187e6d350ceda3dfa9db425b566aa27af75aa7945b1433a4 src/admin/web/index.html 24db88842707f764064657712d3660c6ac129fbfb47d188c0903925cf8244ce5 src/admin/web/integrations.js 7a12f020fc587205694978ab41b660e9a80487d413fbf6e34fc525eabc58308f src/admin/web/login-keys.json -af0fbd94d9b88ce1cce28e39dfd2c35de031adcba412dd4a9a17d9b416c256dd src/admin/web/login.html +cbd41678eadd994dc5252628ee8ef9551fa9f7d8694ca5abd206240d9a96ad30 src/admin/web/login.html 99fe9b8ba7a8a24e5bbe9ae40999484fe13b0faf5eb1d7a9193701f667e51ca5 src/admin/web/login.js 21ddae9743ca545c89f6df0db40810dd680812e58f2b2b6875ed64fd8fb97ac6 src/admin/web/management.js 90d9a0836662135b79fb6f183c4f3eff60f8a64392d6c5795ccf1d8ef6862263 src/admin/web/redirects.js 8fcad0d1a161ac6f45d773d1dc3925db2d32b34bdbe57595924f84eacb4f9136 src/admin/web/settings.js bb67e77af5514478485cfb69f339d63d7eb0cbadd093fa054814a416260ccc1d src/admin/web/streams.js -c4f051f54293156803b48c04d972e9d5a290209826138ce15356d19adf569979 src/admin/web/style.css -d01b615635c7373981b0633cc27298a34e67fe0cbe9d1384d277d46f0cafb1b4 src/admin/web/traffic-panel.js +98bb49a71b43ab0f843acca1d841ef1f53c63dfd1adb53d137c2564eaea58a65 src/admin/web/style.css +d290c048450655b3eeae7c6ec54a82d645d1965513694ab29fdf7eef796a1abd src/admin/web/traffic-panel.js a6e7bbff4fd2d50ecf759ed28161a672d7b483af2b35c9f912746a7a06a9f476 src/admin/web/transform-panel.js 57a2b975716bdab32c33ec1bf9427b25998550e81ce191440ee66ffb7e2a249e src/admin/web/vhosts/core.js -7e0dc397ce6b2f90dfadf0f724688cb952b89183d06042c0e28d716301e5e7ea src/admin/web/vhosts/editor.js +8f373c00cb545a03899ed30943ed82459bf6c5974b40ec89e7c44dfdc96bba43 src/admin/web/vhosts/editor.js 30a6344e76de77b60cd7247a33c68f1cb5d972aebfde56469f2ebd44440c7bf9 src/admin/web/vhosts/health.js f2689283cfe43d6aae911b0df778a69335089be230fbfc6de5dcce1801c38427 src/admin/web/vhosts/render.js 91e3417fba7e8f59023cd64813befccc8341741cd4aa53dfa1ec68a03c7f4632 src/admin/web/web-server-profiles.js @@ -332,10 +334,10 @@ a8a0ce446eb53ffdb8f8d97efa3e1344410dc3798b25f62b8817e8e160d28940 src/cache/tick e2f58279499b7c3b47c65d3a7814bb5c7cd6506544d939875fbb0f91fa9aa59d src/compression/tests.rs 7bc8285cb2012506fd1a38a71dffa561f885f4ab65a9b2a7c8344319e3cd1cf6 src/config.rs 7ba52d3de4d77d1f3cad1b10de418953adc33901727a0791980847ae223929aa src/config/backend.rs -514db7c259e40616b68ffb209ec1272b85bdd897f2766e601abc54dc4d5f4c31 src/config/compile.rs +a3d6a9d01dd9b3b602ea2f3dd5559503ea82129544c8944c26ad39445aaa3ad3 src/config/compile.rs 490d05d84b2fe349ddde54eae44880993fab70811652484dd9ba38b5069d603b src/config/internal_routing.rs -f02def401d5c0a6d619f0e6f88bbccd1db94a32bb004dc88f1581b773c7d6fc9 src/config/model.rs -335639a82f3449be1c41963b16cf5401f9bae0344b829f86170f911266c62864 src/config/routing.rs +d5b851aa5823af638fd2b2ba8ef4eeba12c55296c0a01f8ca7dad0674d124acc src/config/model.rs +f66c2388f31b19142c12469df23d43451e0f7c48b158d4996920282c06d32076 src/config/routing.rs 818545ad36af3ae8d56976f318349dd1f5264ef5665fbf94681a81b79003f8cb src/config/snapshot.rs b9f56cc156deaf1a59daf4da96233bba12a31a4fde3f6fe6cc08909fb6c1823e src/config/tests.rs f712db55af9ac463bc905558d29d068a58ac75269075055724ad187d7ea06fa9 src/config/tests/part_01.rs @@ -366,7 +368,7 @@ fbb6859780309f8cca1aeaba43fa67d13a1edf6f5a60c5c982d2067b71fb0846 src/headers/te d0cd8589321595309834690fa9ddb98488886bc98f34ea45aa0ccb3035412fb5 src/http3.rs 2165a0a4b27d621985e2030326ad52b8bbd59d7b9e4f092f72f7e92b4e58b497 src/http3/body.rs 68bc11160bc5275a904e48af8698b37433096fb6d592364d8281d59bb9e2b3f4 src/http3/tests.rs -3dab8fac75263120e48ee869ad55a3e2ccc2a19fc9f6d6a6fe055657ec55b1e3 src/lib.rs +68fb5ab6a889995a66ad12cb6acf88c5d04f0aca572e2ae15f1a9cd171a5f24b src/lib.rs 2551189a3e5701371026c9abd2c5d44d37495f28fb67e220180421bfd315e3dd src/logging.rs 3849a09a7d7402dc4d867b646cda153c574327ddd3c707578ddb4dc6f95b1563 src/logging/error_runtime.rs a16613114273c54b6d2a16a94fa8748550e91468585469642622bd2c86087baa src/logging/format.rs @@ -408,7 +410,7 @@ cea9e47ecb31b0e4c0bbe0eb83e0e53b5e7bdcb9930107b67a8c34eef01a3f49 src/proxy/forw 902cbf8dc3c1ec8d25411bc7fe0400d301f3a7cc885afada594e4eaee3bedd08 src/proxy/forward/success_prepare.rs 23b6bd26f6175b52c13d2571318fddd12550ba34ee77b87b5ac0e4811f8e2bee src/proxy/forward/upstream.rs 8961d8d649d8f4708f443f1b8b8a73c8b5f42811d2cad8cc6f596833511933f4 src/proxy/forward/web_source.rs -deadb4dd9120e37cb443724f29a891032090761fda4780258a77928d2123726c src/proxy/forward_entry.rs +69cb26cc475cc6fdee235df78227543b7c5e591b7cdf6cb7a2ff1024beffc845 src/proxy/forward_entry.rs c438066616527a3108f2ad53e3698ae9b2a80961714394cdd0a450c92f99d992 src/proxy/headers.rs 5cb7522ceb9a9f083a15596ac4898559295e38d1138c80099577211de8a868ec src/proxy/headers/tests.rs 42155c7742da4173f71dc4ef488288e1fd665fbb295f02745421785571926295 src/proxy/mod.rs @@ -486,12 +488,14 @@ a67e58a597a5eeb7efe605ae6e438a8f84ff6a1cedf8cd228141cb95551872a8 src/upstream.r 225e3078e6775ffaa6dc6f12bc87503a123983cbb347edf38cae4c37ff02a5db src/upstream/client.rs 4aa2420a92d33c50f7027f57ac4eccdc050bdb7e9ec8fa6e0eaa809228b71cd8 src/upstream/client_tests.rs 6f932b8e59de8a5c89d3b29b838eb2a30f6e3ddd342fec29e39af1ce1b1c7d11 src/upstream/tests.rs +3b3e5dd27253cb5626a0e37f670692adeb132aa2e0e89be35abdd730272b86a0 src/user_agent.rs +6d7cb73de7d8c6f3780736480f5f81ef08813f4626fd5ed494aca0552cf214a1 src/user_agent/tests.rs 3aa1b07cd26d4a5b9e5e94d1d6a712ef1397d827e7f1e8c01f7195b98d2c5f67 src/vhost.rs 6678192de56bc10c5cf82bdff8da14eece7685f409592450c07478333aa3b99c src/vhost/error_fallback.rs -bbf2f687b435e7c5d426bb2ad61b73ae26fc3b59be377c73277153ea97a4a7ba src/vhost/model.rs +63e9fc7f7be78dc3e16a89f015a56a2da6ba3aff0197b70ff48321dfb17529bb src/vhost/model.rs 35a0aca3eee8d47d1bab27c83d4532cd572fc3e17b5f0d4a8d4515a8f85b3d96 src/vhost/redirect.rs 9b16b5307e140db97363f56c9fac8319e56df846bb1c919d3bc06091ab6ce69c src/vhost/tests.rs -0e7b7ff1032a3a569acae884e9e2061ba936b22b7ac20f111a8633ca4597c25f src/vhost/validation.rs +b7d0f7f5d0bbd74add3fda9b1308350c4f70f8b6debf979fb977a4f2f659c32f src/vhost/validation.rs 8d1eff4b24f01d137414ef5419f75dd281879228adb70409645b11b3c78c20e9 src/web_server/config.rs 1fae9d57dfee80edcc3073438353254db5b6afdae1d9c1d6173cc6de5f66b2e6 src/web_server/config/compile.rs 7853f6ec127a91163a4418ca45b93a92f424d5c618f27b29f9d526598a26f7ef src/web_server/config/model.rs @@ -531,34 +535,36 @@ e5c58fc9b7685ec911bfa86f6ca4f0cc97c19a90c0cff5b60ea9887f1afe9ced tools/checks/a fd082812411cff7afb71b2289a6f1a469b8d3c8a7cfdcdd9e567a2a4baf81be4 tools/checks/admin_surface_audit_0310.py 412255d383fcc8306c6086784fb416d9814e3ec790e440a0927431ca615f3411 tools/checks/backend_cache_0297_ui_test.mjs 2a493dbce2991a23d807c01952f58608a364f541560836b4cb385e75e108e9e7 tools/checks/backend_health_0300_ui_test.mjs -54c73f3c0a63d0345bbe14a538503a9c63b6a9ff1925e7e51acbce38abd370e9 tools/checks/check_source.py -0b583ee59dcccc35b15ef4e211d7e6546a8aaa7b0286ceab03c0b0dc084c1e07 tools/checks/dashboard_boot_0311_ui_test.mjs +95b2024d6354aae1c40203c6cb3f557062baa184e15a0a01af19495b3899cf6c tools/checks/check_source.py +dc4df51fc15d4b6345c6625890ddbe014103851ebfa16abc583fbb0d809f4015 tools/checks/dashboard_boot_0311_ui_test.mjs 259d4b0985c82b631e151a07e8ef3b4c760019cb99bb2af458ce577497f7df58 tools/checks/dashboard_startup_0310_ui_test.mjs -76884f3c44f18c36eff3a1968aa173d2d782fa8d31ed7cc63e4ff41853488672 tools/checks/dependency_upgrade_0312.py -9add103198795111ccf1b3197cdd2586f54f606d152ff92be2ae09beb628d5aa tools/checks/docs_check.py +3b66874913b429f3f45a52642495829849d25e7f40c5304555e9971f32b25fae tools/checks/dependency_upgrade_0312.py +da49215969f31c0c2194d84dd178e101be5937b56c2f2a030a19df271aca436c tools/checks/docs_check.py +6ca66c0998d1da852ad66976c91e4d7d3628195aa49ddb44cea95f531eb50033 tools/checks/file_logs_tabs_0315_ui_test.mjs e33a9cc51d58bc92a637c0e6a1c88cbd24ba4605be55748212e296c3c35a4060 tools/checks/high_traffic_memory_model.py 4f7821e7e25b09ce251d25a0b3fb4d8ce8396c36a5ed971c2637aeb325425909 tools/checks/langs_check.py f8f87ba891338b565249d02c3cf5f02cec44e606b4205de99fcbea1eb2c24d57 tools/checks/memory_contracts.py af98579ee942e774d88775c261ac351847c28bb4a5396dc67e458bd6b1ff7963 tools/checks/origin_resume_0313_ui_test.mjs -b432ea24933ba28ae819d900892f7ada268560e2816234820394084d6296b595 tools/checks/panel_cache_http_0310_ui_test.mjs -300c114c05f66be214bc2e06187fc4d0063e3dc3cd55f44c43c2e43cbd3106f2 tools/checks/preflight.py +1ac812929c2a8c0699ae3bdebd0d7e05b5bcaf86c87533bb669b5ba7955e3245 tools/checks/panel_cache_http_0310_ui_test.mjs +4cdee49186bdfbde5d10f0a0bfe9577f4c7dc1fd4b735d55c55703dee01c87da tools/checks/preflight.py 021a97bd6763b59f3f7eca15f51efdf88d0c863f1f81a2e42ba80a9a09ca3fec tools/checks/refresh_0310_ui_test.mjs -e24a81c06deceab7f4d3c3bf596712d3eeb70b884c731820637461fbc93aa975 tools/checks/release_smoke.py +75aee85614ad8c8e7e3c4d843197c20ddfa975bf96208d39356532d66f8ad966 tools/checks/release_smoke.py f69845bc26ec33160fc756953991cf32aa8ad6a7fb4a131caa2093db91e783ef tools/checks/rust_module_contracts.py ecfe0bb21a7549225a19b0846bb6de8c99cbfbbf5a706d33cf84db2c7146abed tools/checks/rust_source.py adf84a983e0dbba228e47d71ec2d5c9964f7ca9989d5139de58f96ee961b035a tools/checks/security_contracts.py 23b9250245e6bec5ef728c73cee4c88f64924afc3597f8a9b58a1924235df98e tools/checks/security_dependency_contracts.py -0d211440ae77f16187a5d833156911c7793ed27ac1d77e79de362e47c1ab1abd tools/checks/source_module_contracts.py -b87c6c5b5d5335e95b3afc00f7c9035ddccc5ca1394a4425abcf898092da053a tools/checks/system_info_0301_ui_test.mjs +f4d9a92f25c40b879247750c1ef2764419368afb79bfe74414c0259cbb1330eb tools/checks/security_logs_0315_ui_test.mjs +f7fe205ee0cfe0347f73c10d626aa1dadda6a708bf8a46c25fc90075eeec968a tools/checks/source_module_contracts.py +730c47345deff507044f6989806e9fab9945edaee7dcb75555e2d51a33d6d34a tools/checks/system_info_0301_ui_test.mjs 543a488951ea6ec8fe06e04f168ccc4d4f2af3cfc281c9a728ab7bd87377fa98 tools/checks/system_info_no_subprocess_contracts.py -614ed0c775e32ec5bfb4ee75aaa90206a10da0f43659747a7ec76fd93e33767a tools/checks/version_contracts.py +a5105ab53de67e6b1114974c92b019014e1aa93acc445bc5dd6c82af0b0d3a16 tools/checks/version_contracts.py 161637cb5e2e613e3c71cf8489cc936fa84913a2a02649463f8893a28a674242 tools/checks/vhost_origin_errors_0297_ui_test.mjs 0c071c26fc316934bdfd2bfd19f5fed9521790d8c887760eaabda81a9be96703 tools/checks/vhost_www_ui_test.mjs 08f75c17084352df8d4dd0d1e712486d6d600739451735976de2234903538bc5 tools/checks/web_assets.py -0398892be30fe416a00578915c588f360fd93244e3af77ece834e560e547d1c8 tools/checks/web_server_contracts.py +35464d3d2c4af0c972b99e3cf4c7b8423b68260a8c7bc6a929f8e7ab2a164a96 tools/checks/web_server_contracts.py 35f76060e585a96c623ef83bd5f771969f5be16a0e1ae626a3092d7458b84e84 tools/checks/web_service_ui_test.mjs adaa1ed9e5958739df40190055c780c80d0a8e270ec8dc105ed3fa2dbb8fb74f tools/checks/www_redirect_ui_test.mjs -ddee9083a349b2a56b9ebe5742b5c91b668f31143762bac7d939068fbd2ee7a6 tools/dev/memory_load.py +7c30c280d0b0a8bf855eeffab583ca34dc161255f7ef6124148aa979c17c26d3 tools/dev/memory_load.py d1418f8e574a7390fd45a2aa879bdf5720b773162d65e295ab0350506aa5220a tools/dev/mock_auth.py 2d3ff92300b09ce9cecbbaaa9309b4d33160fa4b2291c9952b784e8cfb599ab5 tools/dev/mock_server.py 538b5f6cfa738355a729940ee3d92e7ab3dc25ca69ae330ddaa08f831775b8ba tools/dev/test_compression.py diff --git a/dev.sh b/dev.sh old mode 100755 new mode 100644 index c3507ba..670d950 --- a/dev.sh +++ b/dev.sh @@ -114,6 +114,7 @@ if [[ "$mode" == "test" ]]; then node tools/checks/admin_public_origin_0314_sim.mjs node tools/checks/admin_public_origin_0314_config_sim.mjs node tools/checks/admin_panel_config_0314_ui_test.mjs + node tools/checks/security_logs_0315_ui_test.mjs node tools/checks/panel_cache_http_0310_ui_test.mjs node tools/checks/vhost_origin_errors_0297_ui_test.mjs node tools/checks/www_redirect_ui_test.mjs diff --git a/docker-agent/install.sh b/docker-agent/install.sh old mode 100755 new mode 100644 diff --git a/docs/FILE_LOGS.md b/docs/FILE_LOGS.md index e41711c..64a9f12 100644 --- a/docs/FILE_LOGS.md +++ b/docs/FILE_LOGS.md @@ -1,6 +1,6 @@ # Stored log explorer -RustProxy 0.29.1 provides an authenticated, read-only **Logs** workspace at `/logs`. The top bar has one **Logs** menu with separate **Stored logs** and **Live Logs** actions; the two data paths remain independent. +RustProxy provides an authenticated, read-only **Logs** workspace at `/logs`. The top bar has one **Logs** menu with separate **Stored logs** and **Live Logs** actions; the two data paths remain independent. ## Separation from Live Logs @@ -45,6 +45,12 @@ At least one column must remain visible. Stored-log column selection is saved pe Selecting a row highlights it and expands parsed field/value pairs directly below the row. A nested **Raw JSON** section shows the normalized complete record. +## Repeated-value analysis + +The **Repeated values** tab (advanced parser) can count repeated values for any parsed field returned by the explorer. Typical security-investigation fields are `user_agent`, `client_ip`, `host` and `path`. Counts are calculated across the bounded matching scan before pagination, so changing the page does not change the count set. Minimum repetitions and top-result count are configurable within server limits. Values in the result table are directly copyable. + +Aggregation does not remove the Stored Logs safety bounds: at most 4,096 distinct aggregate values are retained and at most 200 results are returned. The response marks an aggregate as truncated when a cardinality or result cap is reached. + ## RAW error logs RustProxy's built-in RAW error-log format is also normalized for browsing. The explorer extracts timestamp, level, target, message and recognizable `key=value` fields, while retaining the original line in `raw`. @@ -65,6 +71,8 @@ The browser still never supplies a filesystem path. The server derives sources f Logrotate-style gzip rotations are discovered next to configured combined access/error files and inside the configured per-vhost directory. Only regular non-symlink files matching the active filename rotation pattern (for example `access.jsonl.1.gz`) are eligible. The browser still refers to them by server-generated source IDs rather than paths. +For per-vhost logging, the active `vhost.jsonl` and matching `vhost.jsonl.*.gz` files share one logical group ID. The panel enables **Consolidate rotated logs per vhost** by default, so selecting that vhost searches the active and compressed members together as one logical source. Disable the switch to inspect individual files. Physical source IDs and metadata remain separate internally, and the existing global source/byte/line scan limits still apply. + Gzip reading is bounded before and after decompression. A compressed source larger than 8 MiB is not decompressed for a query, decompressed output is capped at 64 MiB, and only the bounded tail is parsed/searched. These limits prevent a gzip rotation from becoming an unbounded memory/decompression workload. Per-vhost directory discovery now retains only the newest **256** matching JSONL/gzip candidates while scanning, and a combined log’s gzip-rotation scan retains only the newest **64** matches. Both caps apply before sorting, so directory size no longer causes a proportional memory allocation for the inventory. The usual 32-source query budget still applies. Filesystem enumeration still takes time proportional to the number of directory entries. diff --git a/docs/IMPLEMENTATION_0.31.5.md b/docs/IMPLEMENTATION_0.31.5.md new file mode 100644 index 0000000..a2de82c --- /dev/null +++ b/docs/IMPLEMENTATION_0.31.5.md @@ -0,0 +1,55 @@ +# RustProxy 0.31.5 implementation and verification + +## Scope + +0.31.5 adds an operator workflow from traffic/log inspection to a bounded User-Agent blocklist without changing SQLite schema 9 or existing saved configuration requirements. + +## User-Agent blocking + +`user_agent_blocking` is available globally and on each vhost. Both levels use the same bounded policy: + +- disabled by default; +- at most 128 patterns, 256 bytes per pattern and 16 KiB total pattern text; +- case-insensitive ASCII matching; +- a pattern without `*` matches a substring; +- `*` is a wildcard and may be used at the start, middle or end; +- a request with no `User-Agent` header is not denied by this rule; +- global and per-vhost policies are cumulative. + +The check runs after vhost lookup but before redirects, cache/origin processing and before an upstream is opened. A match returns HTTP 403. The global policy is edited under **Settings -> Security**; the vhost policy is edited under **Vhosts -> Security**. + +This is an operator blocklist, not a browser identity/authentication mechanism. User-Agent values are client-controlled and can be changed trivially. + +## Traffic copy workflow + +Traffic dimension labels are rendered as copy controls. The copied value is the raw displayed dimension value rather than a formatted count, so IP addresses, User-Agent values, paths, hosts, methods, status values and other traffic dimensions can be pasted directly into investigation notes or configuration fields. + +## Stored Logs repeated-value aggregation + +`GET /api/file-logs/query` accepts optional `aggregate_field`, `aggregate_min_count` and `aggregate_limit` parameters. Aggregation is computed across matching records in the same bounded scan used by Stored Logs, before page offset/limit are applied. + +The field may be a canonical field such as `user_agent`, `client_ip`, `host` or `path`, or another parsed/nested field accepted by the Stored Logs field resolver. Scalar strings, booleans and numbers are counted; bounded array scalar values are counted individually; objects are not flattened into invented text values. + +Safety limits are retained: the scan remains limited to 8 MiB/20,000 lines/32 sources, aggregate output is at most 200 rows, and aggregate cardinality is capped at 4,096 distinct values. The response reports when aggregation was truncated by those limits. + +## Consolidated per-vhost Stored Logs + +Each per-vhost active JSONL file and matching `.jsonl.*.gz` rotation receives the same logical `group_id`, for example `vhost:example.com`. Individual physical files still have unique source IDs and metadata. + +When `consolidate_vhosts=true` and `source=vhost:`, the query scans all discovered members in that group subject to the existing global source/byte/line bounds. The panel enables this behavior by default and presents one logical vhost option with the member count and summed discovered size. Turning consolidation off exposes the physical files separately again. + +Combined access/error logs keep their existing source behavior; this switch only changes per-vhost grouping. + +## Compatibility and security + +- Existing configuration files deserialize with `user_agent_blocking` disabled through serde defaults. +- No SQLite migration or cache-format migration is introduced. +- Stored Logs still accept server-generated source IDs only, never arbitrary filesystem paths. +- Existing symlink rejection, gzip input/output caps and blocking-worker isolation remain in force. +- User-Agent denial occurs before origin contact, so a blocked request cannot cause a new upstream connection. + +## Verification status + +Rust-independent checks cover language packs, JavaScript syntax/UI source contracts, OpenAPI/source-version consistency, Stored Logs grouping/aggregation source contracts and existing migration/release smoke tests. The source also contains Rust unit/regression tests for User-Agent matching and Stored Logs aggregation/grouping. + +The validation environment used for this source package does not contain Rust/Cargo. Therefore `cargo fmt`, `cargo clippy`, Rust compilation and Rust unit/integration execution remain a release gate. Run `./dev.sh check` and `./dev.sh test` on the supported Rust toolchain before production deployment, then verify real HTTP 403 behavior, vhost/global precedence, clipboard actions and gzip consolidation against representative production logs. diff --git a/docs/README.md b/docs/README.md index ae69864..aefc028 100644 --- a/docs/README.md +++ b/docs/README.md @@ -31,6 +31,7 @@ This is the canonical index for the current project documentation. The authentic - [0.26.3 implementation note](V0.26.3_IMPLEMENTATION.md) - global interface scaling, chart themes, square draft dock and Settings navigation polish. - [0.26.2 implementation note](V0.26.2_IMPLEMENTATION.md) - unified Logs navigation, field-query search, dynamic JSON columns and RAW parsing. - [0.26.1 implementation note](V0.26.1_IMPLEMENTATION.md) - historical vhost-aware protocol logging and Stored Logs UX milestone. +- [0.31.5 User-Agent blocking and consolidated log analysis](IMPLEMENTATION_0.31.5.md) - global/per-vhost UA blocklists, Traffic copy actions, repeated-value aggregation and per-vhost gzip consolidation. - [0.31.4 administration public Origin and settings](IMPLEMENTATION_0.31.4.md) - opt-in HTTPS origin, one-field DNS allowlist, additional hosts in advanced settings, legacy deduplication, CSRF and remaining deployment checks. - [0.31.3 mobile session / invalid-Origin recovery](IMPLEMENTATION_0.31.3.md) - deterministic BFCache and visibility session coordination, 403 diagnostics, recovery and remaining acceptance. - [0.31.1 mobile panel bootstrap](IMPLEMENTATION_0.31.1.md) - panel script readiness, versioned assets, failure recovery and acceptance tests. @@ -83,7 +84,7 @@ This is the canonical index for the current project documentation. The authentic ## Historical or version-labelled context -Version-labelled `V*.md` files are retained as historical implementation/audit records. Their version number describes the audited milestone. The current release implementation note is [IMPLEMENTATION_0.31.4.md](IMPLEMENTATION_0.31.4.md); [IMPLEMENTATION_0.31.3.md](IMPLEMENTATION_0.31.3.md) is retained as the previous mobile recovery milestone; [IMPLEMENTATION_0.31.1.md](IMPLEMENTATION_0.31.1.md) is retained as the prior mobile milestone; [IMPLEMENTATION_0.31.0.md](IMPLEMENTATION_0.31.0.md) is retained as the previous milestone; [V0.29.1_IMPLEMENTATION.md](V0.29.1_IMPLEMENTATION.md) is retained as historical context, and [V0.29.0_IMPLEMENTATION.md](V0.29.0_IMPLEMENTATION.md) is retained as the prior redirect/health/cache/log milestone; [V0.28.2_IMPLEMENTATION.md](V0.28.2_IMPLEMENTATION.md) is retained as the prior certificate-hardening milestone; [V0.28.1_IMPLEMENTATION.md](V0.28.1_IMPLEMENTATION.md) is retained as the prior external-path milestone; [V0.28.0_IMPLEMENTATION.md](V0.28.0_IMPLEMENTATION.md) is retained as the prior GeoIP/listener/cache milestone, and [V0.27.3_IMPLEMENTATION.md](V0.27.3_IMPLEMENTATION.md) is retained as the prior HTTP/3/cache-maintenance milestone, [V0.27.2_IMPLEMENTATION.md](V0.27.2_IMPLEMENTATION.md) is retained as the prior managed-ACME repair, [V0.27.1_IMPLEMENTATION.md](V0.27.1_IMPLEMENTATION.md) as the prior managed-runtime/list-sorting milestone, and older version notes remain historical evidence. The latest retained audit is [V0.25.5_AUDIT.md](V0.25.5_AUDIT.md); older audit files remain available for traceability. +Version-labelled `V*.md` files are retained as historical implementation/audit records. Their version number describes the audited milestone. The current release implementation note is [IMPLEMENTATION_0.31.5.md](IMPLEMENTATION_0.31.5.md); [IMPLEMENTATION_0.31.4.md](IMPLEMENTATION_0.31.4.md) is retained as the previous administration-Origin milestone; [IMPLEMENTATION_0.31.3.md](IMPLEMENTATION_0.31.3.md) is retained as the previous mobile recovery milestone; [IMPLEMENTATION_0.31.1.md](IMPLEMENTATION_0.31.1.md) is retained as the prior mobile milestone; [IMPLEMENTATION_0.31.0.md](IMPLEMENTATION_0.31.0.md) is retained as the previous milestone; [V0.29.1_IMPLEMENTATION.md](V0.29.1_IMPLEMENTATION.md) is retained as historical context, and [V0.29.0_IMPLEMENTATION.md](V0.29.0_IMPLEMENTATION.md) is retained as the prior redirect/health/cache/log milestone; [V0.28.2_IMPLEMENTATION.md](V0.28.2_IMPLEMENTATION.md) is retained as the prior certificate-hardening milestone; [V0.28.1_IMPLEMENTATION.md](V0.28.1_IMPLEMENTATION.md) is retained as the prior external-path milestone; [V0.28.0_IMPLEMENTATION.md](V0.28.0_IMPLEMENTATION.md) is retained as the prior GeoIP/listener/cache milestone, and [V0.27.3_IMPLEMENTATION.md](V0.27.3_IMPLEMENTATION.md) is retained as the prior HTTP/3/cache-maintenance milestone, [V0.27.2_IMPLEMENTATION.md](V0.27.2_IMPLEMENTATION.md) is retained as the prior managed-ACME repair, [V0.27.1_IMPLEMENTATION.md](V0.27.1_IMPLEMENTATION.md) as the prior managed-runtime/list-sorting milestone, and older version notes remain historical evidence. The latest retained audit is [V0.25.5_AUDIT.md](V0.25.5_AUDIT.md); older audit files remain available for traceability. Older context documents such as [INTEGRATIONS_040](INTEGRATIONS_040.md), [MANAGEMENT](MANAGEMENT.md), [SECURITY_AUDIT](SECURITY_AUDIT.md), [ACME_RFC2136](ACME_RFC2136.md) and older version audits remain available for traceability but should not override the current functional/technical documents above. diff --git a/docs/SECURITY.md b/docs/SECURITY.md index 7365e79..56b148a 100644 --- a/docs/SECURITY.md +++ b/docs/SECURITY.md @@ -217,3 +217,9 @@ revision independently; the obsolete hide_mock_origin field is not accepted. X-M removed using ordinary response-header rules. Scope labels on listeners never enforce network restrictions. See ACME_RFC2136.md for private TSIG/account storage and the separate backup requirement for manual DNS-01 issuance. +## User-Agent blocklists + +RustProxy can apply an optional global User-Agent blocklist and an additional per-vhost blocklist. Matching is case-insensitive; plain text matches a substring and `*` is a wildcard. Policies are bounded and disabled by default. They are evaluated before opening an upstream; a match returns HTTP 403. Requests without a User-Agent are not denied by this rule. + +User-Agent is client-controlled and easy to spoof, so this is an operational noise/blocking control rather than an authentication or identity boundary. Keep ACLs, authentication and rate limits as the authoritative controls. + diff --git a/docs/TEST_REPORT.md b/docs/TEST_REPORT.md index 2b56bf7..199c1f1 100644 --- a/docs/TEST_REPORT.md +++ b/docs/TEST_REPORT.md @@ -1,4 +1,28 @@ -# Verification report - RustProxy 0.31.4 +# Verification report - RustProxy 0.31.5 + +## 0.31.5 follow-up - Stored Logs navigation polish + +- Split Stored Logs into **Log entries** and **Repeated values** tabs, sharing source/search filters; the count query is only included while the analysis tab is active. Move the persisted consolidation toggle into the header. Verify via dedicated Node UI source regression. Rust compilation/browser acceptance remain outstanding. + +## 0.31.5 follow-up - build-specific JS/CSS filenames + +- `scripts/build/swagger_assets.py` creates a fresh random 64-bit hex identifier using Python `secrets` when Cargo invokes the build script. The identifier is embedded in the binary and substituted into dashboard, public login and Docs HTML. HTML retains its existing cache-header behavior. Matching fingerprint paths serve the existing embedded asset bytes without runtime disk access; unrecognized fingerprints return 404. Legacy URLs remain for compatibility. Package remains 0.31.5; rebuilding with unchanged source changes these asset URLs whenever the build script reruns. +- Regression checks cover URL templates, version independence, build token generation, asset routing and anonymous allowlist. Source/UI checks run without Cargo; compilation and real browser/cache tests remain open. + +## 0.31.5 follow-up - Stored Logs consolidation preference + +- Stored Logs now restores the last selected consolidation mode from browser-local storage (`rp-file-logs-consolidate-vhosts`) on panel load; saves changes immediately; defaults to the existing checked mode if no preference is present or storage is blocked. Clearing browser storage restores the default. No API or SQLite change. +- Verification: `security_logs_0315_ui_test.mjs` checks preference read/write integration. Rust compile and runtime acceptance remain open as described below. + +## 0.31.5 - User-Agent blocking and consolidated log analysis (2026-10-11) + +- **Scope:** global and per-vhost User-Agent blocklists, copyable Traffic values, Stored Logs repeated-value aggregation and optional per-vhost consolidation of active JSONL plus gzip rotations. +- **Compatibility:** no SQLite migration; existing configurations load with User-Agent blocking disabled. Existing Stored Logs physical source IDs remain available when consolidation is disabled. +- **Security:** User-Agent matching is bounded and runs before upstream contact; Stored Logs continue to accept only server-derived sources with existing symlink/gzip/scan limits. User-Agent is client-controlled and is not treated as an identity boundary. +- **PASS — full Rust-independent release run:** `./dev.sh test` passed module/version/source contracts, SQLite schema-9 migration/upgrade smoke, security/dependency checks, the 300,000-PURGE memory model, 2,000 HTTP requests to the Python test peer, every bundled Node UI/simulation test (including `security_logs_0315_ui_test.mjs`), documentation and both language packs. It then exited **127** only at the missing Rust/Cargo guard; the Python peer is not a RustProxy binary. +- **PASS — focused 0.31.5 regression:** `security_logs_0315_ui_test.mjs` verifies global/per-vhost UA wiring and pre-origin ordering, Traffic copy controls, Stored Logs aggregation/consolidation markers, OpenAPI and English/Polish locale keys. +- **PASS — other available checks:** `go test ./...` in `docker-agent`; `bash -n dev.sh`; Node syntax checks for the modified UI files/test; OpenAPI JSON parsing; `docs_check.py`; `langs_check.py`; `check_source.py`; and `rust_module_contracts.py`. +- **Rust release gate OPEN:** `./dev.sh check` exits **127** because this environment does not provide `rustc`, `cargo` or `rustfmt`; new Rust unit/regression tests are present but cannot be compiled/executed here. ## 0.31.4 reissue - origin protection and one-field administration public URL (2026-10-11) diff --git a/install-debian13.sh b/install-debian13.sh old mode 100755 new mode 100644 diff --git a/install.sh b/install.sh old mode 100755 new mode 100644 diff --git a/integrations/opnsense-acme/opnsense-rustproxy-cert-deploy.sh b/integrations/opnsense-acme/opnsense-rustproxy-cert-deploy.sh old mode 100755 new mode 100644 diff --git a/integrations/routeros-oci-build/build-routeros-oci.sh b/integrations/routeros-oci-build/build-routeros-oci.sh old mode 100755 new mode 100644 diff --git a/integrations/routeros-oci-build/docker-entrypoint.sh b/integrations/routeros-oci-build/docker-entrypoint.sh old mode 100755 new mode 100644 diff --git a/langs/en_US.json b/langs/en_US.json index 463718c..1478a03 100644 --- a/langs/en_US.json +++ b/langs/en_US.json @@ -2336,6 +2336,31 @@ "administrationPublicOrigin": "Public panel HTTPS origin (optional)", "administrationHostsAdvanced": "Advanced: additional allowed DNS hostnames", "administrationHostsHelp": "Enter only additional hostnames. The public panel URL is included automatically. Without that URL, this remains an independent strict allowlist.", - "administrationPublicOriginHelp": "Behind a trusted reverse proxy, enter the external HTTPS panel URL once. Its hostname is automatically allowed. Restart required; public proxy routing is unchanged." + "administrationPublicOriginHelp": "Behind a trusted reverse proxy, enter the external HTTPS panel URL once. Its hostname is automatically allowed. Restart required; public proxy routing is unchanged.", + "userAgentBlocking": "User-Agent blocking", + "globalUaBlockingTitle": "Global User-Agent blocklist", + "globalUaBlockingHelp": "Blocks matching User-Agent values on public proxy vhosts before an upstream is opened. Per-vhost lists are configured in Vhosts → Security.", + "uaBlockPatterns": "Blocked patterns, one per line", + "uaBlockPatternHelp": "Matching is case-insensitive. Plain text matches a substring; * is a wildcard. Requests without User-Agent are not blocked by this rule.", + "vhUaBlockEnabled": "Block User-Agent patterns", + "vhUaBlockPatterns": "Blocked User-Agent patterns", + "vhUaBlockHelp": "Vhost blocklist. Plain text matches a case-insensitive substring; * is a wildcard. It is enforced in addition to the global blocklist.", + "fileLogsConsolidateVhosts": "Consolidate rotated logs per vhost", + "fileLogsAdvancedParser": "Advanced parser · repeated values", + "fileLogsAggregateField": "Field to count", + "fileLogsAggregateMin": "Minimum repetitions", + "fileLogsAggregateLimit": "Top values", + "fileLogsCountRepeats": "Count repeats", + "fileLogsAggregateHelp": "Counts matching values across the bounded scanned log set, not only the current page. Use fields such as user_agent, client_ip, host, path or any parsed nested field.", + "fileLogsAggregateValue": "Value", + "fileLogsAggregateCount": "Count", + "fileLogsAggregateEmpty": "No repeated values for the selected field.", + "fileLogsAggregateTruncated": "aggregation truncated by safety limit", + "fileLogsFiles": "files", + "copied": "Copied", + "copyValue": "Copy value", + "fileLogsConsolidateShort": "Merge rotations", + "fileLogsEntriesTab": "Log entries", + "fileLogsCountsTab": "Repeated values" } } diff --git a/langs/pl_pl.json b/langs/pl_pl.json index 513fd52..6e688a5 100644 --- a/langs/pl_pl.json +++ b/langs/pl_pl.json @@ -2336,6 +2336,31 @@ "administrationPublicOrigin": "Publiczny adres HTTPS panelu (opcjonalnie)", "administrationHostsAdvanced": "Zaawansowane: dodatkowe dozwolone hosty DNS", "administrationHostsHelp": "Wpisz wyłącznie dodatkowe domeny. Domena z publicznego adresu panelu jest dozwolona automatycznie. Bez tego adresu lista pozostaje niezależnym ścisłym ograniczeniem.", - "administrationPublicOriginHelp": "Za zaufanym reverse proxy wpisz tylko publiczny adres HTTPS panelu. Jego domena zostanie dozwolona automatycznie. Wymagany restart; ruch proxy bez zmian." + "administrationPublicOriginHelp": "Za zaufanym reverse proxy wpisz tylko publiczny adres HTTPS panelu. Jego domena zostanie dozwolona automatycznie. Wymagany restart; ruch proxy bez zmian.", + "userAgentBlocking": "Blokowanie User-Agent", + "globalUaBlockingTitle": "Globalna lista blokowanych User-Agent", + "globalUaBlockingHelp": "Blokuje pasujące wartości User-Agent na publicznych vhostach proxy przed otwarciem połączenia do upstreamu. Listy per vhost są w Vhosts → Security.", + "uaBlockPatterns": "Blokowane wzorce, po jednym w wierszu", + "uaBlockPatternHelp": "Dopasowanie nie rozróżnia wielkości liter. Zwykły tekst oznacza fragment, a * jest wildcardem. Brak nagłówka User-Agent nie powoduje blokady tą regułą.", + "vhUaBlockEnabled": "Blokuj wzorce User-Agent", + "vhUaBlockPatterns": "Blokowane wzorce User-Agent", + "vhUaBlockHelp": "Lista blokad dla vhosta. Zwykły tekst dopasowuje fragment bez rozróżniania wielkości liter, a * jest wildcardem. Działa dodatkowo do listy globalnej.", + "fileLogsConsolidateVhosts": "Konsoliduj rotowane logi per vhost", + "fileLogsAdvancedParser": "Zaawansowany parser · powtarzające się wartości", + "fileLogsAggregateField": "Pole do zliczania", + "fileLogsAggregateMin": "Minimalna liczba powtórzeń", + "fileLogsAggregateLimit": "Najczęstsze wartości", + "fileLogsCountRepeats": "Zlicz powtórzenia", + "fileLogsAggregateHelp": "Zlicza pasujące wartości w ograniczonym zbiorze skanowanych logów, a nie tylko na bieżącej stronie. Możesz użyć pól np. user_agent, client_ip, host, path lub dowolnego sparsowanego pola zagnieżdżonego.", + "fileLogsAggregateValue": "Wartość", + "fileLogsAggregateCount": "Liczba", + "fileLogsAggregateEmpty": "Brak powtarzających się wartości dla wybranego pola.", + "fileLogsAggregateTruncated": "agregacja ograniczona limitem bezpieczeństwa", + "fileLogsFiles": "plików", + "copied": "Skopiowano", + "copyValue": "Kopiuj wartość", + "fileLogsConsolidateShort": "Scal rotacje", + "fileLogsEntriesTab": "Wpisy logów", + "fileLogsCountsTab": "Powtarzające się wartości" } } diff --git a/make_zip.py b/make_zip.py old mode 100755 new mode 100644 diff --git a/nagios/check_rustproxy.py b/nagios/check_rustproxy.py old mode 100755 new mode 100644 diff --git a/nagios/check_rustproxy_api b/nagios/check_rustproxy_api old mode 100755 new mode 100644 diff --git a/nagios/check_rustproxy_backends b/nagios/check_rustproxy_backends old mode 100755 new mode 100644 diff --git a/nagios/check_rustproxy_latency b/nagios/check_rustproxy_latency old mode 100755 new mode 100644 diff --git a/nagios/check_rustproxy_queues b/nagios/check_rustproxy_queues old mode 100755 new mode 100644 diff --git a/scripts/build/swagger_assets.py b/scripts/build/swagger_assets.py old mode 100755 new mode 100644 index d8ad374..99f848d --- a/scripts/build/swagger_assets.py +++ b/scripts/build/swagger_assets.py @@ -217,6 +217,10 @@ def main() -> int: checksum = expected_digest(os.environ.get("SWAGGER_UI_ARCHIVE_SHA256")) offline = os.environ.get("CARGO_NET_OFFLINE", "").lower() in ("1", "true") paths = prepare(args.out_dir.resolve(), source, checksum, offline=offline) + # A fresh cryptographically random build identifier, shared by all JS/CSS assets. + # Regenerated only when Cargo actually runs the build script. + import secrets + atomic_write(args.out_dir.resolve() / "asset-build-id.txt", secrets.token_hex(8).encode("ascii")) for path in paths: # These values are Cargo directives, so never allow extra output lines. if any(c in str(path) for c in "\r\n\0"): diff --git a/scripts/deploy/deploy.py b/scripts/deploy/deploy.py old mode 100755 new mode 100644 diff --git a/scripts/deploy/deploy_core.py b/scripts/deploy/deploy_core.py old mode 100755 new mode 100644 diff --git a/scripts/deploy/deploy_runtime.py b/scripts/deploy/deploy_runtime.py old mode 100755 new mode 100644 diff --git a/scripts/deploy/upgrade.py b/scripts/deploy/upgrade.py old mode 100755 new mode 100644 diff --git a/scripts/migration/migrate_npm.py b/scripts/migration/migrate_npm.py old mode 100755 new mode 100644 diff --git a/scripts/migration/migrate_npm_api.py b/scripts/migration/migrate_npm_api.py old mode 100755 new mode 100644 diff --git a/scripts/migration/migrate_npm_core.py b/scripts/migration/migrate_npm_core.py old mode 100755 new mode 100644 diff --git a/scripts/resolve_dependencies_0312.sh b/scripts/resolve_dependencies_0312.sh old mode 100755 new mode 100644 index e3cc9ca..5ba9edc --- a/scripts/resolve_dependencies_0312.sh +++ b/scripts/resolve_dependencies_0312.sh @@ -8,8 +8,8 @@ if ! command -v cargo >/dev/null 2>&1 || ! command -v rustc >/dev/null 2>&1; the exit 127 fi python3 tools/checks/preflight.py -if [[ "$(python3 scripts/version.py)" != '0.31.4' ]]; then - printf 'Expected RustProxy 0.31.4 sources.\n' >&2 +if [[ "$(python3 scripts/version.py)" != '0.31.5' ]]; then + printf 'Expected RustProxy 0.31.5 sources.\n' >&2 exit 1 fi printf 'Toolchain: '; rustc --version diff --git a/scripts/runtime/acme_provider.py b/scripts/runtime/acme_provider.py old mode 100755 new mode 100644 diff --git a/scripts/runtime/acme_rfc2136.py b/scripts/runtime/acme_rfc2136.py old mode 100755 new mode 100644 diff --git a/scripts/version.py b/scripts/version.py old mode 100755 new mode 100644 diff --git a/src/admin/documentation.rs b/src/admin/documentation.rs index 46589b6..20fb542 100644 --- a/src/admin/documentation.rs +++ b/src/admin/documentation.rs @@ -14,8 +14,19 @@ use std::sync::OnceLock; mod documents; use documents::READER_DOCUMENTS; -pub(super) async fn index() -> Html<&'static str> { - Html(include_str!("web/docs.html")) +pub(super) async fn index() -> impl IntoResponse { + Html(include_str!("web/docs.html").replace("__RUSTPROXY_ASSET_ID__", super::asset_build_id())) +} + +pub(super) async fn fingerprinted(axum::extract::Path(name): axum::extract::Path) -> Response { + let full = format!("/docs/{name}"); + match super::fingerprinted_path(&full, "/docs/").as_deref() { + Some("/docs/init.js") => script().await.into_response(), + Some("/docs/style.css") => style().await.into_response(), + Some("/docs/swagger-ui-bundle.js") => asset("swagger-ui-bundle.js"), + Some("/docs/swagger-ui.css") => asset("swagger-ui.css"), + _ => StatusCode::NOT_FOUND.into_response(), + } } pub(super) async fn script() -> impl IntoResponse { diff --git a/src/admin/documentation/documents.rs b/src/admin/documentation/documents.rs index 127c7f0..e641ba3 100644 --- a/src/admin/documentation/documents.rs +++ b/src/admin/documentation/documents.rs @@ -244,6 +244,7 @@ pub(super) static READER_DOCUMENTS: &[(&str, &str, &str)] = &[ "guides", include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/docs/STREAMS.md")), ), + reader_doc!("docs/IMPLEMENTATION_0.31.5.md", "technical"), reader_doc!("docs/IMPLEMENTATION_0.31.4.md", "technical"), reader_doc!("docs/IMPLEMENTATION_0.31.3.md", "technical"), reader_doc!("docs/IMPLEMENTATION_0.31.1.md", "technical"), diff --git a/src/admin/file_log_search.rs b/src/admin/file_log_search.rs index c656fc9..8114ffa 100644 --- a/src/admin/file_log_search.rs +++ b/src/admin/file_log_search.rs @@ -351,6 +351,36 @@ fn parse_raw_error_record(line: &str) -> Value { Value::Object(object) } +fn file_log_aggregate_values(value: &Value, output: &mut Vec) { + if output.len() >= 16 { + return; + } + match value { + Value::Null => {} + Value::Bool(value) => output.push(value.to_string()), + Value::Number(value) => output.push(value.to_string()), + Value::String(value) => { + let rendered = if value.chars().count() > 512 { + value.chars().take(512).collect::() + } else { + value.clone() + }; + if !rendered.is_empty() { + output.push(rendered); + } + } + Value::Array(values) => { + for value in values.iter().take(16) { + file_log_aggregate_values(value, output); + if output.len() >= 16 { + break; + } + } + } + Value::Object(_) => {} + } +} + fn collect_file_log_fields(value: &Value, output: &mut std::collections::BTreeSet) { fn walk(prefix: &str, value: &Value, output: &mut std::collections::BTreeSet) { if output.len() >= 64 { diff --git a/src/admin/file_logs.rs b/src/admin/file_logs.rs index d9b7e46..aa8072c 100644 --- a/src/admin/file_logs.rs +++ b/src/admin/file_logs.rs @@ -4,12 +4,15 @@ const FILE_LOG_SCAN_LINES: usize = 20_000; const FILE_LOG_MAX_SOURCES: usize = 32; const FILE_LOG_MAX_LIMIT: usize = 500; const FILE_LOG_MAX_OFFSET: usize = 10_000; +const FILE_LOG_MAX_AGGREGATE_RESULTS: usize = 200; +const FILE_LOG_MAX_AGGREGATE_VALUES: usize = 4096; const FILE_LOG_GZIP_INPUT_MAX: u64 = 8 * 1024 * 1024; const FILE_LOG_GZIP_OUTPUT_MAX: u64 = 64 * 1024 * 1024; #[derive(Clone)] struct FileLogSource { id: String, label: String, + group_id: String, kind: &'static str, path: std::path::PathBuf, enabled: bool, @@ -28,6 +31,10 @@ struct FileLogQuery { status_class: Option, client_ip: Option, vhost: Option, + consolidate_vhosts: Option, + aggregate_field: Option, + aggregate_limit: Option, + aggregate_min_count: Option, offset: Option, limit: Option, } @@ -42,6 +49,7 @@ fn file_modified_ms(metadata: &std::fs::Metadata) -> u64 { fn source_from_path( id: String, label: String, + group_id: String, kind: &'static str, path: std::path::PathBuf, enabled: bool, @@ -58,6 +66,7 @@ fn source_from_path( FileLogSource { id, label, + group_id, kind, path, enabled, @@ -109,9 +118,12 @@ fn configured_file_log_sources(config: &crate::config::Config) -> Vec Vec Vec Value { json!({ "id": source.id, "label": source.label, + "group_id": source.group_id, "kind": source.kind, "path": source.path.to_string_lossy(), "enabled": source.enabled, @@ -281,191 +296,4 @@ fn normalized_record( value } -fn contains_ci(value: Option<&Value>, needle: &str) -> bool { - value - .and_then(Value::as_str) - .is_some_and(|text| text.to_ascii_lowercase().contains(needle)) -} - -fn record_matches(value: &Value, query: &FileLogQuery, search: &[FileLogSearchClause]) -> bool { - let Some(object) = value.as_object() else { - return false; - }; - if !record_matches_search(value, search) { - return false; - } - if let Some(event) = query - .event - .as_deref() - .filter(|v| !v.is_empty() && *v != "all") - { - if object.get("event").and_then(Value::as_str) != Some(event) { - return false; - } - } - if let Some(class) = query - .status_class - .as_deref() - .filter(|v| !v.is_empty() && *v != "all") - { - let Some(status) = object.get("status").and_then(Value::as_u64) else { - return false; - }; - let expected = class - .as_bytes() - .first() - .and_then(|v| v.checked_sub(b'0')) - .map(u64::from); - if expected != Some(status / 100) { - return false; - } - } - if let Some(client) = query.client_ip.as_deref().filter(|v| !v.is_empty()) { - if !contains_ci(object.get("client_ip"), &client.to_ascii_lowercase()) { - return false; - } - } - if let Some(vhost) = query.vhost.as_deref().filter(|v| !v.is_empty()) { - if !contains_ci(object.get("vhost"), &vhost.to_ascii_lowercase()) { - return false; - } - } - true -} - -fn validate_file_log_query(query: &FileLogQuery) -> Result<(), ApiError> { - let text_ok = |value: &Option, max: usize| { - value - .as_ref() - .is_none_or(|v| v.len() <= max && !v.chars().any(char::is_control)) - }; - if !text_ok(&query.source, 192) - || !text_ok(&query.q, 512) - || !text_ok(&query.client_ip, 128) - || !text_ok(&query.vhost, 256) - || query.offset.unwrap_or(0) > FILE_LOG_MAX_OFFSET - || query - .limit - .is_some_and(|v| v == 0 || v > FILE_LOG_MAX_LIMIT) - || query.event.as_ref().is_some_and(|v| { - !matches!( - v.as_str(), - "" | "all" | "request" | "protocol_error" | "error" - ) - }) - || query.status_class.as_ref().is_some_and(|v| { - !matches!( - v.as_str(), - "" | "all" | "1xx" | "2xx" | "3xx" | "4xx" | "5xx" - ) - }) - { - return Err(ApiError( - StatusCode::BAD_REQUEST, - "invalid file log filter".into(), - )); - } - Ok(()) -} - -async fn file_logs_query( - State(state): State>, - Query(query): Query, -) -> Result, ApiError> { - validate_file_log_query(&query)?; - let search = parse_file_log_search(query.q.as_deref().unwrap_or(""))?; - let config = state.config.load().envelope.config.clone(); - tokio::task::spawn_blocking(move || { - let mut sources = configured_file_log_sources(&config); - let log_format = config.logging.format.clone(); - let requested = query.source.as_deref().unwrap_or("all"); - let mut truncated = false; - if requested != "all" { - sources.retain(|source| source.id == requested); - if sources.is_empty() { - return Err(ApiError( - StatusCode::BAD_REQUEST, - "unknown file log source".into(), - )); - } - } else { - sources.sort_by(|a, b| b.modified_ms.cmp(&a.modified_ms)); - if sources.len() > FILE_LOG_MAX_SOURCES { - truncated = true; - sources.truncate(FILE_LOG_MAX_SOURCES); - } - } - let offset = query.offset.unwrap_or(0); - let limit = query.limit.unwrap_or(100).clamp(1, FILE_LOG_MAX_LIMIT); - let mut remaining_bytes = FILE_LOG_SCAN_BYTES_TOTAL; - let mut scanned_bytes = 0u64; - let mut remaining_lines = FILE_LOG_SCAN_LINES; - let mut scanned_lines = 0usize; - let mut records = Vec::new(); - let mut fields = std::collections::BTreeSet::new(); - for (index, source) in sources.iter().enumerate() { - if remaining_bytes == 0 || remaining_lines == 0 { - truncated = true; - break; - } - let sources_left = sources.len().saturating_sub(index).max(1); - let line_budget = (remaining_lines / sources_left).max(1); - match read_tail(source, remaining_bytes, line_budget) { - Ok((lines, bytes, source_truncated)) => { - remaining_bytes = remaining_bytes.saturating_sub(bytes); - remaining_lines = remaining_lines.saturating_sub(lines.len()); - scanned_bytes += bytes; - scanned_lines += lines.len(); - truncated |= source_truncated; - for line in lines { - if line.is_empty() { - continue; - } - let value = normalized_record(source, &line, &log_format); - if record_matches(&value, &query, &search) { - collect_file_log_fields(&value, &mut fields); - records.push(value); - } - } - } - Err(error) if error.kind() == std::io::ErrorKind::NotFound => {} - Err(_) => truncated = true, - } - } - records.sort_by(|a, b| { - let numeric = |value: &Value| value.get("time_ms").and_then(Value::as_u64); - match (numeric(a), numeric(b)) { - (Some(left), Some(right)) => right.cmp(&left), - _ => b - .get("timestamp") - .and_then(Value::as_str) - .unwrap_or("") - .cmp(a.get("timestamp").and_then(Value::as_str).unwrap_or("")), - } - }); - let total = records.len(); - let page = records - .into_iter() - .skip(offset) - .take(limit) - .collect::>(); - Ok(Json(json!({ - "entries": page, - "total": total, - "offset": offset, - "limit": limit, - "truncated": truncated, - "scanned_bytes": scanned_bytes, - "scanned_lines": scanned_lines, - "fields": fields, - "sources": file_log_sources_json(&sources), - }))) - }) - .await - .map_err(|_| { - ApiError( - StatusCode::INTERNAL_SERVER_ERROR, - "file log query worker failed".into(), - ) - })? -} +include!("file_logs_query.rs"); diff --git a/src/admin/file_logs_io.rs b/src/admin/file_logs_io.rs index ac31a3f..a34b622 100644 --- a/src/admin/file_logs_io.rs +++ b/src/admin/file_logs_io.rs @@ -49,6 +49,7 @@ fn rotated_gzip_sources( entries.into_iter().map(|(name,path,_)| source_from_path( format!("{id_prefix}:gz:{name}"), format!("{label_prefix} · {name}"), + id_prefix.to_owned(), kind, path, enabled, format )).collect() } diff --git a/src/admin/file_logs_query.rs b/src/admin/file_logs_query.rs new file mode 100644 index 0000000..69753fa --- /dev/null +++ b/src/admin/file_logs_query.rs @@ -0,0 +1,232 @@ +fn contains_ci(value: Option<&Value>, needle: &str) -> bool { + value + .and_then(Value::as_str) + .is_some_and(|text| text.to_ascii_lowercase().contains(needle)) +} + +fn record_matches(value: &Value, query: &FileLogQuery, search: &[FileLogSearchClause]) -> bool { + let Some(object) = value.as_object() else { + return false; + }; + if !record_matches_search(value, search) { + return false; + } + if let Some(event) = query + .event + .as_deref() + .filter(|v| !v.is_empty() && *v != "all") + { + if object.get("event").and_then(Value::as_str) != Some(event) { + return false; + } + } + if let Some(class) = query + .status_class + .as_deref() + .filter(|v| !v.is_empty() && *v != "all") + { + let Some(status) = object.get("status").and_then(Value::as_u64) else { + return false; + }; + let expected = class + .as_bytes() + .first() + .and_then(|v| v.checked_sub(b'0')) + .map(u64::from); + if expected != Some(status / 100) { + return false; + } + } + if let Some(client) = query.client_ip.as_deref().filter(|v| !v.is_empty()) { + if !contains_ci(object.get("client_ip"), &client.to_ascii_lowercase()) { + return false; + } + } + if let Some(vhost) = query.vhost.as_deref().filter(|v| !v.is_empty()) { + if !contains_ci(object.get("vhost"), &vhost.to_ascii_lowercase()) { + return false; + } + } + true +} + +fn validate_file_log_query(query: &FileLogQuery) -> Result<(), ApiError> { + let text_ok = |value: &Option, max: usize| { + value + .as_ref() + .is_none_or(|v| v.len() <= max && !v.chars().any(char::is_control)) + }; + if !text_ok(&query.source, 192) + || !text_ok(&query.q, 512) + || !text_ok(&query.client_ip, 128) + || !text_ok(&query.vhost, 256) + || query.aggregate_field.as_ref().is_some_and(|field| !valid_raw_field_key(field)) + || query.aggregate_limit.is_some_and(|value| value == 0 || value > FILE_LOG_MAX_AGGREGATE_RESULTS) + || query.aggregate_min_count.is_some_and(|value| value < 2 || value > FILE_LOG_SCAN_LINES) + || query.offset.unwrap_or(0) > FILE_LOG_MAX_OFFSET + || query + .limit + .is_some_and(|v| v == 0 || v > FILE_LOG_MAX_LIMIT) + || query.event.as_ref().is_some_and(|v| { + !matches!( + v.as_str(), + "" | "all" | "request" | "protocol_error" | "error" + ) + }) + || query.status_class.as_ref().is_some_and(|v| { + !matches!( + v.as_str(), + "" | "all" | "1xx" | "2xx" | "3xx" | "4xx" | "5xx" + ) + }) + { + return Err(ApiError( + StatusCode::BAD_REQUEST, + "invalid file log filter".into(), + )); + } + Ok(()) +} + +async fn file_logs_query( + State(state): State>, + Query(query): Query, +) -> Result, ApiError> { + validate_file_log_query(&query)?; + let search = parse_file_log_search(query.q.as_deref().unwrap_or(""))?; + let config = state.config.load().envelope.config.clone(); + tokio::task::spawn_blocking(move || { + let mut sources = configured_file_log_sources(&config); + let log_format = config.logging.format.clone(); + let requested = query.source.as_deref().unwrap_or("all"); + let mut truncated = false; + if requested != "all" { + let consolidate_vhosts = query.consolidate_vhosts.unwrap_or(false) && requested.starts_with("vhost:"); + sources.retain(|source| source.id == requested || (consolidate_vhosts && source.group_id == requested)); + if sources.is_empty() { + return Err(ApiError( + StatusCode::BAD_REQUEST, + "unknown file log source".into(), + )); + } + } else { + sources.sort_by(|a, b| b.modified_ms.cmp(&a.modified_ms)); + if sources.len() > FILE_LOG_MAX_SOURCES { + truncated = true; + sources.truncate(FILE_LOG_MAX_SOURCES); + } + } + let offset = query.offset.unwrap_or(0); + let limit = query.limit.unwrap_or(100).clamp(1, FILE_LOG_MAX_LIMIT); + let mut remaining_bytes = FILE_LOG_SCAN_BYTES_TOTAL; + let mut scanned_bytes = 0u64; + let mut remaining_lines = FILE_LOG_SCAN_LINES; + let mut scanned_lines = 0usize; + let mut records = Vec::new(); + let mut fields = std::collections::BTreeSet::new(); + let aggregate_field = query.aggregate_field.as_deref().map(str::to_ascii_lowercase); + let aggregate_min_count = query.aggregate_min_count.unwrap_or(2); + let aggregate_limit = query.aggregate_limit.unwrap_or(50).clamp(1, FILE_LOG_MAX_AGGREGATE_RESULTS); + let mut aggregate_counts = std::collections::HashMap::::new(); + let mut aggregate_truncated = false; + for (index, source) in sources.iter().enumerate() { + if remaining_bytes == 0 || remaining_lines == 0 { + truncated = true; + break; + } + let sources_left = sources.len().saturating_sub(index).max(1); + let line_budget = (remaining_lines / sources_left).max(1); + match read_tail(source, remaining_bytes, line_budget) { + Ok((lines, bytes, source_truncated)) => { + remaining_bytes = remaining_bytes.saturating_sub(bytes); + remaining_lines = remaining_lines.saturating_sub(lines.len()); + scanned_bytes += bytes; + scanned_lines += lines.len(); + truncated |= source_truncated; + for line in lines { + if line.is_empty() { + continue; + } + let value = normalized_record(source, &line, &log_format); + if record_matches(&value, &query, &search) { + collect_file_log_fields(&value, &mut fields); + if let Some(field) = aggregate_field.as_deref() { + if let Some(field_value) = file_log_field(&value, field) { + let mut values = Vec::new(); + file_log_aggregate_values(field_value, &mut values); + for aggregate_value in values { + if aggregate_counts.contains_key(&aggregate_value) || aggregate_counts.len() < FILE_LOG_MAX_AGGREGATE_VALUES { + *aggregate_counts.entry(aggregate_value).or_insert(0) += 1; + } else { + aggregate_truncated = true; + } + } + } + } + records.push(value); + } + } + } + Err(error) if error.kind() == std::io::ErrorKind::NotFound => {} + Err(_) => truncated = true, + } + } + records.sort_by(|a, b| { + let numeric = |value: &Value| value.get("time_ms").and_then(Value::as_u64); + match (numeric(a), numeric(b)) { + (Some(left), Some(right)) => right.cmp(&left), + _ => b + .get("timestamp") + .and_then(Value::as_str) + .unwrap_or("") + .cmp(a.get("timestamp").and_then(Value::as_str).unwrap_or("")), + } + }); + let mut aggregation = aggregate_counts.into_iter() + .filter(|(_, count)| *count >= aggregate_min_count) + .map(|(value, count)| json!({"value": value, "count": count})) + .collect::>(); + aggregation.sort_by(|left, right| { + let left_count = left.get("count").and_then(Value::as_u64).unwrap_or(0); + let right_count = right.get("count").and_then(Value::as_u64).unwrap_or(0); + right_count.cmp(&left_count).then_with(|| { + left.get("value").and_then(Value::as_str).unwrap_or("") + .cmp(right.get("value").and_then(Value::as_str).unwrap_or("")) + }) + }); + if aggregation.len() > aggregate_limit { + aggregation.truncate(aggregate_limit); + aggregate_truncated = true; + } + let total = records.len(); + let page = records + .into_iter() + .skip(offset) + .take(limit) + .collect::>(); + Ok(Json(json!({ + "entries": page, + "total": total, + "offset": offset, + "limit": limit, + "truncated": truncated, + "scanned_bytes": scanned_bytes, + "scanned_lines": scanned_lines, + "fields": fields, + "sources": file_log_sources_json(&sources), + "aggregation": aggregate_field.map(|field| json!({ + "field": field, + "min_count": aggregate_min_count, + "entries": aggregation, + "truncated": aggregate_truncated, + })), + }))) + }) + .await + .map_err(|_| { + ApiError( + StatusCode::INTERNAL_SERVER_ERROR, + "file log query worker failed".into(), + ) + })? +} diff --git a/src/admin/logging_api.rs b/src/admin/logging_api.rs index 52d04b6..0cffded 100644 --- a/src/admin/logging_api.rs +++ b/src/admin/logging_api.rs @@ -113,6 +113,10 @@ async fn update_section( config.host_policies = serde_json::from_value(value).map_err(|e| invalid_section(§ion, e))? } + "user-agent-blocking" => { + config.user_agent_blocking = + serde_json::from_value(value).map_err(|e| invalid_section(§ion, e))? + } _ => { return Err(ApiError( StatusCode::NOT_FOUND, diff --git a/src/admin/openapi.json b/src/admin/openapi.json index c6b3a0f..3fba555 100644 --- a/src/admin/openapi.json +++ b/src/admin/openapi.json @@ -2,7 +2,7 @@ "openapi": "3.0.3", "info": { "title": "RustProxy Administration API", - "version": "0.31.4", + "version": "0.31.5", "description": "Administration API for the active panel origin. Sign in through the panel; the same-origin HttpOnly cookie authenticates all private operations. Mutations also require the exact Origin, X-RustProxy-Request: 1 and a session-bound X-CSRF-Token. Loopback HTTP is the default; Settings can enable HTTPS before using a non-loopback administration bind. This document contains no live configuration or credentials. Object schemas marked extensible are summaries: full configuration validation is authoritative in Rust." }, "servers": [ @@ -659,6 +659,9 @@ ], "nullable": true, "description": "Error fallback for the main vhost source." + }, + "user_agent_blocking": { + "$ref": "#/components/schemas/UserAgentBlocking" } }, "additionalProperties": true, @@ -917,6 +920,9 @@ "$ref": "#/components/schemas/PhpFpmProfile" }, "description": "Named PHP-FPM configurations reusable by Web Services." + }, + "user_agent_blocking": { + "$ref": "#/components/schemas/UserAgentBlocking" } }, "additionalProperties": true @@ -5607,6 +5613,14 @@ "modified_ms": { "type": "integer", "format": "int64" + }, + "group_id": { + "type": "string", + "description": "Stable logical source group. Per-vhost active JSONL and rotated .gz members share the same vhost: group." + }, + "compressed": { + "type": "boolean", + "description": "True for a rotated gzip source member." } }, "additionalProperties": false @@ -5675,6 +5689,52 @@ "items": { "$ref": "#/components/schemas/FileLogSource" } + }, + "aggregation": { + "type": [ + "object", + "null" + ], + "description": "Optional repeated-value aggregation computed across the bounded matching scan before pagination.", + "properties": { + "field": { + "type": "string" + }, + "min_count": { + "type": "integer", + "minimum": 2 + }, + "entries": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "properties": { + "value": { + "type": "string" + }, + "count": { + "type": "integer", + "minimum": 1 + } + }, + "required": [ + "value", + "count" + ] + } + }, + "truncated": { + "type": "boolean" + } + }, + "additionalProperties": false, + "required": [ + "field", + "min_count", + "entries", + "truncated" + ] } }, "additionalProperties": false @@ -5887,6 +5947,25 @@ "additionalProperties": true } } + }, + "UserAgentBlocking": { + "type": "object", + "additionalProperties": false, + "properties": { + "enabled": { + "type": "boolean", + "default": false + }, + "patterns": { + "type": "array", + "maxItems": 128, + "items": { + "type": "string", + "maxLength": 256 + }, + "description": "Case-insensitive User-Agent block patterns. Plain text matches a substring; * is a wildcard." + } + } } } }, @@ -8686,7 +8765,8 @@ "not_found_hosts", "streams", "acl-profiles", - "host-policies" + "host-policies", + "user-agent-blocking" ] }, "description": "Configuration section to replace." @@ -10720,6 +10800,16 @@ "type": "string" } }, + { + "name": "consolidate_vhosts", + "in": "query", + "required": false, + "description": "When true and source is vhost:, scan the active JSONL plus all discovered rotated .gz members that share that vhost group.", + "schema": { + "type": "boolean", + "default": false + } + }, { "name": "q", "in": "query", @@ -10782,6 +10872,40 @@ "maxLength": 256 } }, + { + "name": "aggregate_field", + "in": "query", + "required": false, + "description": "Optional parsed field to count across matching records before pagination. Supports nested fields accepted by Stored Logs search.", + "schema": { + "type": "string", + "maxLength": 192 + } + }, + { + "name": "aggregate_min_count", + "in": "query", + "required": false, + "description": "Minimum occurrences required for an aggregate value.", + "schema": { + "type": "integer", + "minimum": 2, + "maximum": 20000, + "default": 2 + } + }, + { + "name": "aggregate_limit", + "in": "query", + "required": false, + "description": "Maximum aggregate values returned.", + "schema": { + "type": "integer", + "minimum": 1, + "maximum": 200, + "default": 50 + } + }, { "name": "offset", "in": "query", @@ -10899,4 +11023,4 @@ } } } -} \ No newline at end of file +} diff --git a/src/admin/panel_assets.rs b/src/admin/panel_assets.rs index 0f20ab6..5d56b0c 100644 --- a/src/admin/panel_assets.rs +++ b/src/admin/panel_assets.rs @@ -47,7 +47,13 @@ static ASSETS: LazyLock> = LazyLock::new(|| { }); pub(super) async fn serve(State(state): State>, request: Request) -> Response { - let Some(asset) = ASSETS.get(request.uri().path()) else { + let resolved = super::fingerprinted_path(request.uri().path(), "/assets/"); + let path = match resolved.as_deref() { + Some(path) => path, + None if matches!(request.uri().path(), "/assets/login-langs.json" | "/assets/langs.json" | "/favicon.svg") => request.uri().path(), + None => return StatusCode::NOT_FOUND.into_response(), + }; + let Some(asset) = ASSETS.get(path) else { return StatusCode::NOT_FOUND.into_response(); }; let snapshot = state.config.load(); diff --git a/src/admin/public.rs b/src/admin/public.rs index 5cf37be..55872e5 100644 --- a/src/admin/public.rs +++ b/src/admin/public.rs @@ -11,6 +11,7 @@ static ADMIN_INDEX: OnceLock = OnceLock::new(); fn panel_response(headers: &HeaderMap, compression: &PanelCompression) -> Response { let content = ADMIN_INDEX.get_or_init(|| { let html = include_str!("web/index.html") + .replace("__RUSTPROXY_ASSET_ID__", super::asset_build_id()) .replace("__RUSTPROXY_VERSION__", env!("CARGO_PKG_VERSION")); // Exactly one static panel document shared between all administrator accounts. PanelContent::new(Box::leak(html.into_boxed_str())) @@ -24,7 +25,9 @@ pub(super) async fn index( headers: HeaderMap, ) -> Response { if session.is_none() { - return Html(include_str!("web/login.html")).into_response(); + let html = include_str!("web/login.html") + .replace("__RUSTPROXY_ASSET_ID__", super::asset_build_id()); + return Html(html).into_response(); } let snapshot = state.config.load(); panel_response(&headers, &snapshot.envelope.config.administration.panel_compression) diff --git a/src/admin/public/tests.rs b/src/admin/public/tests.rs index a5a1fc8..853b798 100644 --- a/src/admin/public/tests.rs +++ b/src/admin/public/tests.rs @@ -16,7 +16,9 @@ async fn panel_chooses_zstd_brotli_gzip_by_quality_and_support() { assert!(!plain.headers().contains_key(header::CONTENT_ENCODING)); let source = plain.into_body().collect().await.unwrap().to_bytes(); assert!(std::str::from_utf8(&source).unwrap().contains("id=\"app-startup\"")); - assert!(std::str::from_utf8(&source).unwrap().contains(env!("CARGO_PKG_VERSION"))); + assert!(std::str::from_utf8(&source).unwrap().contains(super::super::asset_build_id())); + assert!(std::str::from_utf8(&source).unwrap().contains(&format!("RustProxy {}", env!("CARGO_PKG_VERSION")))); + assert!(!std::str::from_utf8(&source).unwrap().contains("__RUSTPROXY_VERSION__")); for (accept, selected) in [ ("gzip, br, zstd", "zstd"), ("gzip, br", "br"), diff --git a/src/admin/router.rs b/src/admin/router.rs index 6b734c5..c7ab565 100644 --- a/src/admin/router.rs +++ b/src/admin/router.rs @@ -2,6 +2,21 @@ mod documentation; mod public; mod panel_content; mod panel_assets; + +// Embedded asset URLs change with every build, independent of Cargo package version. +// Only an exact identifier from this binary is accepted; arbitrary suffixes are 404. +fn asset_build_id() -> &'static str { + include_str!(concat!(env!("OUT_DIR"), "/asset-build-id.txt")) +} +fn fingerprinted_path(path: &str, prefix: &str) -> Option { + let name = path.strip_prefix(prefix)?; + let (stem, extension) = name.rsplit_once('.')?; + if extension != "js" && extension != "css" { return None; } + let (base, token) = stem.rsplit_once('_')?; + if token != asset_build_id() { return None; } + Some(format!("{prefix}{base}.{extension}")) +} + use crate::{ auth::{self, Session}, config::Update, @@ -107,45 +122,18 @@ pub fn router(state: Arc) -> Router { .route("/logging", get(public::index)) .route("/logs", get(public::index)) .route("/settings", get(public::index)) - .route("/assets/login.js", get(panel_assets::serve)) + .route("/assets/{fingerprinted}", get(panel_assets::serve)) .route("/assets/login-langs.json", get(panel_assets::serve)) .route("/docs", get(documentation::index)) .route("/docs/", get(documentation::index)) .route("/docs/openapi.json", get(documentation::schema)) .route("/docs/content.json", get(documentation::content)) - .route("/docs/init.js", get(documentation::script)) - .route("/docs/style.css", get(documentation::style)) - .route( - "/docs/swagger-ui-bundle.js", - get(|| async { documentation::asset("swagger-ui-bundle.js") }), - ) - .route( - "/docs/swagger-ui.css", - get(|| async { documentation::asset("swagger-ui.css") }), - ) + .route("/docs/{fingerprinted}", get(documentation::fingerprinted)) .route( "/docs/licenses.txt", get(|| async { documentation::asset("licenses.txt") }), ) .route("/assets/langs.json", get(panel_assets::serve)) - .route("/assets/app.js", get(panel_assets::serve)) - .route("/assets/management.js", get(panel_assets::serve)) - .route("/assets/drafts.js", get(panel_assets::serve)) - .route("/assets/integrations.js", get(panel_assets::serve)) - .route("/assets/traffic-panel.js", get(panel_assets::serve)) - .route("/assets/cache-panel.js", get(panel_assets::serve)) - .route("/assets/transform-panel.js", get(panel_assets::serve)) - .route("/assets/file-logs.js", get(panel_assets::serve)) - .route("/assets/settings.js", get(panel_assets::serve)) - .route("/assets/acme.js", get(panel_assets::serve)) - .route("/assets/vhosts.js", get(panel_assets::serve)) - .route("/assets/web-server-profiles.js", get(panel_assets::serve)) - .route("/assets/web-server.js", get(panel_assets::serve)) - .route("/assets/docker.js", get(panel_assets::serve)) - .route("/assets/streams.js", get(panel_assets::serve)) - .route("/assets/redirects.js", get(panel_assets::serve)) - .route("/assets/host404.js", get(panel_assets::serve)) - .route("/assets/style.css", get(panel_assets::serve)) .route("/favicon.svg", get(panel_assets::serve)) .route("/healthz", get(|| async { Json(json!({"status":"ok"})) })) .route("/api/status", get(status)) @@ -309,11 +297,11 @@ fn panel_document_path(path: &str) -> bool { fn anonymous_allowed(method: &Method, path: &str) -> bool { if method == Method::GET || method == Method::HEAD { panel_document_path(path) + || fingerprinted_path(path, "/assets/") + .is_some_and(|base| matches!(base.as_str(), "/assets/login.js" | "/assets/style.css")) || matches!( path, - "/assets/login.js" - | "/assets/login-langs.json" - | "/assets/style.css" + "/assets/login-langs.json" | "/favicon.svg" | "/healthz" | "/api/auth/status" diff --git a/src/admin/tests/part_02.rs b/src/admin/tests/part_02.rs index a4a4020..3d6e3a7 100644 --- a/src/admin/tests/part_02.rs +++ b/src/admin/tests/part_02.rs @@ -397,7 +397,7 @@ .unwrap(); assert!(std::str::from_utf8(&public_body) .unwrap() - .contains("/assets/login.js")); + .contains(&format!("/assets/login_{}.js", super::asset_build_id()))); let private = router .clone() @@ -415,7 +415,7 @@ .unwrap(); assert!(std::str::from_utf8(&private_body) .unwrap() - .contains("/assets/app.js")); + .contains(&format!("/assets/app_{}.js", super::asset_build_id()))); } store.close().await; } diff --git a/src/admin/tests/part_05.rs b/src/admin/tests/part_05.rs index 41ddba7..0d0adb0 100644 --- a/src/admin/tests/part_05.rs +++ b/src/admin/tests/part_05.rs @@ -15,13 +15,9 @@ "/", "/docs", "/docs/", - "/docs/init.js", - "/docs/style.css", - "/docs/swagger-ui-bundle.js", - "/docs/swagger-ui.css", "/docs/licenses.txt", "/docs/content.json", - "/assets/app.js", + &format!("/assets/app_{}.js", super::asset_build_id()), ] { let response = router .clone() diff --git a/src/admin/tests/part_06.rs b/src/admin/tests/part_06.rs index d59a009..bd742d2 100644 --- a/src/admin/tests/part_06.rs +++ b/src/admin/tests/part_06.rs @@ -34,6 +34,7 @@ fn stored_log_tail_enforces_line_budget() { let source = source_from_path( "access".into(), "Access log".into(), + "access".into(), "access", path, false, @@ -61,6 +62,7 @@ fn stored_log_normalizes_mapped_client_ip_and_searches_diagnostics() { let source = source_from_path( "vhost:app".into(), "app.jsonl".into(), + "vhost:app".into(), "access", dir.path().join("app.jsonl"), true, @@ -168,6 +170,7 @@ fn stored_log_tail_reads_rotated_gzip_and_inventory_discovers_it() { let source = source_from_path( "access:gz:access.jsonl.1.gz".into(), "Access log · access.jsonl.1.gz".into(), + "access".into(), "access", rotated.clone(), true, @@ -186,3 +189,48 @@ fn stored_log_tail_reads_rotated_gzip_and_inventory_discovers_it() { let sources = configured_file_log_sources(&config); assert!(sources.iter().any(|item| item.path == rotated && item.compressed)); } + +#[test] +fn stored_log_per_vhost_inventory_groups_active_and_rotated_files() { + use std::io::Write; + + let dir = tempfile::tempdir().unwrap(); + let active = dir.path().join("app.jsonl"); + std::fs::write(&active, b"{\"event\":\"request\"}\n").unwrap(); + let rotated = dir.path().join("app.jsonl.1.gz"); + let file = std::fs::File::create(&rotated).unwrap(); + let mut encoder = flate2::write::GzEncoder::new(file, flate2::Compression::default()); + writeln!(encoder, r#"{{"event":"request"}}"#).unwrap(); + encoder.finish().unwrap(); + + let mut config = Config::default(); + config.logging.file.per_vhost = true; + config.logging.file.directory = dir.path().to_string_lossy().into_owned(); + let sources = configured_file_log_sources(&config); + let app = sources.iter().filter(|source| source.group_id == "vhost:app").collect::>(); + assert_eq!(app.len(), 2); + assert!(app.iter().any(|source| source.id == "vhost:app")); + assert!(app.iter().any(|source| source.compressed)); +} + +#[test] +fn stored_log_aggregate_values_support_scalars_and_bounded_arrays() { + let value = json!(["BadBot/1", "BadBot/1", 503, true, {"ignored":"object"}]); + let mut values = Vec::new(); + file_log_aggregate_values(&value, &mut values); + assert_eq!(values, vec!["BadBot/1".to_string(), "BadBot/1".to_string(), "503".to_string(), "true".to_string()]); +} + +#[test] +fn stored_log_query_validates_aggregation_controls() { + assert!(validate_file_log_query(&FileLogQuery { + aggregate_field: Some("user_agent".into()), + aggregate_min_count: Some(2), + aggregate_limit: Some(50), + ..FileLogQuery::default() + }).is_ok()); + assert!(validate_file_log_query(&FileLogQuery { + aggregate_field: Some("bad field".into()), + ..FileLogQuery::default() + }).is_err()); +} diff --git a/src/admin/tests/part_07.rs b/src/admin/tests/part_07.rs index 316e94a..7615e37 100644 --- a/src/admin/tests/part_07.rs +++ b/src/admin/tests/part_07.rs @@ -1,4 +1,35 @@ #[tokio::test] + async fn only_current_build_fingerprinted_javascript_and_css_are_served() { + let (router, store, _dir, secret, _) = fixture().await; + let cookie = format!("{}={secret}", auth::COOKIE_NAME); + let id = super::asset_build_id(); + for (name, suffix) in [ + ("/assets/app", "js"), + ("/assets/host404", "js"), + ("/assets/style", "css"), + ("/assets/login", "js"), + ("/docs/init", "js"), + ("/docs/style", "css"), + ("/docs/swagger-ui-bundle", "js"), + ("/docs/swagger-ui", "css"), + ] { + let legacy = format!("{name}.{suffix}"); + let wrong = format!("{name}_0000000000000000.{suffix}"); + let current = format!("{name}_{id}.{suffix}"); + for path in [legacy, wrong] { + let response = router.clone().oneshot(request(&path) + .header("cookie", &cookie).body(Body::empty()).unwrap()) + .await.unwrap(); + assert_eq!(response.status(), StatusCode::NOT_FOUND, "{path}"); + } + let response = router.clone().oneshot(request(¤t) + .header("cookie", &cookie).body(Body::empty()).unwrap()) + .await.unwrap(); + assert_eq!(response.status(), StatusCode::OK, "{current}"); + } + store.close().await; + } + #[tokio::test] async fn public_document_and_discovery_contain_only_login_information() { let (router, store, _dir, _, _) = fixture().await; for (path, expected) in [ @@ -22,7 +53,7 @@ .await .unwrap(); let text = std::str::from_utf8(&bytes).unwrap(); - assert!(text.contains("/assets/login.js")); + assert!(text.contains(&format!("/assets/login_{}.js", super::asset_build_id()))); for absent in [ "/assets/app.js", "id=\"workspace\"", diff --git a/src/admin/web/app/core.js b/src/admin/web/app/core.js index dccbf3d..c45b279 100644 --- a/src/admin/web/app/core.js +++ b/src/admin/web/app/core.js @@ -116,7 +116,7 @@ function appConfirm(message, options = {}) { } function apiArea(path) { const section = path.match(/^\/api\/config\/([^/?]+)/)?.[1]; - if (section) return ({ "listeners": "settings", "http2": "settings", "docker-agents": "dockerVhosts", "response": "settings", "pages": "settings", "cache": "cache", "transform": "settings", "logging": "settings", "geoip": "settings", "forwarding": "settings", "headers": "settings", "prometheus": "settings", "administration": "administration", "tls": "tls", "vhosts": "vhosts", "redirects": "redirects", "not_found_hosts": "notfound", "streams": "streams", "acl-profiles": "aclProfiles", "host-policies": "security" })[section] || "configuration"; + if (section) return ({ "listeners": "settings", "http2": "settings", "docker-agents": "dockerVhosts", "response": "settings", "pages": "settings", "cache": "cache", "transform": "settings", "logging": "settings", "geoip": "settings", "forwarding": "settings", "headers": "settings", "prometheus": "settings", "administration": "administration", "tls": "tls", "vhosts": "vhosts", "redirects": "redirects", "not_found_hosts": "notfound", "streams": "streams", "acl-profiles": "aclProfiles", "host-policies": "security", "user-agent-blocking": "security" })[section] || "configuration"; if (path === "/api/config") return "configuration"; if (path.startsWith("/api/traffic")) return "traffic"; if (path.startsWith("/api/cache")) return "cache"; @@ -345,6 +345,9 @@ function renderConfig() { $("host-policy-list").replaceChildren(...(config.host_policies || []).map((policy, index) => { const box = el("div", undefined, "entity"); box.append(el("span", policy.hosts.join(", "), "entity-title"), el("p", `${t("aclDefault")}: ${t(policy.acl.default || "deny")}\n${rulesText(policy.acl)}`, "entity-details")); const buttons = el("div", undefined, "entity-actions"); buttons.append(action(t("edit"), () => { $("policy-hosts").value = policy.hosts.join(", "); $("policy-default").value = policy.acl.default || "deny"; $("policy-rules").value = rulesText(policy.acl); const editor = $("security-policy-editor"); if (editor) { editor.open = true; editor.scrollIntoView({ block: "nearest" }); } }), action(t("remove"), async () => { if (await appConfirm(t("confirmDelete"), { danger: true })) { const next = structuredClone(config); next.host_policies.splice(index, 1); await saveSection("host-policies", next.host_policies); } })); box.append(buttons); return box; })); if (!config.host_policies?.length) $("host-policy-list").append(el("p", t("noPolicies"), "empty")); if ($("security-policy-count")) $("security-policy-count").textContent = String((config.host_policies || []).length); + const uaPolicy = config.user_agent_blocking || {}; + if ($("global-ua-block-enabled")) $("global-ua-block-enabled").checked = uaPolicy.enabled === true; + if ($("global-ua-block-patterns")) $("global-ua-block-patterns").value = (uaPolicy.patterns || []).join("\n"); } async function saveRequest(path, payload) { if (busy) { toast(t("busy"), true); return false; } if (!authenticated || !envelope) return false; busy = true; const epoch = authEpoch; try { const value = await api(path, { method: "PUT", body: JSON.stringify(payload) }); if (!authenticated || epoch !== authEpoch) return false; envelope = value; if (path === "/api/config" || path === "/api/config/tls") await refreshCertificateInventory(); if (!authenticated || epoch !== authEpoch || !envelope) return false; renderConfig(); toast(t("saved")); await refresh(true); return true; } catch (e) { toast(e.message, true); if (e.status === 401) lock(); return false; } finally { busy = false; } } async function save(config) { return saveRequest("/api/config", { expected_revision: envelope?.revision, config }); } @@ -464,6 +467,7 @@ document.addEventListener("click", event => { if (dashboardSidebar?.classList.co document.addEventListener("keydown", event => { if (event.key === "Escape" && dashboardSidebar?.classList.contains("mobile-nav-open")) { setMobileMenu(false); mobileMenuToggle?.focus(); } }); window.addEventListener("popstate", () => { setPanel(panelFromLocation(), false); setMobileMenu(false); }); $("host-policy-form").addEventListener("submit", async event => { event.preventDefault(); const form = event.currentTarget; if (!envelope || busy) return; try { await withFormSubmitState(form, event.submitter, async () => { const hosts = [...new Set($("policy-hosts").value.split(",").map(h => h.trim().toLowerCase()).filter(Boolean))]; const acl = { default: $("policy-default").value, rules: parseRules($("policy-rules").value, $("policy-rules")) }; const next = structuredClone(envelope.config); next.host_policies = (next.host_policies || []).map(p => ({ ...p, hosts: p.hosts.filter(h => !hosts.includes(h)) })).filter(p => p.hosts.length); next.host_policies.push({ hosts, acl }); if (await saveSection("host-policies", next.host_policies)) { form.reset(); if ($("security-policy-editor")) $("security-policy-editor").open = false; } }); } catch (e) { toast(e.message, true); } }); +$("global-ua-block-form")?.addEventListener("submit", async event => { event.preventDefault(); const form = event.currentTarget; if (!envelope || busy) return; try { await withFormSubmitState(form, event.submitter, async () => { const patterns = [...new Set($("global-ua-block-patterns").value.split(/\r?\n/).map(value => value.trim()).filter(Boolean))]; await saveSection("user-agent-blocking", { enabled: $("global-ua-block-enabled").checked, patterns }); }); } catch (e) { toast(e.message, true); } }); $("password-form").addEventListener("submit", async event => { event.preventDefault(); const form = event.currentTarget; try { await withFormSubmitState(form, event.submitter, async () => { if ($("new-password").value !== $("new-password-confirm").value) throw fieldError("new-password-confirm", t("passwordMismatch")); await api("/api/auth/password", { method: "POST", body: JSON.stringify({ current_password: $("current-password").value, new_password: $("new-password").value }) }); lock(); toast(t("passwordChanged")); }); } catch (e) { toast(e.message, true); } }); $("revoke-sessions").addEventListener("click", async () => { if (!(await appConfirm(t("confirmRevoke"), { danger: true }))) return; try { await api("/api/auth/logout-all", { method: "POST" }); lock(); toast(t("locked")); } catch (e) { toast(e.message, true); } }); $("security-audit-open").addEventListener("click", () => { const dialog = $("audit-dialog"); if (!dialog.open) dialog.showModal(); void loadAudit(); }); diff --git a/src/admin/web/docs.html b/src/admin/web/docs.html index 9837b71..0ecf3e7 100644 --- a/src/admin/web/docs.html +++ b/src/admin/web/docs.html @@ -6,10 +6,10 @@ RustProxy / Docs - - - - + + + + diff --git a/src/admin/web/file-logs.js b/src/admin/web/file-logs.js index d2825cc..4559d09 100644 --- a/src/admin/web/file-logs.js +++ b/src/admin/web/file-logs.js @@ -2,7 +2,7 @@ let inventory = null, offset = 0, lastTotal = 0, busy = false; let currentEntries = [], selectedIndex = null; let preferencesLoaded = false, preferenceSaveRunning = false, preferenceSaveQueued = false; - let searchTimer = null; + let searchTimer = null, aggregateField = "", activeView = "entries"; const defaultColumnIds = ["time", "event", "status", "client_ip", "vhost", "record", "source"]; const specialLabels = new Map([ ["time", "time"], ["event", "fileLogsEvent"], ["status", "status"], @@ -13,17 +13,57 @@ let visibleColumns = new Set(defaultColumnIds); + const CONSOLIDATION_STORAGE_KEY = "rp-file-logs-consolidate-vhosts"; + const consolidationToggle = $("file-log-consolidate-vhosts"); + try { + const saved = localStorage.getItem(CONSOLIDATION_STORAGE_KEY); + if (consolidationToggle && (saved === "1" || saved === "0")) consolidationToggle.checked = saved === "1"; + } catch { /* Browser storage can be unavailable; use the HTML default. */ } + const sourceSelect = () => $("file-log-source"); + function setView(view) { + activeView = view === "counts" ? "counts" : "entries"; + for (const name of ["entries", "counts"]) { + const selected = name === activeView; + $("file-log-tab-" + name).classList.toggle("active", selected); + $("file-log-tab-" + name).setAttribute("aria-selected", String(selected)); + $("file-log-tab-" + name).tabIndex = selected ? 0 : -1; + $("file-log-view-" + name).hidden = !selected; + } + if (activeView === "counts" && aggregateField) void refresh(true); + } function sourceOption(source) { - const label = `${source.label} · ${source.format} · ${bytes(Number(source.size) || 0)}`; + const grouped = Number(source.member_count) > 1 ? ` · ${fmt(source.member_count)} ${t("fileLogsFiles")}` : ""; + const label = `${source.label}${grouped} · ${source.format} · ${bytes(Number(source.size) || 0)}`; const option = el("option", label); option.value = source.id; return option; } + function displaySources(sources) { + if (!$("file-log-consolidate-vhosts")?.checked) return sources; + const output = [], groups = new Map(); + for (const source of sources) { + if (!String(source.group_id || "").startsWith("vhost:")) { output.push(source); continue; } + const id = source.group_id; + let group = groups.get(id); + if (!group) { + group = { ...source, id, label: id.slice("vhost:".length), size: 0, member_count: 0, compressed: false }; + groups.set(id, group); output.push(group); + } + group.size += Number(source.size) || 0; group.member_count += 1; + group.modified_ms = Math.max(Number(group.modified_ms) || 0, Number(source.modified_ms) || 0); + } + return output; + } function renderInventory(value) { inventory = value; const select = sourceSelect(); - const selected = select.value || "all"; - const sources = Array.isArray(value?.sources) ? value.sources : []; + let selected = select.value || "all"; + const rawSources = Array.isArray(value?.sources) ? value.sources : []; + if ($("file-log-consolidate-vhosts")?.checked && selected !== "all") { + const member = rawSources.find(source => source.id === selected); + if (member?.group_id?.startsWith("vhost:")) selected = member.group_id; + } + const sources = displaySources(rawSources); const options = [Object.assign(el("option", t("fileLogsAllSources")), { value: "all" }), ...sources.map(sourceOption)]; select.replaceChildren(...options); select.value = sources.some(source => source.id === selected) ? selected : "all"; @@ -35,6 +75,13 @@ } const currentLimit = () => Math.max(1, Math.min(500, Number($("file-log-limit").value) || 100)); + async function copyText(value) { + if (navigator.clipboard?.writeText) { + try { await navigator.clipboard.writeText(value); toast(t("copied")); return; } catch { } + } + const area = document.createElement("textarea"); area.value = value; area.setAttribute("readonly", ""); + area.style.position = "fixed"; area.style.opacity = "0"; document.body.append(area); area.select(); document.execCommand("copy"); area.remove(); toast(t("copied")); + } function entryTime(entry) { if (entry.time_ms) return formatDateTime(Number(entry.time_ms)); @@ -89,6 +136,11 @@ if (normalized) availableColumns.add(normalized); } for (const id of visibleColumns) availableColumns.add(id); + const suggestions = $("file-log-field-suggestions"); + if (suggestions) { + const names = new Set(["user_agent", "client_ip", "vhost", "host", "path", "method", "status", "country", "source", ...fields]); + suggestions.replaceChildren(...[...names].filter(Boolean).sort().map(name => Object.assign(el("option"), { value: normalizedField(name) || name }))); + } syncColumnPicker(); syncHeaders(); } function syncHeaders() { @@ -166,6 +218,19 @@ raw.append(summary, el("pre", JSON.stringify(entry, null, 2), "metrics-example")); detail.append(heading, fields, raw); cell.append(detail); row.append(cell); return row; } + function renderAggregation(aggregation) { + const rows = $("file-log-aggregate-rows"), empty = $("file-log-aggregate-empty"), state = $("file-log-aggregate-state"); + if (!rows || !empty || !state) return; + const entries = Array.isArray(aggregation?.entries) ? aggregation.entries : []; + rows.replaceChildren(...entries.map(item => { + const row = el("tr"), value = String(item.value ?? ""), valueCell = el("td"), button = el("button", value, "quiet file-log-aggregate-copy"); + button.type = "button"; button.title = t("copyValue"); button.addEventListener("click", () => void copyText(value)); valueCell.append(button); + row.append(valueCell, el("td", fmt(Number(item.count) || 0))); return row; + })); + empty.hidden = !aggregation || entries.length !== 0; + state.textContent = aggregation ? `${aggregation.field} · ${entries.length}${aggregation.truncated ? ` · ${t("fileLogsAggregateTruncated")}` : ""}` : ""; + } + function renderRows(entries = currentEntries) { currentEntries = entries; const ids = orderedColumns().filter(id => visibleColumns.has(id)); @@ -189,6 +254,12 @@ const source = sourceSelect().value || "all"; if (source !== "all") params.set("source", source); const q = $("file-log-q").value.trim(); if (q) params.set("q", q); + if ($("file-log-consolidate-vhosts")?.checked) params.set("consolidate_vhosts", "true"); + if (activeView === "counts" && aggregateField) { + params.set("aggregate_field", aggregateField); + params.set("aggregate_min_count", String(Math.max(2, Number($("file-log-aggregate-min")?.value) || 2))); + params.set("aggregate_limit", String(Math.max(1, Number($("file-log-aggregate-limit")?.value) || 50))); + } params.set("offset", String(offset)); params.set("limit", String(currentLimit())); return params.toString(); } @@ -202,6 +273,7 @@ lastTotal = Number(value.total) || 0; selectedIndex = null; applyFields(Array.isArray(value.fields) ? value.fields : []); renderRows(Array.isArray(value.entries) ? value.entries : []); + renderAggregation(value.aggregation || null); const from = lastTotal ? offset + 1 : 0, to = Math.min(offset + currentLimit(), lastTotal); $("file-log-summary").textContent = t("fileLogsSummary") .replace("{from}", fmt(from)).replace("{to}", fmt(to)).replace("{total}", fmt(lastTotal)) @@ -213,12 +285,14 @@ finally { busy = false; $("file-logs-refresh").disabled = false; } } function reset() { - inventory = null; offset = 0; lastTotal = 0; currentEntries = []; selectedIndex = null; + inventory = null; offset = 0; lastTotal = 0; currentEntries = []; selectedIndex = null; aggregateField = ""; preferencesLoaded = false; preferenceSaveRunning = false; preferenceSaveQueued = false; availableColumns = new Set(defaultColumnIds); visibleColumns = new Set(defaultColumnIds); sourceSelect().replaceChildren(Object.assign(el("option", t("fileLogsAllSources")), { value: "all" })); - $("file-log-rows").replaceChildren(); syncColumnPicker(); syncHeaders(); + $("file-log-rows").replaceChildren(); $("file-log-aggregate-rows")?.replaceChildren(); renderAggregation(null); syncColumnPicker(); syncHeaders(); } + $("file-log-tab-entries").addEventListener("click", () => setView("entries")); + $("file-log-tab-counts").addEventListener("click", () => setView("counts")); $("file-logs-open").addEventListener("click", () => { $("logs-menu").open = false; setPanel("filelogs"); }); $("settings-file-logs-open").addEventListener("click", () => setPanel("filelogs")); $("file-logs-refresh").addEventListener("click", () => refresh(true)); @@ -226,6 +300,12 @@ $("file-log-q").addEventListener("input", () => { clearTimeout(searchTimer); searchTimer = setTimeout(() => void refresh(true), 250); }); $("file-log-limit").addEventListener("change", () => refresh(true)); sourceSelect().addEventListener("change", () => refresh(true)); + consolidationToggle?.addEventListener("change", () => { + try { localStorage.setItem(CONSOLIDATION_STORAGE_KEY, consolidationToggle.checked ? "1" : "0"); } catch { } + if (inventory) renderInventory(inventory); + void refresh(true); + }); + $("file-log-aggregate-form")?.addEventListener("submit", event => { event.preventDefault(); aggregateField = $("file-log-aggregate-field").value.trim().toLowerCase(); if (!aggregateField) { renderAggregation(null); return; } void refresh(true); }); $("file-logs-prev").addEventListener("click", () => { offset = Math.max(0, offset - currentLimit()); void refresh(); }); $("file-logs-next").addEventListener("click", () => { if (offset + currentLimit() < lastTotal) { offset += currentLimit(); void refresh(); } }); syncColumnPicker(); syncHeaders(); diff --git a/src/admin/web/index.html b/src/admin/web/index.html index 2fb3297..e4529c4 100644 --- a/src/admin/web/index.html +++ b/src/admin/web/index.html @@ -7,9 +7,9 @@ RustProxy / Control room - + - + @@ -1675,6 +1675,7 @@ error:"connection reset" + @@ -1697,6 +1698,25 @@ +
+ + +
+ +

@@ -1717,6 +1737,7 @@
+