gru/RustProxy
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v0.33.8

Browse Source
This commit is contained in:
Mateusz Gruszczyński committed 2026-10-17 11:57:16 +02:00
1 parent 6b937aaf85
commit 87c8393d86
79 files changed
+6727 -297
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

AGENTS.md
+1 -1
View File
@@ -2,7 +2,7 @@
Read README.md, ROADMAP.md and docs/TEST_REPORT.md before making changes.
The project name is RustProxy; the Cargo package/binary remains `rust-proxy` for
compatibility. Current source version: 0.33.7.
compatibility. Current source version: 0.33.8.
## Delivery
CHANGELOG.md
+27
View File
@@ -3,6 +3,33 @@
> Operator-facing release history. Historical engineering notes are kept in
> [`docs/CHANGELOG_TECHNICAL_ARCHIVE.md`](docs/CHANGELOG_TECHNICAL_ARCHIVE.md).
## 0.33.8
- Skip cache-only policy maps, request snapshots and ticket work when caching
is ineligible. Preserve independently enabled transformations, header policy,
security decisions, telemetry and runtime configuration reloads.
- Lazily allocate idle timers while preserving the original construction
deadline, progress resets, trailers, cancellation and permit lifetime.
- Retain only request headers consumed by compression, and use direct hex
formatting for request IDs while preserving per-request entropy and hashing.
- Shorten live-history mutex holds by cloning entries and releasing evicted
entries outside the lock; preserve retention, drop counters, live streaming
and external sink behavior.
- Reduce filesystem job context copies and root-descriptor duplication. Use
adaptive file reads up to 64 KiB, preserving pinned-root and symlink checks.
- Report validated local file end-of-stream on the final frame. Fix successful
HTTP/1 file responses reported as cancelled and full/empty file cache fills
that were never committed. Reject short or oversized local streams; keep
upstream end-of-stream and trailer handling unchanged.
- Correct inherited Host-port validation, long-form Expires parsing, retained
cache inventory, diagnostic-header spoof protection, escaped-JSON log
redaction and certificate-token authorization ordering.
- Repair obsolete test fixtures/assertions without relaxing existing behavior;
add actual process/transport regression, benchmark, profiling and soak runners
with exact configurations, checksums, raw samples and resource observations.
- Keep the existing LB panel and persistence schema. Performance outcomes are
documented separately in `docs/PERFORMANCE_0.33.8.md` and the evidence archive.
## 0.33.7
- Apply the same theme-aware subdued `--muted` color used on small metric
Cargo.lock
Generated
+1 -1
View File
@@ -1881,7 +1881,7 @@ dependencies = [
[[package]]
name = "rust-proxy"
version = "0.33.7"
version = "0.33.8"
dependencies = [
"anyhow",
"arc-swap",
Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "rust-proxy"
version = "0.33.7"
version = "0.33.8"
edition = "2021"
rust-version = "1.99"
description = "RustProxy - independent reverse proxy with local administration"
README.md
+35 -1
View File
@@ -1,4 +1,38 @@
# RustProxy 0.33.7
# RustProxy 0.33.8
## 0.33.8: request-path performance and compatibility verification
This release reduces unnecessary cache-policy work and header copies when
caching is disabled, creates body-idle timers only when a read actually waits,
and formats request IDs without a generic per-byte formatter. Request IDs retain
fresh operating-system entropy and the same SHA-256 construction.
Live-log history moves entry cloning and eviction cleanup outside its mutex
critical section. Its bounded retention, best-effort insertion, drop counters,
live stream and external sinks retain their existing contracts. Saturation tests
record history losses explicitly alongside throughput and resource use.
The Web Server uses a narrow filesystem context and borrows its pinned root
descriptor for nonempty paths. Adaptive reads are bounded at 64 KiB. A local
file-body wrapper reports completion with the last expected byte, fixing HTTP/1
cache publication and complete-response accounting for ordinary and empty files.
Range, conditional, HEAD, sidecar, cancellation and short-file cases are covered
by additional tests; arbitrary upstream bodies retain their trailer semantics.
Compatibility testing also fixes existing explicit-Host-port validation,
long-form `.htaccess` Expires parsing, expired cache inventory visibility,
reserved backend-diagnostic header protection, escaped-JSON secret redaction,
and certificate deployment authorization ordering. Obsolete tests are updated
to exercise the existing documented behavior, including scheme-isolated cache
keys, purge generations, private administration responses and bundled error pages.
The LB panel, database schema, default TLS/caching policies and bounded pool
recycling retain their existing behavior. Live process regression, verified TLS,
H2/H3/QUIC and WebSocket runners, repeated nginx comparisons, resource sampling,
syscall profiling and a mixed-traffic soak are included in
[`scripts/performance/`](scripts/performance/README.md). See the
[verification contract](docs/TEST_REPORT.md) and the separately generated
[performance results](docs/PERFORMANCE_0.33.8.md) for measured outcomes and limits.
## 0.33.7: named dashboard presets, panel components and chart typography
ROADMAP.md
+20
View File
@@ -1,5 +1,25 @@
# RustProxy roadmap
## 0.33.8 - Performance with retained functionality
- [x] Reduce cache-disabled request-policy work, header copies, timer allocation
and request-ID formatting overhead without changing security or telemetry.
- [x] Move live-history cloning and eviction cleanup outside the mutex while
retaining bounded history, explicit loss counters, live streams and sinks.
- [x] Narrow filesystem job context, borrow pinned root descriptors and bound
adaptive static-file reads at 64 KiB; preserve range and symlink behavior.
- [x] Correct local HTTP/1 file completion for cache publication and accounting,
with explicit empty, short-read, cancellation and multipart coverage.
- [x] Reproduce baseline test failures and distinguish real inherited defects
from obsolete fixtures; correct both without weakening functional contracts.
- [x] Add repeatable live HTTP, verified TLS/H2/H3/WebSocket, nginx comparison,
resource sampling, syscall profiling and mixed-traffic soak runners. Measured
outcomes and limits are recorded in `docs/PERFORMANCE_0.33.8.md` and raw evidence.
- [ ] Complete deployment-specific 24-hour soak and remote-network testing with
actual PHP-FPM, Tinyauth and DNS infrastructure before production rollout.
- [ ] Evaluate further proxy pool/queue costs and a secure static-file sendfile
path from measured profiles; preserve TLS/H2/H3 and all observability semantics.
## 0.33.7 - named dashboard presets, focused HTML extraction and chart typography
- [x] Apply uniform 12px sentence-case labels to all Overview widgets,
SHA256SUMS
+77 -49
View File
@@ -1,13 +1,13 @@
61bf40a0572889dc8b0dda9eae4e93e44b0de9364b3c798106c85700ffb37c23 .dockerignore
4140a2956264b44f259e6a37fc00ba02097bbe68b09296e04d3d243c1287129b .env.example
9e6356475bf587091bab72246ec9e325b498f7e48a6220e9a18baa50d0a385a6 .gitignore
c793a8be7f4b76350e04ab36dbeec3792bb2dcf33c87a86f76fa2f330e26dd9f AGENTS.md
91edfded868ff72e97fbb34d08e1be2df5d5b8fa6582b534a8e4d109cd312acd CHANGELOG.md
d5dc2b778af98f52ad52a7515e2d1b4ba8c2e01a04cc6e02564377075b01c501 Cargo.lock
7fd5dd1ccae8fac77bbbea4bbf960f5cf762f712ff17a2e629a5dd8629eb7ff9 Cargo.toml
293fdbc2fc7318a9f544543c2ad518061ba1377edab1c73e93bd3c881d916195 AGENTS.md
38124997eb546aa2fb54c1b7801a127994a7911c469ae93b04b96e9816dec842 CHANGELOG.md
fcbd81ae693bf052a67a8815e227221a236e0486669e2b4668eece1f4e73bee7 Cargo.lock
8e72d66bf61579ceb5237d9195a4c09baf936abc9c9b541f17454c7946acdb58 Cargo.toml
25c7f3c8162cbf5cc156439774118a81dae9fdc52b611ec7e40c230fc99a772c Dockerfile
30708454b96c0dd57c9c621050ace58b92dbdbd85591c7b8199ebb54937dc372 README.md
e796ab6e379b3d71cf5d77ca3eb30bc6a320080af0c9e35f07c82737e09cbc2e ROADMAP.md
bcf49d2ed52433772e249d9d991fdaf7cc82636da4de30f098193c08c4b0d7b8 README.md
61870c209a50f1c683d221b3a11b18507b084c3b602111ef62f8bd90f285f51a ROADMAP.md
da8311e3db945290666324a2f24bf4521b365272d7ac3f862ded1be993eea646 UPDATE_CHECK.toml
f645f218dd062bf6cb54a8ee634290562cad364a4cccf09f16e3c8303582af8c build.rs
ea8d091843f86f4c66a5c76b20268b2feb397070851b6e05c5bc7d7a8d9157eb deploy/README.md
@@ -48,8 +48,9 @@ d8829cc68d982c933d04207cf7bc78cc918044c72e024e7ecad5220953bb3a79 docs/MEMORY_ST
e36247774051aa9c83c1a2fda91c1f7857991dcaa4fa90320ca5b492e31dbcf8 docs/MIGRATION.md
5f090a190000c6badab2a74e5d4968e70c34de43b59c1452f5b05503b5c5c18a docs/MOCK_AUTH.md
1f85fac0e06e432ef6f744f0b5f9c1b7a27a1e9e1ad30dc1a218eab96b1caffe docs/MOCK_SERVER.md
e7516646dfb107d6240370f4e7460b1189f2eb21f398740381d5d102847bbfe6 docs/NOT_FOUND_HOSTS.md
be53a58afbeddbd7deb4b5f6447f2dc74db0327839bd83407c20915df98dcfa0 docs/NOT_FOUND_HOSTS.md
89a1af6c9fe10c572b211f9632ce256166e67990b5dacacf2ae07265d99a5172 docs/NPM_MIGRATION.md
02126c23264b3afdf1e639c0334d621520e7538ff92d60eb6398c5fb10a23f2e docs/PERFORMANCE_0.33.8.md
e68d5a5cf0991de9308f78afb0264e02979f7463a5f395eb7a6e9ec2bd3a3d54 docs/PHP_FPM.md
bc068a176056f5ea4d3b637071e644ed58a2140494c233ef22aa66e9700e3299 docs/PHP_FPM_ISOLATION.md
ad6e3f003b820a4f45e30e39577d120a69d167c70910bda4891971759458c3dd docs/PROMETHEUS.md
@@ -64,7 +65,7 @@ ec7e32fd19e6ba6ed1732f1f3dd92ac0860609e8c0bbd3c1998840cdfb24bf54 docs/SECURITY.
007537ee971463072b2482dfa4fdcf1955f837ced0b711d4eef3b6267d16d4f6 docs/SECURITY_AUDIT_0.31.0.md
4dcf3243a13d9ae1827b5cefdbb9e27bd6310321c50d4d5934527d44437df6ef docs/SETTINGS.md
8fc112cb45a2f35ef5b0dd4371aa6f43aca4df85db62c6ae02e56d74a3b702ae docs/STREAMS.md
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 docs/TEST_REPORT.md
95148bfa1d2f8ff52b591c9acefc6c8992dadc4ebda43a195101d9b31d67a576 docs/TEST_REPORT.md
cfb452d8496839e832b5fbacdd4e52525b0ecc3112fbb62340c1e580bd5c1717 docs/TODO.md
751c9b13b47d0959a8575d6e82bdecba988b950a374ba0b084fbba08b4b65ebe docs/TRANSFORM.md
fe3ec6ca16ff933239c12dacf39dd7b45717c0551a8b9f6ce601bf78136c9e37 docs/TRANSPORT.md
@@ -124,6 +125,24 @@ a641bd35400c56f3b68b0a92e97b602d06090f03972ca8f01fa744e90dd97403 scripts/deploy
2cc658faaa610b21b48942c59b955525539ad4503fd60f2a49c7e626a22faaff scripts/migration/migrate_npm.py
a459d95bbb0076f33ade8b273b813308167f9d8d45242b309a2d18320ca0093c scripts/migration/migrate_npm_api.py
31ccad3ec3c880119f3b7002f0c6652f8b43634edad8ab3763cb0e4497d0bb1f scripts/migration/migrate_npm_core.py
afea7479fd10b7d723f147f6605283174ef81ff5e482f65819db4f01ffb02fe5 scripts/performance/README.md
0380406a059ee9ec27473f8a46435f638f73a6dc729382ae1561c08dbf7ccb53 scripts/performance/benchmark.py
9ac220163256e08513597c397c016a48345b114d7876b439259cc310b36c83fc scripts/performance/checks_cache.py
f09125d081d0eed9a44fdca75de14e6bbf09dcb00fe1f027ba2270f89e3ff192 scripts/performance/checks_policy.py
96e01db050987af431a4662fce34752842bb2575237fd3239bcfc10ae55def6b scripts/performance/checks_proxy.py
c1d8c184811ae8fb1960d1ada08d8a4a95405c3b83dd18e1a27d711b596140be scripts/performance/checks_static.py
1044eb8d225e2fa117d5435697abdf0a014aa2f9063d4b41566e74ef47d50a34 scripts/performance/common.py
0f1fa62ee3429509dc21fe1cbfa7159e38d260ada98bfc6376e8078c9840ef48 scripts/performance/concurrency_regression.py
044da66a00a04f558a6b6e6157db0a7656dd5ee22adf0b0a47482c7f951380e9 scripts/performance/lb_regression.py
1b601d68d15909d786ac63890731f0592bd9057a68993a1d90bc70d6618e6f82 scripts/performance/peer.py
830a360609e2b8ebbb1ef6f3d89e3e6151e8fefb36c3673d20df68ff8932eabb scripts/performance/profile_syscalls.py
ceb26650fc8b63d0107eb9100ccd572e5c70f7a0ce5cc533a94083e20485095d scripts/performance/regression.py
d6480962a23f55d19a54aba27648ca1b28b50f03afe2decbfa272efd142df756 scripts/performance/resources.py
f85dd0c29d5a96ad59a71e035ac4303c7b6a70a8d61280a15cdb966b32a218c7 scripts/performance/soak.py
fe6ff5bfd76ac7a2752a8a2e1e45c63e549c4a61c753008671e478df3ebff304 scripts/performance/telemetry.py
7c1ba46c623e1ade47b872fcdf4526a48ca85c56515dffeac626188b0357d23c scripts/performance/transport_clients.py
bd19431c89d7dcfe6743dffff9c4b16260a976d1936bdc4c11d4fcf6b3a7dda6 scripts/performance/transport_regression.py
1d747278da3fca093ed91c96d4a9480d0f711d0773d673906dbb56081635603c scripts/performance/wrk_report.lua
b5b3135449f6be4e97088b76ef7cf2428e2de3c7944c9cbbcb1edd2016c0fd4a scripts/preflight.py
3d6a0503afc25beab79ddc86864bcbb15248ded7368d33aa64321b8cdff59933 scripts/resolve_dependencies_0312.sh
20124173452b4514bad8850c1637569a9a191882478451e549bd16b432605fc1 scripts/runtime/acme_provider.py
@@ -153,6 +172,7 @@ b0136544af717b75df2cb107900ea65c3f945fdd0040369dbb93919008fb954c scripts/tests/
12f2b18b0ed2171c49eb8cde2c2b88fc627054c2f574c04bd9847821b00cd4c1 scripts/tests/system_ram_meter_0332_browser_test.py
046b4282550f5cb4e53906665c91c7442be70fc68fd0743466fd832582f7c979 scripts/tests/test_stream_tcp.py
f0b48ec0cc4411755cc0b4128eebce710375afa99344cb76198cf31074f8eb63 scripts/tests/test_stream_udp.py
6fa1667a844b7bc2adaa78738339b58ac6c94a6d6e7be34573433dcd323ae809 scripts/tests/upgrade_performance_test.py
f0a2e3daa0a49aac54723cb5a281360e49a97c076281372c5cca2a85e096c02c scripts/tests/vhost_backend_popup_0337_browser_test.py
23cb1f790cd5cee1b76ecd85f6a2a44ef2e16b0840bf06beaa2ae13048063b2f scripts/tests/web_observability_0330_browser_test.py
3babdd0fd6e2998755ae88615112f6ea056f5c808245abb6de60fbda555594d1 scripts/tests/web_observability_0330_source_test.py
@@ -167,14 +187,14 @@ a22e5ea6381a1eacafe650be0f5cd6e653773e31f6491a562fe0459a309fb7cc src/acl/tests.
587097620b2f25ce7c6f59f907b47660184b095e3040c7615e74509a82e7cdd8 src/acme/jobs.rs
3f158f46e80b403b658b4e9b1e3710e38c70e47b2d476b8223d9c394ffeeba95 src/acme/model.rs
c5334de6c732d76b2bea7bfc17b3ac305a250bbaad31a5420d6a7d7abb200449 src/acme/providers.rs
b9e57db4bd80ef37564dfc533a96897d8a494671d53acea9fb4bf28d23ba370d src/acme/providers/tests.rs
d1dcb7bdfb7cfdc9c3b88cfb6908201c58532bc888744a13f45853079f214659 src/acme/providers/tests.rs
29ff0647417b0f1d50a4cc075ab51afecff9ddf47db3eb7948d51db5a38f33f3 src/acme/runtime_bootstrap.rs
87f64198770ae026c03a242fa542db2590ca734b02e756f1e5c28d66236f753a src/acme/runtime_env.rs
f8075a5ef9ae5651d72a33ac0f0fd5b44ad0ae30f6e2d2318412177e0ed837f5 src/acme/runtime_versions.rs
e38331a1b186b49fe9d04051de52b603e1fa028def37e4e2f52e386b4d7b29ac src/acme/tests.rs
b6f38f3fd75d6685bbc882ed1077a38bf495cfcf01c397f69ed9b98be68b446f src/admin/access.rs
b4a2a068acfa1143b0032b9cd072c19cf3fdda2580f0b932d11d6368cd2ab3ad src/admin/certificate_api.rs
019f4aa33e3596bc9c11ef8cf5703bcfffe46c15288c0eba83c8dd9e38055fb2 src/admin/certificates.rs
51f46a2cb3167c4e3a784dfbb59b8fe0e9d5d3d51365b26699498e31d0afb6a9 src/admin/certificates.rs
b5f6d49f1add583cfc8cd4aa36ffe4db56dac4de3b151fe8239fe42f507019cc src/admin/dashboard_presets.rs
5e150d4ef0413a2f92db5b6631669933b84d4b4796633d01476c4b39188cd05c src/admin/documentation.rs
958546c43c05bf4bed8121b24db3b3096f9243f3c56a71e1762592b755684013 src/admin/documentation/documents.rs
@@ -201,7 +221,7 @@ a40adfbaa908da3ea726d7aa3d45394a70a6a55b6fc0063270ec4d2a2f2026a3 src/admin/sess
4b8a0483ede651d08489a4d6e5bb616f06423094bbbefcee70b8755d65c04398 src/admin/status_live.rs
c97881b17b7a42511868b0e48e2f25aab6fb5b613d538f5e1f1967508454e486 src/admin/status_system.rs
cadd7dca29b9aad8c77c7bc23dd249ea37a15adfa4d636a8ef00273a91ce35dc src/admin/tail.rs
3d9dcf96d8f05f1f03ad41b6f53d2b743ce6f9fadd608745d04fced0efabb9af src/admin/tests.rs
ccea3b72782aa9d36e41b89f40a755289d285692155c27cc81a0968f8780fe2a src/admin/tests.rs
48fcd818c522fe80f2c7b2e218f87de6a7772ffd0a24e93502780b03ec78e335 src/admin/web/acme.js
c0d08c5b8fc13dec9bf53fd842d224b9df3874fc3bdf31413e3ab182a8cec2aa src/admin/web/app/backend-dashboard.js
a085ee336e6669229f3ad2f74536f19bd088412ca79bcaff30dff293f1e1baf9 src/admin/web/app/chart-scales.js
@@ -279,7 +299,7 @@ c96a87e82ff0c1c9ce3d0cd5135903bb53af78d1c4fb6ebfe5fea503bf5644ea src/auth/store
76aa2afe9e7302be39b3512f30d0448d9134300f8ae930d64ada40cf3ad4a8a3 src/backup/archive.rs
732cd4f9ed38ae69f8bbae61b1db8baaa6f86b07ca21c41f24a206a30f0a96fc src/backup/ownership.rs
076350506215faec3cfe7d9de4e60baf29672cc22a41eabe601dcc510c257799 src/backup/snapshot.rs
a00571b409d305af86ab2a57a25434b5ab58706772df440d017fc1fec097f7b4 src/backup/tests.rs
5c19490d834d9971166b6b784a75b081bb95fd50c8c634ef3c87924dfffd4a19 src/backup/tests.rs
f5afae218f3a6f91bc2f1f5a90094630ce5b7ca574de61807b5a317b13937ec2 src/backup/validation.rs
426a946b2c9e244f09f6cbc5f9664b8ad0737529a9bb5a3d549d27196a1f8043 src/balance.rs
d3257fa5222dd4f70a82400f5060d3cbd33c13862a9320931361fe0e1c5bd452 src/balance/health.rs
@@ -297,7 +317,7 @@ e022003be144598100389cf38d76f509b5dbcc035d5bae556bb9b94cc8bd30ba src/cache/cold
9fc18368e2ce278ef0ad2b73ff016aa24936c2186be371ca4010a1f6f63bb0ed src/cache/cold/startup.rs
ea319b6e200b0c6ca93da5e2af7a3f86e3667cef918716f54f82b8012111b9bd src/cache/cold/worker.rs
d51fa4f320be56c572f767617428c3f3838543cb0eeb7d1f8ae791e1a1ce9a71 src/cache/collapse.rs
27a35689fef639f1aea00d9f5a86adc981c0c304dc3e28e6da40494dffbb6bfa src/cache/dashboard.rs
284f1cc56113fbcc3b1e617103d75a7b70fd7d43e536d6017c15912154921424 src/cache/dashboard.rs
780c2654c67dbe176d8310678e6deec8022bcebfd85b95df7262069ed41f57b1 src/cache/fill.rs
675d18afb53f25c56cba49bbb431fa39ecd9d7bdd192c7b39b0698225de7a2a2 src/cache/invalidation.rs
c1357a13658e1ed5158084d3a41ba9ea906512774a5863ce9c56cb508d3fa6c8 src/cache/invalidation/ban.rs
@@ -306,7 +326,7 @@ b74945c7460b76ed7afcb02a45ede09d76f08e0703c47920b9b9d794d4b74980 src/cache/life
bd831807ca4720b3a41a6ea1d9a6c0ed2d935225e2dacbd9313be064f2e4a20e src/cache/lookup.rs
88f6210c03d7a8e7a35be1f123c9c15675af6e6a599c50db927d99866851b634 src/cache/lookup/hot.rs
1779abcee77600c13050a2882f896a5b35625b1ae53737c2554179492ce0e40d src/cache/lookup/insert.rs
55a39fbcb8c034a98eaad44d9efc092902cd591ef167b8d8e14a56fd1a9f1794 src/cache/lookup/tiered.rs
107a589974b9689009a9bbf088818dfaec71cd1aab8676bbc6d0118359a54a31 src/cache/lookup/tiered.rs
58c03fde09e5eab40968679793b364f4f1cf5fd832bd79ebcdc92bb23871ad82 src/cache/mod.rs
0c8b0d550e94d4474dcdf991aa0340228dc5f2f904f39350e6e666a16f16fe87 src/cache/model.rs
ba94226addfd6cdab67d666dc15b8412c4d117111e44639e503e89a95175d0a2 src/cache/model/cache.rs
@@ -321,23 +341,23 @@ e26dce02df9c043329ca164fed215d271781bce8fb03240f528331bf3df6b9d7 src/cache/poli
d426d341a7678f8ec70406d86052c97d0f49804bbcd8402951103d61a735b9da src/cache/policy/response.rs
7e2bcfbb2b57732cdc704d9df7a9298fb80e735e37984431bc936630e41e7f8a src/cache/range.rs
7ad96b3baae36ec49c90d03b380e749944fb3d54abb045c90db3a35f6b89b1c1 src/cache/tests.rs
981c33d8321c0d5482a00dc3cf0911e95fa69a29c68a2f5b6cd4cff27cec1cbd src/cache/tests/cold_tier_tests.rs
22cb9bf20a0d25bee30c1e2213dfe8278ad7d0d5d294275d47d4a6ff9ac646ab src/cache/tests/dashboard_tests.rs
a1c750a48b4562d88d904b45dee5c783d4bf0ae1a62e67719df6bb2a7374465b src/cache/tests/cold_tier_tests.rs
3842cdcf7c3414b9c30f5fbed927b93e835eec3ef96d4c6ee44250038b834276 src/cache/tests/dashboard_tests.rs
d7a02eb7bad37e6a5524600aabb5e60ccf50e26c7a487e2806891c9d8acd48f3 src/cache/tests/http_invalidation_tests.rs
1db70ea5f3cc1f1543504a68498ba23cafce5f9bb48c4616a4bf70d9a536790e src/cache/tests/scope_tests.rs
9aa6bfdb88b5ea4ef0ecd72e8e3c6e82b5ab9b37051ad300b6a5dc12f8579ddd src/cache/tests/tests.rs
444d9a364339248bdb86f2285de6d7e598b880502e25a70be0ee877ff3e52ea7 src/cache/tests/v022_cache_tests.rs
ddbcf27a667d779065d7a654d8094910e92958d2b8862a2ec2a85ca41b7f21e5 src/cache/tests/tests.rs
af28534625859ff66011e10c54cff8f82c2cfa1c173c4d99b89c27c1437f0659 src/cache/tests/v022_cache_tests.rs
a8a0ce446eb53ffdb8f8d97efa3e1344410dc3798b25f62b8817e8e160d28940 src/cache/tickets.rs
28f904cf4827294f2f759e6d3624608ed90059b64e42864e55dc9213824eb655 src/compression.rs
e2f58279499b7c3b47c65d3a7814bb5c7cd6506544d939875fbb0f91fa9aa59d src/compression/tests.rs
e0146bf5d2e2e33aedcccd9205e5bd457559c9d77dc7ce266f8c165d1b867f0f src/compression.rs
4bdbde197d135fee8be20e93427eba8ca7b60b801717ed7ca734319bedb6ef05 src/compression/tests.rs
7bc8285cb2012506fd1a38a71dffa561f885f4ab65a9b2a7c8344319e3cd1cf6 src/config.rs
34be288c858a037c7174f6c4dee23178b01b565aa9d65f0951f351e1685847df src/config/backend.rs
beb831a208c4e5f7ebe8327686b1e0edc66608c7032f895cb4fc3366b189aae0 src/config/compile.rs
490d05d84b2fe349ddde54eae44880993fab70811652484dd9ba38b5069d603b src/config/internal_routing.rs
a19fc675720c7c2dbf4bba706e1b1c30c83c6df1da3f29e3cbb3b82442e7246a src/config/model.rs
f66c2388f31b19142c12469df23d43451e0f7c48b158d4996920282c06d32076 src/config/routing.rs
818545ad36af3ae8d56976f318349dd1f5264ef5665fbf94681a81b79003f8cb src/config/snapshot.rs
76cbb4f1339de9f13d441561a3c4ee6bc5ff06e90ff0f019b7f0239c74f9542d src/config/tests.rs
b25e212a493c6d9f6cd1d1136935182504163c7b8d7e776364389451ca2247aa src/config/snapshot.rs
f5431467dec55540f1f881541bf13d9402753680c7f83e3deb9d0e1fa747313e src/config/tests.rs
23541660151d9005555a07db6d491455d61b185f0cc70c3ec3a0bee2aef08dc3 src/db.rs
30f0b7269224a64e3f1ec9e6fbe9e27ddb1d308dc84cc9882e50545cf612695d src/db/certificate_api.rs
20ba52d3755c359d670968efa53b0604ff7cbed63a416f9f765b50a1f45074ac src/db/dashboard_presets.rs
@@ -359,8 +379,8 @@ d3dc58628683472ef4d205fa5c42f0b7ef83cf90846146d96941cd3b36a40766 src/geoip/test
70091bdf6f38f03b6df8c8c4407c300677e57f293130122503f936500963d37c src/geoip/update.rs
c1b49bef28d87712e4c46138e70b52c4bcfb1a00196579031fac043d0b4aaa7b src/geoip/update_http.rs
9a912dad0b8d1e10410fd95d0da3684656092ef85cf8b23eda4ffe7fb92921d8 src/geoip/update_tests.rs
13269be4ad8d3e4c91893753f654c3022e6f86d171c5d1ff21eda2d9185ec61f src/headers.rs
fbb6859780309f8cca1aeaba43fa67d13a1edf6f5a60c5c982d2067b71fb0846 src/headers/tests.rs
f6ebafafa82391f8cb3b5a09331f3762b6ca5a9fb64bafb5772cde3ab814dfcf src/headers.rs
e4f07bfb7eb75b6897ef812870caff158679a3046d64582428abe8c6df490dbe src/headers/tests.rs
9e21edc67a14ef8a2ce3dda0ed18784d0fd0be305973e5d216058b7ac802c6cb src/http3.rs
eb57117938a447d9a6c2ec1806ebd9fecc640a68c88950a8b35fa9bad14d0dc6 src/http3/body.rs
ed25da264a2d25b40c4fcddc97c626a3a43d4f3a9be6c6d31af58b9e4925c198 src/http3/diagnostics.rs
@@ -369,11 +389,11 @@ f2a8c2a98f4ccfb708f9d75a48ec7b351232be9b8a4364b7e63ba14d28eead7c src/http3/sett
a31db7eac12b578b1eb5a74fae98bbe36312da95b7d7173135dbc41fc454da21 src/http3/tests.rs
68fb5ab6a889995a66ad12cb6acf88c5d04f0aca572e2ae15f1a9cd171a5f24b src/lib.rs
2551189a3e5701371026c9abd2c5d44d37495f28fb67e220180421bfd315e3dd src/logging.rs
3849a09a7d7402dc4d867b646cda153c574327ddd3c707578ddb4dc6f95b1563 src/logging/error_runtime.rs
a16613114273c54b6d2a16a94fa8748550e91468585469642622bd2c86087baa src/logging/format.rs
18c8d74c21a47f234be93551d7264b1f8f8c00d1bbfbe555d91618c58cea0740 src/logging/error_runtime.rs
87948a511b680e1c274eb46d13b938b45792e63bd6cff1621afe3477708ad2a8 src/logging/format.rs
f5815d778cc1785723faed4abb15f9880f22b6524b5289fe1e7158905aac0986 src/logging/model.rs
42fef80af50735cd566c6801dad86ba81be98cf4a8ec7d45c2ea26f53f556a22 src/logging/runtime.rs
03c50bd14fecd421b0901a5f0af891dc277e35420e32068781e3a6d78c88c24b src/logging/tests.rs
584782215ea78a88526cdb5515031d8d5b87910875e26c0024b93a06680f6791 src/logging/tests.rs
3836ed14b6710e183f048ee4d7807a35366c27a27338057fe4a2811129b63302 src/main.rs
fc8e99eac83c083d65101f6402aacbcc8392d66508c31566a50de3ebcb12e7da src/main/admin_runtime.rs
42a9197475823f67c1ca762951ede5d7a2f10124bd6ed4d1a3365f4297be894c src/main/bootstrap.rs
@@ -386,7 +406,7 @@ e5c2e9189777725f4afa1c3c052e4f75047ec3f2f486630d990f5d3c194bcc96 src/main/tasks
49b11df65b4fb0b5b727f9f867436584680b45e3f4fbd53f74bc0cab8b9a88ec src/metrics/vhost.rs
a54b5b34ea65f29723b89b81bf7968a238e8247b9c9760f3430d3504cb1017c9 src/metrics/vhost/tests.rs
a4702c95c7e5077c2ccff86e2e3216e1da6ff9ee4383ae199a0aff2f78db325f src/not_found.rs
668840e7bd70590e2e13f59b6d5b9e43ca7c877f3d07c71907baea91b5bba6cb src/not_found/tests.rs
c915d30bbd7032d867e7916b4cd45f3bb7e7604d3f4558f4383c59394b4300c6 src/not_found/tests.rs
d66936cc15fd5d56566d40fa8cab93f80d94b05f10ac44478a30f63572a95e32 src/pages.rs
da40e8ece87245efc49a0c4e750526b1d19fd97565856bd5caf641fcc1dfe18d src/pages/tests.rs
0dba7b6a33866cc7bbe05509b01961e545892645d1dad4eb88ac8a4bec65e3bf src/prometheus.rs
@@ -397,29 +417,34 @@ d3c6b08088df0003144544936a55e2a3661aac85292821b594f00adfd19a2951 src/prometheus
028e1dd9f2ec2afaf699a6b21c2d4b0d96e76f11f5e4ed009c233dadf0f3be51 src/prometheus/traffic.rs
e03f0d6b3f35dae6b98c394837aa0d17e4d73a8306767b4c607868bc00c30286 src/prometheus/vhosts_backends.rs
fb2d8d45cd2e947f783eb981b37fa418c39fcb327a3af17a648fadea7f37f888 src/prometheus/web_php.rs
b5856fd38a957b2ca9c7e3606d84d13d5cfcf3a45b487b15ae1d2dc248cbda57 src/proxy/body.rs
16b9d4db3b20b2ab14c47ac14fcd18d3d8ca77c3ad9a1f37866ebfa60f379862 src/proxy/body/tests.rs
3d2c8462fd259c299ed26493cb2316d0b7349bd32fedc9b0c11e77b21d6faefc src/proxy/cache_helpers.rs
fb02fb31ab62ac47c998d418fb8fe2f1d810a451b68805f31d6ecc244ae20820 src/proxy/body.rs
b8fb54359a7c60e5a8bd8ca45a5a64641633c595f0deb84d6d906c31fbbc1312 src/proxy/body/tests.rs
c3dcd88f4b00a9dbcbb65f6fa70ee2ab98c2fba20e9a2dde3b8852306111836f src/proxy/cache_helpers.rs
aa49c9928bb91e3d5bae0d340d88396a0417b9edec6192d41e1403533319a897 src/proxy/cache_http_invalidation.rs
676c1b325fd8e55bcdd86451c7d1d8b8d347ca00e8da6b3e5b148a58c376a9b4 src/proxy/cache_plan.rs
3404b03096862de1050088f62cdd811ea1ab61d191f902501173107f81f23b9b src/proxy/cache_plan/tests.rs
771cc1a38927f96e593b144dbab66cd64a5dc0ed42d54b1615c359845b2b7223 src/proxy/error_fallback.rs
bf2e6bbd64fb45fceed54af0bff1c1e00da22b79c64dd8f5c23d2af0dbab0b7d src/proxy/forward/cache_lookup.rs
cfa941b60b5d167ec9d8e2566ea3342b588d1a9078a6ca711ad6062ddf00fec9 src/proxy/forward/response.rs
3b220cfa474325bfe7853d62daf2c71059d0dc582ac749fe61e2864094545d63 src/proxy/forward/route_cache.rs
cea9e47ecb31b0e4c0bbe0eb83e0e53b5e7bdcb9930107b67a8c34eef01a3f49 src/proxy/forward/success_finish.rs
902cbf8dc3c1ec8d25411bc7fe0400d301f3a7cc885afada594e4eaee3bedd08 src/proxy/forward/success_prepare.rs
ce782dce2f7fe65ef93d08f88eb8277162ad9b975c1391da9c45869511196621 src/proxy/forward/upstream.rs
b05c4f5c009c5e410f472ceaa999fa3a9c7fb60364d5b62de219f0924c9d524e src/proxy/forward/web_source.rs
b4be19f2dccc97270f3596d7e56dc60237b7d4d254cd9362a355812026d90295 src/proxy/forward/route_cache.rs
96c97928986ca89c2d6d415d5b107b9cc7228410bae93a2c08c24974324f4d8d src/proxy/forward/success_finish.rs
0a921474ef2b93eabdaf3888651cf29130eace5ccb1e0f5d6bc25e6aa69280be src/proxy/forward/success_prepare.rs
a064cf54ee62a374c1753f7a64940990a591888efc4516a07942c52c4e0842e2 src/proxy/forward/upstream.rs
1f36225516caefc756a49bbdf37b227a7ea97b72ab78df021c9234e5c3dd41fc src/proxy/forward/web_source.rs
a9c011061585c067f04f7885215d53e6c40474bdced15d2be5a5ba65190910a0 src/proxy/forward_entry.rs
9b25f5e3ea4443a63ee6f45d12e1957dd316fe8e85325fc00ea416faaab1025a src/proxy/h2_graceful.rs
c438066616527a3108f2ad53e3698ae9b2a80961714394cdd0a450c92f99d992 src/proxy/headers.rs
5cb7522ceb9a9f083a15596ac4898559295e38d1138c80099577211de8a868ec src/proxy/headers/tests.rs
8d17b97d90fd6a749ba8d37a8d28c94e6d7302b6c86905f073e2dc595506961a src/proxy/mod.rs
3f25429418941eb72c3847125340db74684ad316eb278ecf658953026fd41f4f src/proxy/mod.rs
55133ceace2e5262d9c54fa41bea099a27b817d1e09c1cd0e2493fd9cee4da38 src/proxy/redirect_helpers.rs
ad701033f005c9c8c4b160f9b9b97bcabc17e936d8ade273a111937c483a125a src/proxy/tests/authority_tests.rs
c61738672f210e97333c8c89f9b35f8db454cbc1c36955288dfa4cfff2bbbc04 src/proxy/request_id.rs
6ff45990f6beb914e89924bec2c41022f3240a14357cf66ec4f8ada414a49767 src/proxy/request_id/tests.rs
8ad3619359f3befa14ea71ba43c638fea8f4d8a859cd07e4ed505731b8e32e72 src/proxy/tests/authority_tests.rs
1aa9bbbfb2659facdaeaf185e11221b1c8cd5fe6e23c7c1dacd3aa55e3db4fcb src/proxy/tests/cache_fast_path_tests.rs
0f2eacadf1447db8aa1c1a78005aa262e0f747096bfe7e0a0e41a2cd31a0d17f src/proxy/tests/h3_navigation_tests.rs
d134b0eb617f82f9971b3f3461379ea37c57bd0079bf54748d0c434b2ef1b475 src/proxy/tests/http_invalidation_tests.rs
591b47fc40cbcb36d7c71998933dc873340dfcf2657e16c561ab64361c4dbab6 src/proxy/types.rs
1b58f39045e82b1a290435502f5bf8a52c208c283bfb0d13618e0a7d150bb6b1 src/proxy/upstream_helpers.rs
8b053027bbdbbebb624164d9c1b15b2b8f5a13b8a23a9e1477ef4e840ce0334f src/proxy/upstream_helpers.rs
2d9142e3ae95895b5271b4009ab48236a3a2074ce51179c021193b35777560e4 src/proxy/web_cache_revalidation.rs
d7b4b809b12ee89293df45e9a64b1c9981fb3f89deaab3762f4a3ad3992dd7b1 src/proxy/web_cache_revalidation/tests.rs
69f775f6746b4c6d729dc10961dbef54168846f4ce9cf7fe453ab5a2b879c9b8 src/proxy/websocket.rs
@@ -486,7 +511,7 @@ cf5195ab6bacccae0082da7b3f1bd0382b73fdeb3607f4ae569ca6daab9d10d0 src/transform/
fde2fa3aa94b30d8536afec1c11348cfbd922d0b5ff89e91d34266e8e3863e26 src/transform/model.rs
a4e1d2aa2913587c250f251356431097b76995aee6e8bf9e35c4978a95491de1 src/transform/svg.rs
0f53c64f442390ee20a59087a10f8f9ffb58053263e9aa848ddabfae1a6f3f51 src/transform/svg/tests.rs
8ed4baeaf79edfa2a84912cc00f1d13c564e1b997efe6888d2e4b5a16aaa8905 src/transform/tests.rs
d1f3f53fedae0d8ef69eeee457f7d30ed5a670a1ed4ced60683400ee46680260 src/transform/tests.rs
6146c59f36cf419623747d497b3de685da6c84964df43317bebdcb21deb9ee09 src/update_check.rs
c42d000d1d576721af3ffe930589388693001b8113554a7658f6dcbc665b5d1b src/update_check/tests.rs
92efcdaafb9f5a8514daf63d7fcde29be977d7bdcd90162eda0ce00d6967ffbd src/upstream.rs
@@ -509,26 +534,29 @@ d9178aa934ee1a7236cb04d864fa842faf64156e9103786e3103c92bc4e18fde src/web_server
86c744216d7132d8294a6e90ba9be5fd871646006313d1c62506d40c81db9b12 src/web_server/fastcgi.rs
200cba8cb3482e49db73d2c60cb60786175177c7d4be2a566cd281fec7ba7fca src/web_server/fastcgi/protocol.rs
cef44ccdf882d84892a64bdbfdc1ba01380fce5e5f86d2829b0f9613f738b964 src/web_server/fastcgi/response.rs
8186363315bd998919c6c783e6f7501528f257a5c0da77c5d60f423e8670db5e src/web_server/filesystem.rs
0aafd7f37c4fe784b047646e4ce5ddcb5fe821231a41ed0a9f519529385288a8 src/web_server/htaccess.rs
013830a79a78ce803ce776a9b333de406367751e0cea7e33c3f6a98e6600c243 src/web_server/htaccess/model.rs
e15361622683268c5606a39a9e5db3938e630999767898391226e94ad2e421e8 src/web_server/filesystem.rs
d4fec3682532b0f2d11a0aefcd024f69fb623a39922ddb6c528a2d2de265347d src/web_server/htaccess.rs
ca3cd376b313af1ce0ee65081b3327eb213f000d5a215aa4ee6b764de9702126 src/web_server/htaccess/model.rs
76de1d11df75f4a64556b2821fd0a1774d26c9f0ca0e5f2b5fcfad0722d43d47 src/web_server/htaccess/parse.rs
66260d159986d41eab0dfde2cb9f0713fc1cc209c2fb47e9023bd24fd37919e8 src/web_server/htaccess/rewrite.rs
8ca74d9cc4470045301f70eb4a9fbabd0dbfe74434429148d2230df9f996c06c src/web_server/htaccess/tests.rs
3341264f6e8041ac5a5d3c4b1601a69787386116019b5febc1a5e22bdb1186f9 src/web_server/metrics.rs
7b945065e636ca74147048aa53cc5b34028cced72a780fbaf0aa2499b00718fa src/web_server/mod.rs
67ab8bef567452f123c4b9b46a942b39c93f1f765c128edca7ee808ff1c5e304 src/web_server/request.rs
c5cb561b0f276e8082f904b9446fa5885f6b3c0608f25ea2ef425061e9f7f6b4 src/web_server/response.rs
cb461fb68c2740270edb9030a63e9d8c70c3ab241ba20d92f1a6e5b83915996e src/web_server/response/file.rs
d98c80663847d6d83073df074ebff65952755415574708abcc153bd8d3adfaa9 src/web_server/response.rs
c89e412584cf524cd1cd332ce555ff922a9e5549cd5d20b2e9a29393456be889 src/web_server/response/file.rs
ba793a140699956a9aef6bc79b0c731c481af8ab7ac3d816a28695743febcb30 src/web_server/response/file_body.rs
c2e54fd803a10f74490f183e8c030fd0e3e72f5de4e79c55bc393ab57d90ab9a src/web_server/response/file_body/tests.rs
20509f9302b79d3a7be76dda6edaf5711826d3ddf47d19ebe07cb289427ca328 src/web_server/response/headers.rs
9b58d6bb3ff1520060fc5aa86a3594f99192b1a8f8183255be1fb3ff35b4bb35 src/web_server/routing.rs
305c0109b4f869a99381a142853f9e2205f5c7c4f904b2dfa4643274bfc542af src/web_server/routing.rs
400e1c8aa2fe857d96036a64499fbc3de44c1e3a53811c5b25101fb843359792 src/web_server/security.rs
a082e1d32aeff7b64976c943251b0c6487409f191883bc822ac4a60bb4d33e7d src/web_server/serve.rs
6b3ade894f02fb68f3507e1ecc578f173050d97100d41261467faeb9fd10c31b src/web_server/tests.rs
54cf72a82e023fe46513b9cf3130adfef1cd96c40f790c533016b4825d542e06 src/web_server/serve.rs
bda82f531f61251772d3bb4bc67d98ca613aebd51b489b7cac1fd5086c653b1b src/web_server/tests.rs
f00743af2cf4a5952caef7a1dcf68e7817b5599a252a9973e1c22045517c0b0e tests/fixtures/argon2id.json
1725c10331745dc04a07675784d385260da7b14567b8dba3cbae5e891eb5dadf tests/http2.rs
347a4ff9e028a1e999ad129590c355b5734cf928dd81e5099aea0c8aca8d5908 tests/http3.rs
eecf680f001f2110dde4eceaccf2f922859a16c946a644f8854e8de4d3771f7c tests/policies.rs
728999110a6d8cacb62bc0399bc333fa23a66526ee23f563a3c93bf00e5fad19 tests/transport_config.rs
7a28cfdb43a3cecf90136cf8898c060c382af557ce4ff168a6672391a0e5c26b tests/transport_config.rs
eca68a07c2b1e45fdb8f512294f5a432db3ae5a6aec8fb200b47f155cd9a31d6 tools/README.md
358c1e94dd2b4a756430e5335747f0b437345b1713670a9972c5bcf78b837ccb tools/audit/admin_audit.py
7c30c280d0b0a8bf855eeffab583ca34dc161255f7ef6124148aa979c17c26d3 tools/dev/memory_load.py
docs/NOT_FOUND_HOSTS.md
+1 -1
View File
@@ -17,7 +17,7 @@ The response can be:
- an empty/status-only response,
- bounded custom HTML.
The status is selected rather than typed manually. Application-style pages expose only the bundled 401/403/404/429/503/504 templates. Placeholder, empty and custom-HTML modes expose the supported application error set: 400/401/403/404/408/413/429/500/502/503/504. HTTP/2 and HTTP/3 toggles are shown beside the listener controls because they apply to the selected frontend listeners. Host/listener ownership is
The status is selected rather than typed manually. Application-style pages use the bundled 400/401/403/404/408/410/413/429/500/502/503/504 templates. Placeholder, empty and custom-HTML modes expose the supported application error set: 400/401/403/404/408/413/429/500/502/503/504. HTTP/2 and HTTP/3 toggles are shown beside the listener controls because they apply to the selected frontend listeners. Host/listener ownership is
validated against vhosts and redirect hosts, so two active configuration objects
cannot claim the same frontend binding.
docs/PERFORMANCE_0.33.8.md
+353
View File
@@ -0,0 +1,353 @@
# RustProxy 0.33.8: performance and compatibility results
RustProxy 0.33.8 substantially improves local static-file throughput in this
experiment: **5.23 times the rebuilt original for 64 KiB and 10.13 times for
1 MiB**. Small proxy and WWW responses improve by 15.71% and 10.41%, respectively.
Results are mixed for larger proxy responses: 64 KiB improves by 2.41%, while
1 MiB decreases by 2.77%. This is a first optimization package, with remaining
distance to nginx and visible memory and observability costs. It is not a
universal speedup, nginx-equivalence or production-capacity claim. [B1, B2]
The changes preserve the LB panel, schema version 10, existing feature controls,
TLS/cache defaults and bounded pool recycling. Actual process tests verify the
release separately from throughput measurements. The complete package is the
unit of attribution; no experiment isolates the contribution of each edit. [F1,
F2, S1]
## Executable and evidence identity
The tested final executable is `rust-proxy 0.33.8`, SHA256:
```text
24cc77a084ee1db4cd007f86c4b18b319bdc2e3f19afae7804739abff5e62a12
```
The rebuilt original 0.33.7 uses the same Rust 1.99.1 compiler, locked dependency
versions and release profile. Its SHA256 is
`85a91318b41a405d1322cbfb84979e05490a7c2ad429ee7cb0c0836acca63899`.
The independently supplied 0.33.7 executable has SHA256
`c68d63326ddf1e25911a583d994a213132dfb1b094425fc6cf358470299f0a92`.
Differences from the supplied executable therefore remain a separate comparison;
they cannot be assigned solely to source edits or solely to compiler provenance.
[B1, F1]
The primary run finished at `2026-10-17T09:01:41Z`. All **2,077** files in its
sealed manifest were independently rehashed. The SHA256 of that `SHA256SUMS`
file is `5036d1f27a53dd6434b4be5a9be99a6e300f10821cffcc15b79f4da06ff147ee`;
`runs.json` is
`ebd9cc06d52496f776deefc48615598a64f018a10be5811ac89401b06e821e5e`.
This report was created after the release build; raw results identify the tested
binary independently of subsequent documentation. Evidence identifiers below
refer to workspace-relative paths in the accompanying verification package. [B2]
## What changed, and what remains deferred
The supplied PERF-00..08 plan is broader than this release. Its boundaries are:
| Plan item | Implemented here | Remaining work |
|---|---|---|
| PERF-00: reproducible measurement | Real HTTP runners, interleaved comparison, resource/telemetry capture, exact payload checks, separate syscall tracing | Full concurrency sweep, remote generator/origin, allocator/cycle profiles and per-edit experiments |
| PERF-01: keep-alive tuning | Existing controls and default limits retained | Controlled idle-pool/recycle sweep; no claim from the old configuration-only experiment |
| PERF-02: compiled feature plan | Request-local cache eligibility plan, credited below | Broader immutable `CompiledRequestPlan` for all optional features |
| PERF-03: cache OFF path | Skip inapplicable cache-policy/key/header/revalidation preparation; retain bypass reasons and cache ON semantics | Further changes only with profiles and equivalent regression coverage |
| PERF-04: body/telemetry cost | Narrow compression-header copies, direct ID hex formatting, lazy idle timers, shorter history mutex section | Wider `Exchange`/analytics redesign and other locking changes |
| PERF-05: routing context | Existing validated snapshots retained | Shared validated route context and rewrite invalidation redesign |
| PERF-06: static I/O | Narrow filesystem job context, borrowed pinned root, reads capped at 64 KiB, precise local EOF | Buffer sweep, secure sendfile/io_uring and extra FD/content caches |
| PERF-07: pools/connections | Existing semantics tested with real peers | Pool/selection optimization; no LB UI additions |
| PERF-08: release gate | Listed laboratory checks and 30-minute soak completed; documented rollback | Memory plateau, deployment-specific 24-hour test and staged rollout |
`CachePlan` avoids force-cache/cookie matching and request-policy work when
ineligible, while retaining diagnostic reasons, origin conditionals, privacy,
range/transform interactions and publication tickets. WWW requests move headers
into local serving when cache conditionals do not need a retained copy.
Compression copies only relevant header values when needed. Request IDs retain
fresh OS randomness, SHA-256 and their 64-character lowercase representation.
Idle deadlines and cancellation semantics remain even when timer allocation is
deferred until a pending read. [S1, F1]
Filesystem jobs receive a narrow root context. Nonempty paths borrow the pinned
root descriptor for the first secure open, while root-directory operations keep
independent directory offsets. Local file bodies report EOF with their final
bounded DATA frame; short reads and cancellation do not publish complete cache
entries. Arbitrary upstream bodies are not declared complete from Content-Length
alone, preserving trailers and framing checks. [S1, F3]
History entry cloning and destruction of evicted entries occur outside its
mutex. The 4,096-entry history, chronological retention, `try_lock` admission,
drop counter, live broadcast and configured sinks retain their contract.
Compatibility repairs also cover explicit Host ports, long-form `.htaccess`
Expires, expired cache inventory, reserved diagnostic headers, escaped-JSON
secret redaction and certificate authorization ordering. These are included in
package-level results, without separately assigned RPS gains. [F1, F4, S1]
## Measurement method
The primary matrix contains **120 measured samples**: proxy/WWW, three object
sizes, four executables and five interleaved repetitions. Each run has 15 seconds
of measurement after three seconds of excluded warm-up, using 64 actual
HTTP/1.1 keep-alive connections. Payloads are 1,040, 65,536 and 1,048,576 bytes;
cache is OFF and requests use identity encoding. [B1]
Four frontend workers are pinned to CPUs 0–3, two origin workers to 4–5, and
four wrk threads share CPUs 6–8. All processes share one Linux host/kernel and
an eight-CPU cgroup quota. nginx 1.26.3 uses sendfile, access logging OFF,
upstream keep-alive 64 and `proxy_buffering off`. RustProxy retains its existing
history/analytics and request-policy work; nginx is a differently featured
reference. This is a warm local-file-cache experiment. [B1]
Before measurement, the actual ACME/Python startup must finish: authenticated
status must be terminal, with no active jobs/configured issuers and one stable
RustProxy process for five seconds. The harness validates process identities,
nginx worker counts and affinity throughout sampling. A discarded early attempt
had incomplete nginx descendant accounting and is retained only as diagnostic
evidence. Earlier syscall traces without this quiescence gate are also excluded.
[B1, S1]
Exact HTTP 200 and payload hashes are checked before and after every measured
run; aggregate transfer bytes receive a gross sanity check. wrk does not hash
every timed response and counts status errors above 399. All 120 runs passed
these checks, with **77,713,019 completed requests and zero reported errors**.
No completed sample was removed to improve the result. [B1, B2]
## Throughput and repeatability
The table reports median requests/second across all five runs.
| Scenario | Supplied 0.33.7 | Rebuilt 0.33.7 | Final 0.33.8 | nginx |
|---|---:|---:|---:|---:|
| Proxy, 1,040 B | 27,341.75 | 25,350.63 | 29,334.12 | 170,610.16 |
| Proxy, 64 KiB | 20,485.54 | 20,246.28 | 20,735.07 | 35,393.29 |
| Proxy, 1 MiB | 4,428.31 | 4,588.06 | 4,461.19 | 2,730.23 |
| WWW, 1,040 B | 22,535.57 | 22,558.35 | 24,906.20 | 351,605.13 |
| WWW, 64 KiB | 3,960.03 | 3,990.22 | 20,856.67 | 203,709.91 |
| WWW, 1 MiB | 272.40 | 279.80 | 2,833.71 | 12,611.49 |
Gains use `100 × (median final / median reference − 1)`. nginx attainment uses
`100 × median final / median nginx`. [B1, B2]
| Scenario | Gain vs rebuilt | Gain vs supplied | nginx attainment | Paired gain range vs rebuilt |
|---|---:|---:|---:|---:|
| Proxy, 1,040 B | +15.71% | +7.29% | 17.19% | +6.40% to +17.52% |
| Proxy, 64 KiB | +2.41% | +1.22% | 58.58% | −4.54% to +14.40% |
| Proxy, 1 MiB | **−2.77%** | +0.74% | 163.40% | −2.77% to +5.48% |
| WWW, 1,040 B | +10.41% | +10.52% | 7.08% | +1.73% to +23.63% |
| WWW, 64 KiB | +422.70% | +426.68% | 10.24% | +385.45% to +477.68% |
| WWW, 1 MiB | +912.75% | +940.29% | 22.47% | +818.88% to +991.33% |
The five final/rebuilt ratios, in scheduled repetition order, are:
| Scenario | Rep 1 | Rep 2 | Rep 3 | Rep 4 | Rep 5 |
|---|---:|---:|---:|---:|---:|
| Proxy, 1,040 B | 1.0679 | 1.0767 | 1.0640 | 1.1752 | 1.0747 |
| Proxy, 64 KiB | 1.0160 | 0.9546 | 1.0789 | 1.0707 | 1.1440 |
| Proxy, 1 MiB | 0.9723 | 0.9770 | 0.9868 | 1.0548 | 0.9883 |
| WWW, 1,040 B | 1.0173 | 1.0210 | 1.0893 | 1.1092 | 1.2363 |
| WWW, 64 KiB | 5.3555 | 4.8545 | 5.7768 | 5.3480 | 5.2270 |
| WWW, 1 MiB | 10.2897 | 10.5518 | 9.1888 | 10.9133 | 9.2866 |
Five pairs support the large WWW effect's direction and magnitude, but do not
provide a production confidence interval. Proxy 64 KiB has a small headline
improvement with one negative pair. Proxy 1 MiB is slower in four pairs, so the
modest regression remains explicit. For proxy-small, the paired median gain is
7.47%, different from the 15.71% ratio of variant medians; both statistics are
available. Its supplied comparison also contains one negative pair. Do not
average these scenario percentages into a universal speedup. [B2]
## Latency, CPU and resources
Values below are medians of per-run measurements; latency percentiles are not
pooled. Frontend CPU includes its threads and background work. [B2]
| Scenario | p95 ms, rebuilt → final | p99 ms, rebuilt → final | CPU µs/request, rebuilt → final |
|---|---:|---:|---:|
| Proxy, 1,040 B | 4.093 → 3.591 | 5.066 → 4.707 | 147.05 → 127.41 |
| Proxy, 64 KiB | 6.188 → 5.921 | 36.594 → 37.245 | 182.66 → 178.44 |
| Proxy, 1 MiB | 29.206 → 30.222 | 49.315 → 50.747 | 861.28 → 885.37 |
| WWW, 1,040 B | 4.956 → 4.432 | 6.038 → 5.432 | 155.68 → 140.05 |
| WWW, 64 KiB | 27.929 → 5.217 | 35.301 → 6.393 | 657.26 → 171.85 |
| WWW, 1 MiB | 291.222 → 31.234 | 318.346 → 35.295 | 8,388.17 → 1,219.50 |
WWW 64-KiB/1-MiB CPU cost per request falls 73.85%/85.46%, while p95 falls
81.32%/89.27%. Proxy 1-MiB CPU rises 2.80%, p95 rises 3.48% and p99 rises
2.90%. Proxy 64-KiB p99 also rises 1.78%; improvement is not uniform. [B2]
| Scenario | Sampled peak RSS MiB, rebuilt → final | Peak FDs, rebuilt → final |
|---|---:|---:|
| Proxy, 1,040 B | 56.965 → 59.137 | 162 → 162 |
| Proxy, 64 KiB | 68.543 → 69.105 | 160 → 157 |
| Proxy, 1 MiB | 81.359 → 83.684 | 153 → 151 |
| WWW, 1,040 B | 77.441 → 77.727 | 124 → 123 |
| WWW, 64 KiB | 79.043 → 102.156 | 145 → 124 |
| WWW, 1 MiB | 78.473 → 118.664 | 145 → 145 |
Large WWW responses cost **23.113 MiB and 40.191 MiB** additional sampled peak
RSS. Processes persist across scenarios, so this includes allocator/workload
history. The source-derived 7.5-MiB extra reader-capacity example is not total
measured memory cost. Candidate 1-MiB end RSS increases from 114.508 to 118.664
MiB across five repetitions; these short runs do not prove a plateau. End FDs
remain 17–19 for large WWW, without an accumulating signal here. Peaks are
sampled, not absolute bounds; nginx process RSS can double-count shared pages.
[B2, B3]
Only one cgroup throttle event occurs: 7,839 µs during nginx proxy-small; Rust
intervals record none. The nginx proxy-small origin consumes approximately
1.982 of its two CPUs, plausibly limiting that reference. nginx static 64-KiB
uses 2.573 of three generator CPUs. Rust proxy uses roughly 3.70–3.95 frontend
CPUs, with generator/origin aggregate use below their allocations. These bounds
and the shared kernel limit capacity interpretation. nginx static 64-KiB p95
is 34.361 ms despite p50 0.215 ms; its corrected closed-loop histogram is not
a generic nginx latency verdict. [B2, B3]
## Observability under saturation
History still uses best-effort admission. Median rejected-history entries as a
percentage of wrk completions are shown separately from HTTP failures. [B2]
| Scenario | Supplied | Rebuilt | Final | Final minus rebuilt, percentage points |
|---|---:|---:|---:|---:|
| Proxy, 1,040 B | 3.350984% | 3.325181% | 1.290544% | −2.034637 |
| Proxy, 64 KiB | 2.231502% | 2.203964% | 0.780042% | −1.423922 |
| Proxy, 1 MiB | 0.763138% | 0.765001% | 0.221602% | −0.543399 |
| WWW, 1,040 B | 3.908337% | 3.851348% | 1.504784% | −2.346564 |
| WWW, 64 KiB | 0.707405% | 0.700592% | 1.230427% | **+0.529835** |
| WWW, 1 MiB | 0.114600% | 0.120948% | 0.165351% | **+0.044403** |
For WWW 64 KiB, every final fraction exceeds its paired baseline at 5.23-times
throughput. For 1 MiB, four fractions increase at 10.13-times throughput. These
are saturation comparisons, not fixed-arrival-rate experiments. The increases
must remain visible; this release does not guarantee zero history rejection.
History rejection does not itself reject live broadcast or external sinks. [B2,
F1]
Analytics queue drops are zero in all Rust samples. Sink, error-file and
cold-cache queue drops are also zero, with those paths disabled in this primary
configuration; this does not prove their active throughput. Asynchronous
recorded analytics counts differ from wrk completions. Status/traffic snapshots
bracket load, postflight and possible in-flight work, making rate denominators
approximate. Raw counters, every repetition and aggregate weighted fractions are
retained; they are not presented as exact loss probabilities. [B1, B2]
## Functional verification and inherited defects
| Verification layer | Final result | Evidence |
|---|---|---|
| Rust suites | **706 passed**, zero failed, one optional ID timing microbenchmark ignored | F1 |
| Actual HTTP process regression | **152/152 cases passed** | F2 |
| Actual TLS/H2/H3/QUIC and H1 WebSocket tests | **111/111 assertions passed** | F3 |
| Two-origin LB behaviour | **17/17 cases passed** | F2 |
| Finite concurrency | **18 cases, 1,920 exact responses**, zero failures | F2 |
| Separate syscall experiment | **1,042 exact response checks** | P1 |
The suites protect cache OFF/ON, privacy, Vary/scheme separation, purge barriers,
range/conditional/HEAD, compression/transforms, ACL/SSRF policy, administration
session/CSRF, streaming uploads, cancellation, non-replayed writes and leases
through EOF. History tests check exact fields, eviction accounting and real
file/live delivery during forced contention. SQL and disposable stopped-source
upgrade checks pass; UI checks remain simulated API tests. PHP coverage uses a
deterministic FastCGI peer, not a deployed PHP-FPM application. [F1–F4]
Both original executables reproduced three live WWW failures:
`observability.static_complete_after_content_length`, `cache.static_hit` and
`cache.static_empty_hit`. All three pass on the final binary; the final expanded
original matrix retained 149 passes and those three failures. Precise local EOF
and initially empty-body handling restore complete-response accounting and cache
publication. Separately, the 22 original Rust failures were audited as six real
production defects, one missing embedded document and fifteen stale/incomplete
fixtures. Corrections retain privacy and invalidation guarantees. [F2, F4]
The syscall experiment traced 500 measured static 64-KiB requests per binary,
plus warm-up/post-detach checks. Selected calls decreased from 60,374 to 7,502
(87.57%); `read` decreased from 8,135 to 501. All 1,042 responses were exact,
and both traced processes detached and stopped cleanly. This supports a reduction
in observed system-call work; it is not an 87.57% throughput gain, allocation
profile or per-edit attribution. [P1]
## Completed 30-minute mixed-traffic soak
The final executable completed **1,800.000810 seconds** of paced HTTP/1.1
proxy/static traffic at an offered 100 requests/s. All **180,000 workload
operations passed**, with zero response errors and zero scheduler-lateness
events. This includes **1,782 intentional client closes after 128 bytes**;
the remaining **178,218 operations** checked the complete expected response
body and status, including HEAD and ranges. A controlled close does not prove
that the server had not already sent the rest of the body. Every operation used
a new frontend TCP connection; this is not a long-lived keep-alive soak. [F5]
All **14 cache toggles** passed **32 source/mode checks comprising 64 additional
GETs**, covering real HIT behaviour after enabling and bypass after disabling.
The fixture restored its base configuration, completed a 15-second cooldown,
and verified **backend active exchanges = 0**. All four recorded server/origin
processes and the disposable fixture directory were absent after teardown.
The final binary hash remained unchanged. [F5]
| Observation | Final soak result |
|---|---|
| Analytics queue rejects | **0**; recorded-request delta **180,060**, including post-baseline cache probes and cooldown |
| History admission rejects | **31 observed**, from counter 0 at 0.001 s to 31 at 1,798.842 s; this counter was not resampled after cooldown |
| Upstream pool recycling | **1 observed rotation**; the normal bounded recycling policy remained active |
| Sampled RSS peak | **60.641 MiB** |
| RSS median, first minute after warm-up to final sampled minute | **52.273 → 60.016 MiB**, a **7.742 MiB increase** |
| Open file descriptors | **20 at start, 23 sampled peak, 18 after cooldown** |
The final sampled minute includes the cooldown. RSS growth and the descriptive
linear fit of approximately 16.165 MiB/hour are observations over this short,
cache-toggling run, not an extrapolation or a diagnosis of a leak. This test
**does not establish a memory plateau**. There was no accumulating FD signal,
but low-rate correctness and resource observations do not establish stability
at the much higher saturation rates in the primary benchmark. There is no
matched baseline soak from which to attribute this RSS change to the patch.
The RSS/FD samples, both telemetry series and the independent interpretation
are retained. No 24-hour, TLS/H2/H3 or production-traffic soak was executed. [F5]
## Reproduction and deployment boundary
Use project entry points with the private umask they establish:
```bash
./dev.sh check
CARGO_INCREMENTAL=0 ./dev.sh test
./dev.sh release
```
Set the executable/tool variables to reviewed absolute paths and use a fresh
results directory. Build the untouched baseline with the same toolchain/lock
before comparing it with the candidate:
```bash
python3 -B scripts/performance/benchmark.py \
--binary "supplied=$SUPPLIED_BINARY" --binary "before=$REBUILT_BASELINE" \
--binary "after=$CANDIDATE_BINARY" --nginx "$NGINX_BINARY" --wrk "$WRK_BINARY" \
--out "$RESULTS_DIR" --workers 4 --origin-workers 2 \
--server-cpus 0-3 --origin-cpus 4-5 --client-cpus 6-8 --client-threads 4 \
--concurrency 64 --modes proxy,static --sizes small,large,huge \
--repeats 5 --duration 15 --warmup 3
```
See `scripts/performance/README.md` for dependencies and separate regression,
transport, profile and soak commands. The offline analyzer refuses incomplete
or unsealed data. Its outputs include `comparisons.csv`, `paired-ratios.csv`,
`variants.csv`, `observability.csv` and `observability-paired.csv`. [B2]
Remaining acceptance includes remote-network latency/loss, actual PHP-FPM,
Tinyauth and DNS infrastructure, H2/H3 WebSocket tunnels, deployment-specific
24-hour testing and staged rollout. No production deployment or service-manager
rollback was executed. The detailed script-specific upgrade/rollback review is
`verification/release-scope-and-rollback.txt`: current `update.sh` retains two
installed releases and three pre-update backups, conflicting with the older
installation document's no-pruning statement. Keep an independent recovery copy
outside automatic retention and restore matching code/state when needed. [S1]
## Evidence index
| ID | Workspace-relative evidence |
|---|---|
| B1 | `verification/benchmark-primary/`: `runs.json`, `schedule.json`, `environment.json`, `completion.json`, fixture configurations, raw responses/resources/telemetry and `SHA256SUMS` |
| B2 | `verification/analysis/benchmark-primary/analysis.json`, companion CSV/text files, and `verification/analysis/primary-review.txt` |
| B3 | `verification/static-primary-resource-review.txt` and `verification/primary-teardown-audit.json` |
| F1 | `verification/build/final-tests-summary.json`, `dev-test-final.log`; source `docs/TEST_REPORT.md` |
| F2 | `verification/finalrelease-summary.json`; `http-release/`, `lb-release/`, `concurrency-release/` reports beneath `verification/` |
| F3 | `verification/transport-release/release/report.json`; `verification/static-compatibility-audit.txt` |
| F4 | `verification/baseline-failure-audit.txt`; `verification/source_checks/results.json` |
| F5 | `verification/soak-release/`; `verification/soak-release-review.json` and `.txt`; `verification/soak-process-teardown.json` |
| P1 | `verification/syscalls-release/` and `verification/syscalls-release-analysis.txt` |
| S1 | `verification/release-scope-and-rollback.txt`; supplied `AGENT_PERFORMANCE_EXECUTION_PLAN.md`; implementation references listed in that review |
docs/TEST_REPORT.md
+115
View File
@@ -0,0 +1,115 @@
# RustProxy 0.33.8 verification report
This report accompanies the performance and compatibility work on the supplied
0.33.7 source archive and executable. All measurements use disposable local
state. Production data, certificates, configuration and installed services are
not modified. Raw results and exact binary hashes are supplied in the separate
verification archive. The live runners are in `scripts/performance/`.
## Scope and acceptance
The target is reduced work per uncached reverse-proxy request and per local
static-file response while retaining routing, security, cache policy, streaming,
observability and administrative functionality. The LB panel, database schema,
default TLS policy, cache defaults and bounded connection recycling are unchanged.
The performance comparison requires the same compiler, locked dependencies,
release profile, CPU allocation, payloads and client configuration for the
unmodified source and the candidate. The supplied executable is also retained as
an independent compatibility baseline.
Acceptance has separate layers: compilation and Rust tests; source/SQL/upgrade
checks; actual HTTP process tests; actual TLS/H2/H3/WebSocket tests; repeated
before/after/nginx throughput measurements; and a bounded mixed-traffic soak.
A passing source assertion or simulated DOM check is not a live runtime result.
Passing this matrix is not a claim that every possible deployment is covered.
## Baseline findings
The first candidate Rust run completed 658 tests successfully, with 22 failures
and one explicitly ignored formatting microbenchmark. An unmodified-source run
reproduces the old failures; the audit records each stale fixture and each actual
defect separately. The raw initial failed runs remain in the evidence archive.
Tests that expected obsolete behavior are corrected to exercise the currently
documented contract; security, cache invalidation and transport guarantees are
not weakened to obtain a passing result.
The initial live HTTP matrix gave identical results for the supplied executable
and rebuilt 0.33.7: 145 passes, three failures and one skipped optional Brotli
decoder case. All three failures concerned local HTTP/1 file-body completion:
normal responses were recorded as cancelled, and full or empty files failed to
publish a cache entry. The candidate uses a bounded local-file body with accurate
end-of-stream reporting. Arbitrary upstream bodies retain their original trailer
and end-of-stream handling.
The supplied executable passed 111 actual transport assertions covering verified
TLS, H2, H3/QUIC, stream multiplexing, uploads, cache behavior, ranges, validators
and HTTP/1 WebSocket data/control frames. The same runner is used for the final
candidate. The H3 client adapter accounts for HEAD semantics without disabling
GET body-length validation.
## Compatibility checks
The Rust suites exercise cache privacy, Vary and scheme separation, publication
generation barriers, stale access rules, disk tiers, header policies, ACL and
SSRF decision paths, timeout and permit lifetime, actual codecs, filesystem
bounds, symlink policies, range/conditional handling, SQL-backed administration,
TLS configuration, loopback upstream H1/H2 and raw H2 listener behavior. Stream
tests include real TCP/UDP peers. PHP tests use a deterministic FastCGI protocol
peer rather than an installed PHP-FPM application.
The HTTP process runner checks exact response bytes and hashes, streaming before
the final upload chunk, non-replayable POST failures, client cancellation,
configuration reloads, scoped headers and security policy, administration
session/CSRF, cache OFF/ON and privacy behavior, compression and transformations.
The transport runner uses actual local TLS and QUIC connections and trusts the
fixture certificate explicitly.
History-specific tests retain exact log fields, bounded chronological retention,
drop accounting and delivery to live subscribers and configured sinks during
forced history-lock contention. Cloning and destruction of evicted entries occur
outside the history mutex. This preserves the existing best-effort policy;
saturation measurements must still disclose actual history drops.
SQL checks apply all ten migrations twice, exercise named dashboard presets and
account isolation, and verify the 16-preset bound and cascade behavior. The source
upgrade test uses disposable installations to check the stopped-process guard,
held lock rejection, data/configuration preservation, managed source replacement,
backup creation and symlink rejection. Existing JavaScript checks retain their
status as simulated UI checks; no browser acceptance is inferred from them.
## Reproduction
Use the project entry points, which set the private process umask required by
key-store and certificate fixtures:
```bash
./dev.sh check
CARGO_INCREMENTAL=0 ./dev.sh test
python3 scripts/tests/dashboard_presets_sql_test.py
python3 scripts/tests/upgrade_performance_test.py
```
The isolated validation environment uses Rust 1.99.1 with the original locked
dependency versions, nginx 1.26.3 and wrk 4.2.0. See
`scripts/performance/README.md` for live regression, transport, profiling,
benchmark and soak commands. Supply explicit executable paths and new output
directories; preserve failed runs as evidence.
## Measurement boundaries
Throughput measurements use HTTP/1.1 keep-alive, cache OFF, identity encoding,
1040-byte, 64-KiB and 1-MiB objects, separate server/origin/client CPU sets,
interleaved variants and excluded warm-up. Resource samples retain CPU time,
RSS, file descriptors, generator errors and application drop counters. Medians
use all scheduled repetitions. Closed-loop wrk latency is not an open-loop SLA.
Syscall profiles are collected separately from timed throughput runs.
Remaining deployment-specific checks include a 24-hour production-like soak,
remote-network latency and loss, actual PHP-FPM applications, external Tinyauth,
DNS rebinding against a real resolver, and HTTP/2 or HTTP/3 WebSocket tunnels.
The measured same-host comparison does not establish internet-facing capacity.
Final executable-specific outcomes, run counts, percentages and resource costs
are recorded in the separately generated performance/evidence report after the
release build. This embedded document describes the test contract and baseline;
it does not substitute planned work for a completed measurement.
rust-proxy
Executable
BIN
View File
Binary file not shown.
scripts/performance/README.md
+276
View File
@@ -0,0 +1,276 @@
# Live performance and regression harness
These scripts exercise actual RustProxy executables through loopback sockets.
They do not simulate Rust request handling. The Python server in `peer.py` is a
controllable test **upstream**, used to verify streaming, non-replayable writes,
headers, cache policy and WebSocket frames. Performance measurements use nginx
as the shared upstream instead of Python.
## Requirements
- Linux with readable `/proc`, Python 3.10+, executable RustProxy binaries.
- nginx and wrk for performance comparisons. Both paths are explicit; no
system-wide installation is required.
- `taskset` from util-linux for the benchmark's explicit CPU affinity and for
optional affinity settings in the other scripts.
- Optional Python `brotli` and `zstandard` decoders increase compression
acceptance coverage. A missing optional decoder is reported as **skipped**.
- Permission to bind unprivileged loopback sockets and start local processes.
In a sandbox without sockets or `/proc`, run in the approved local execution
environment. Do not replace real runtime results with a peer self-test.
Each run creates a new private temporary data directory and initializes its own
administrator with a random password supplied only over stdin. Configurations
are applied through RustProxy's existing authenticated CLI/API and listener
changes receive a restart. No historical SQLite database is copied or edited.
TLS keys, databases and passwords are never written to the evidence directory.
Temporary state is outside the evidence tree and is removed on normal exit.
`fixture-cleanup.json` records the cleanup check. Interrupted runs can leave an
unrelated `/tmp/rustproxy-perf-*` directory; do not include it in a release.
Before timed benchmark, soak or syscall-profile work, `wait_quiescent()` waits
for the existing authenticated ACME runtime API to report `ready` or
`unavailable`. A fresh RustProxy data directory can start a Python/Certbot
environment bootstrap even when no certificates have an issuer. The gate
requires no configured issuers, jobs or renewal observations, the same live Rust
PID and start time, and no child processes for five continuous seconds. It polls
every 0.5 seconds and fails after 180 seconds instead of measuring installation
work. Selected status, process count and timing evidence is saved in
`<label>-quiescence.json`. No PATH, pip settings or application features are
changed. An `unavailable` result permits the HTTP fixture to proceed; these runs
do not test ACME issuance or certificate-provider availability.
## Regression tests
Run from the repository root, providing absolute binary and output paths:
```bash
python3 scripts/performance/regression.py \
--binary before=/absolute/path/rust-proxy-before \
--binary after=/absolute/path/rust-proxy-after \
--out /absolute/path/results/regression
```
`--groups static,proxy,policy,cache,transform` selects groups; all are enabled by
default. `--workers 4` sets the same `TOKIO_WORKER_THREADS` for each binary.
The report contains individual pass/fail/skip observations and a comparison
matrix. An existing failure in the supplied binary remains a failure in its
report; it is not silently converted to a successful baseline expectation.
Coverage includes:
- Static GET/HEAD and byte checksums at 1,040 B, 64 KiB and 1 MiB; empty files;
ETag/Last-Modified conditional requests; prefix/suffix/open/multipart ranges,
If-Range and 416; directory redirect/index/listing; decoded path rejection;
deny/inside-root symlinks; precompressed gzip; MIME/download headers;
`try_files`, redirects and the compiled `.htaccess` header subset.
- Proxy POST/PUT, 1 MiB upload checksums, chunked upload delivery **before the
last chunk is sent**, early response delivery, client disconnect, original
status/redirect preservation, failed-POST no replay, request/response header
scopes, hop-by-hop stripping, request-ID shape/uniqueness and WebSocket echo.
- Global host/vhost/profile/location ACL composition, upstream suppression on
denial, trusted versus spoofed forwarding, untrusted country hints, method
and rate policies, hot changes and rejection of invalid configuration.
- Private administration routes, login, session/CSRF and Origin enforcement,
Basic Auth, ACL-before-auth, private-origin rejection, retained exchange
logging fields and byte accounting.
- Cache global/vhost/route OFF and ON, MISS/HIT, Vary isolation, cookie,
Authorization/private/no-store bypass, PURGE, concurrent fills, validator
revalidation, static cached ranges/conditionals; transforms OFF/ON with cache
OFF/ON; compression negotiation, exclusions and scope/global codec ceilings.
The generic suite does not establish TLS/H2/H3, Tinyauth, real PHP-FPM, real DNS
rebinding, concurrent filesystem rename-race safety or long-soak acceptance.
Use the dedicated transport tests and existing Rust integration suites for
their covered cases. The sequential symlink-target test is explicitly not a
concurrent rename-race proof.
## TLS, HTTP/2, HTTP/3 and WebSocket interoperability
```bash
python3 scripts/performance/transport_regression.py \
--binary before=/absolute/path/rust-proxy-before \
--binary after=/absolute/path/rust-proxy-after \
--out /absolute/path/results/transport
```
This suite requires `aioquic`, `httpx[http2]` and `websockets`; use
`--python-deps /absolute/path/pydeps` for a separate dependency directory.
It verifies the temporary server certificate, negotiated protocol, response
status and bytes for TLS H2 and QUIC H3, including HEAD, validators, ranges,
cache ON/OFF, 1 MiB bodies and concurrent streams. Dedicated WebSocket checks
include bidirectional echo and a normal close handshake. Client protocol
self-checks are reported separately from RustProxy acceptance cases.
This establishes covered interoperability, not TLS/H2/H3 throughput or
production-certificate compatibility.
## Two-origin load-balancer regression
```bash
python3 scripts/performance/lb_regression.py \
--binary before=/absolute/path/rust-proxy-before \
--binary after=/absolute/path/rust-proxy-after \
--out /absolute/path/results/lb
```
This separate, small suite sends real requests through RustProxy to two local
TCP origins. It verifies equal round-robin allocation, GET failover after a 503
and after a stopped origin refuses connections, and the absence of POST replay.
A controlled response pauses after its first body fragment while configuration
changes drain or disable its member. New requests must use the other member,
the admitted response must finish intact, and its active lease must remain
counted until completion and then return to zero. The main HTTP case matrix is
unchanged. This suite does not cover sticky sessions, active health probes,
DNS rotation or TLS upstream failures.
## Finite concurrency correctness stress
```bash
python3 scripts/performance/concurrency_regression.py \
--binary before=/absolute/path/rust-proxy-before \
--binary after=/absolute/path/rust-proxy-after \
--nginx /absolute/path/nginx --out /absolute/path/results/concurrency
```
This separate functional check uses 1, 16 and 128 concurrent client tasks for
proxy and static responses at 1,040 B, 64 KiB and 1 MiB. Each finite batch sends
`max(32, 2 * clients)` requests and verifies every status, length and body SHA-256.
After the clients finish, backend active-exchange counters must settle to zero.
RSS and file descriptors are recorded before the batch and after its drain.
These are not peak resource measurements or throughput results. Client tasks
start behind a barrier; configured concurrency does not establish that the same
number of server exchanges were active simultaneously.
## Before/after/nginx benchmark
First verify starts, loopback routing and exact payload hashes without invoking
the load generator:
```bash
python3 scripts/performance/benchmark.py \
--binary before=/absolute/path/rust-proxy-before \
--binary after=/absolute/path/rust-proxy-after \
--nginx /absolute/path/nginx --wrk /absolute/path/wrk \
--server-cpus 0-3 --origin-cpus 4-5 --client-cpus 6-8 \
--out /absolute/path/results/preflight --prepare-only
```
For a host with CPUs 0–8 available, a reproducible primary run is:
```bash
python3 scripts/performance/benchmark.py \
--binary before=/absolute/path/rust-proxy-before \
--binary after=/absolute/path/rust-proxy-after \
--nginx /absolute/path/nginx --wrk /absolute/path/wrk \
--out /absolute/path/results/primary \
--workers 4 --origin-workers 2 --client-threads 4 \
--server-cpus 0-3 --origin-cpus 4-5 --client-cpus 6-8 \
--concurrency 64 --sizes small,large,huge --modes proxy,static \
--duration 15 --warmup 3 --repeats 5
```
Choose explicit, nonempty and disjoint CPU sets allowed in the run environment;
adapt all three sets on a smaller host. Frontends use the same CPU set and worker
count, the shared origin uses another set, and the generator uses a third. The
harness checks each process/thread's actual affinity and refuses unavailable
resource counters. Four wrk threads share the three client CPUs in this example.
The structured result records thread count and effective connection count; at
64 connections, four threads keep exactly 64 connections, avoiding wrk's integer
division of connections between threads. Only one frontend is loaded at a time.
Stop builds, other tests and profilers during throughput measurement.
The primary run takes approximately 27 minutes: six scenarios, three variants,
five samples, and 18 seconds per sample including excluded warm-up. To explore
concurrency, repeat in a fresh output directory with
`--concurrency 1,16,64,128`. More scenarios intentionally take more time.
Short sanity checks require the explicit `--exploratory` option and cannot
satisfy the >=5 samples / >=15-second measurement gate.
Cache is explicitly disabled for the primary benchmark. Transforms retain their
disabled defaults. Identity encoding is requested. Logging history, analytics,
request IDs, origin policy, body accounting, health/lease plumbing and normal
application functionality remain active. nginx uses `sendfile on`, HTTP/1.1,
keep-alive and streaming proxy delivery. The generated exact configurations are
saved, so this intentional difference in nginx/application features is visible.
No cache-disabled claim is inferred from a cache-on speedup.
Optional keep-alive tuning uses `--max-idle label=64`. Keep this as a separately
labelled experiment. The harness does not turn off bounded pool recycling.
## Recorded evidence and interpretation
- `schedule.json`: complete deterministic crossover order, rotating and
reversing variants between repetitions. No best-run selection.
- `preflight.json`, `fixture/payloads.json`, binary/version/config metadata:
exact status, body SHA-256, payload sizes and inputs.
- Per-sample exact status/body checks before and after the measured interval,
plus a wire-bytes-per-request sanity check, detect wrong content and gross
transfer-size anomalies. These do not hash every response inside wrk.
- `raw/*.txt` and `raw/*.json`: unmodified wrk output and structured requests,
duration, bytes, RPS, p50/p95/p99, effective concurrency, and
connect/read/write/timeout/status errors.
- `raw/*-resources.json`: timestamped `/proc` samples for each server process
and its children, plus the separate origin process group.
- `raw/*-affinity.json`: observed process/thread CPU masks for the front end,
origin and generator.
- `runs.csv/json`, `summary.csv/json`: all samples, medians, RPS range and
coefficient of variation, latency medians, server CPU time per completed
request, peak/end RSS and file descriptors.
- `status/*.json`: numeric drop and pool-generation/rotation counters read
outside each measured interval. Warm-up output is kept separately.
- `completion.json` and `SHA256SUMS`: completion state and hashes of evidence.
Any generator/network/status error stops the run with `failure.json`; partial
data remains available and is not presented as completed acceptance.
Use `(after_median / before_median - 1) * 100` for throughput improvement and
`after_median / nginx_median * 100` for relative nginx throughput. Rebuilding
both revisions with one compiler, target and resolved lockfile is required for
source-change attribution. Compare the supplied executable separately: build
flags and dependency versions can change its performance.
These are same-host, closed-loop HTTP/1.1 diagnostics. They do not establish
production capacity, TLS/H2/H3 performance, an open-loop latency SLA, allocator
rates, CPU cycles or 24-hour resource stability. wrk latency follows wrk's own
histogram/correction behavior. RSS and descriptor peaks are sampled observations,
so shorter spikes may be missed. Summed nginx RSS can double-count shared pages.
Origin or generator CPU saturation can limit observed throughput; inspect their
resource data and move the generator/origin to separate machines for release
measurements. Run profilers separately so profiling overhead does not enter the
headline A/B throughput numbers.
## Mixed-traffic resource soak
Run this in its own phase, after correctness and throughput checks:
```bash
python3 scripts/performance/soak.py \
--binary /absolute/path/rust-proxy-after --label after \
--nginx /absolute/path/nginx --out /absolute/path/results/soak \
--duration 1800 --rate 100 --clients 8 \
--workers 4 --origin-workers 2 --server-cpus 0-3 --origin-cpus 4-5 \
--toggle-seconds 120 --sample-seconds 5 --abort-every 101
```
The bounded client executor alternates proxy/static and 1,040 B/64 KiB/1 MiB
payloads, verifies full bytes and occasional HEAD/range semantics, closes some
clients after an initial body prefix, and applies live cache OFF/ON changes.
Every workload request opens a new frontend HTTP/1.1 connection; upstream
keepalive retains its configured defaults. This exercises connection churn.
The primary wrk benchmark separately uses persistent frontend connections.
The nominal request rate is paced; scheduler delays and actual completion rate
are reported instead of hiding an under-driven client. Pending jobs are bounded
to twice the client count. The script restores base configuration, drains and
rechecks both listeners before shutdown. After each cache apply, two separate
GETs per source verify exact contents and require a HIT on the second GET when
enabled; globally disabled cache must expose no X-Cache header. These probe
requests are recorded separately from paced traffic. Backend active-exchange
counters must settle to zero after the final drain and cooldown.
Five-second resource/status records, body/status failures, observed cache-status
counts, cache transitions/probes, RSS/FD trend and completed traffic totals are
retained. Controlled client closes mean that the client closes after reading a
prefix. The server may already have sent the complete response, so those counts
are not presented as proven server-side cancellations.
Memory trend is descriptive: switching caching and allocator retention affect
RSS. This is a 30-minute resource gate, not a 24-hour leak proof. A short
`--exploratory --duration ...` sanity run stays explicitly outside that gate.
scripts/performance/benchmark.py
Executable
+479
View File
@@ -0,0 +1,479 @@
#!/usr/bin/env python3
"""Interleaved before/after/nginx process benchmarks with immutable raw evidence.
This is a same-host diagnostic experiment, not a simulation or release sign-off.
Runs >=5 measured samples per variant/scenario by default, each >=15 seconds.
"""
from __future__ import annotations
import argparse
import csv
import hashlib
import itertools
import json
import os
from pathlib import Path
import platform
import resource
import statistics
import subprocess
import sys
import time
import traceback
from common import Fixture, http, seal_results, sha256, write_json
from resources import (Monitor, affinity_snapshot, cgroup_cpu_delta, cgroup_cpu_stat,
parse_cpu_set, validate_cpu_sets, wait_for_affinity)
from telemetry import traffic_analytics_delta, traffic_analytics_snapshot
SIZES = {"small": "small.html", "large": "large.bin", "huge": "huge.bin"}
DROP_COUNTER_FIELDS = {
"logs_dropped": "history_dropped_delta",
"logging.sinks.queue_dropped": "logging_sinks_queue_dropped_delta",
"logging.error_file.queue_dropped": "logging_error_file_queue_dropped_delta",
"cache.cold_queue_dropped": "cache_cold_queue_dropped_delta",
}
DROP_INTERVAL_NOTE = ("Delta brackets measured wrk plus one postflight request and possible in-flight completions; "
"percentage uses wrk completed requests as an approximate denominator. "
"History rejection does not imply rejection by live broadcast or external sinks.")
def usage_snapshot():
usage = resource.getrusage(resource.RUSAGE_CHILDREN)
return usage.ru_utime + usage.ru_stime
def effective_threads(requested, concurrency):
if requested < 1 or concurrency < 1:
raise ValueError("client threads and concurrency must be positive")
# wrk floor-divides the connection count between threads. Select a divisor
# so its advertised and effective connection counts remain identical.
return next(value for value in range(min(requested, concurrency), 0, -1)
if concurrency % value == 0)
def wrk_run(args, port, path, duration, concurrency, prefix, roots=None, expected_processes=None):
threads = effective_threads(args.client_threads, concurrency)
command = [str(args.wrk), "--latency", "-t", str(threads),
"-c", str(concurrency), "-d", f"{duration}s", "--timeout", "5s",
"-H", "Accept-Encoding: identity", "-s", str(Path(__file__).with_name("wrk_report.lua")),
f"http://127.0.0.1:{port}/{path}"]
if args.client_cpus:
command = ["taskset", "-c", args.client_cpus, *command]
affinities = {}
expected_processes = expected_processes or {}
if roots:
for label, pid in roots.items():
expected = parse_cpu_set(args.server_cpus if label == "server" else args.origin_cpus)
affinities[label] = affinity_snapshot(pid, expected, expected_processes.get(label))
monitor = Monitor(roots, args.resource_interval, expected_processes) if roots else None
if monitor:
monitor.start()
cpu_start = usage_snapshot()
started = time.monotonic()
process = None
stdout, stderr = "", ""
try:
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
affinities["generator"] = wait_for_affinity(process, parse_cpu_set(args.client_cpus))
stdout, stderr = process.communicate(timeout=duration + 30)
finally:
if process is not None and process.poll() is None:
process.kill()
stdout, stderr = process.communicate(timeout=5)
if monitor:
monitor.stop()
prefix.parent.mkdir(parents=True, exist_ok=True)
prefix.with_suffix(".txt").write_text(stdout + "\n" + stderr)
write_json(prefix.with_name(prefix.name + "-affinity.json"), affinities)
if monitor:
write_json(prefix.with_name(prefix.name + "-resources.json"), monitor.samples)
if monitor.errors:
write_json(prefix.with_name(prefix.name + "-resource-errors.json"), monitor.errors)
cpu_seconds = usage_snapshot() - cpu_start
wall_seconds = time.monotonic() - started
raw = stdout + "\n" + stderr
if process.returncode:
raise RuntimeError(f"wrk exited {process.returncode}: {raw[-2000:]}")
line = next((line for line in stdout.splitlines() if line.startswith("RUSTPROXY_BENCH_JSON ")), None)
if line is None:
raise RuntimeError("wrk produced no structured report")
result = json.loads(line.removeprefix("RUSTPROXY_BENCH_JSON "))
if result["requests"] <= 0 or result["duration_us"] <= 0:
write_json(prefix.with_suffix(".json"), result)
raise RuntimeError("wrk completed no requests or reported an invalid duration; raw evidence retained")
result.update({"command": command, "wall_seconds": wall_seconds,
"client_threads": threads, "requested_connections": concurrency,
"client_threads_requested": args.client_threads,
"effective_connections": threads * (concurrency // threads),
"generator_cpu_seconds": cpu_seconds,
"rps": result["requests"] * 1_000_000 / result["duration_us"],
"error_total": sum(value for key, value in result.items() if key.startswith("errors_"))})
if monitor:
result["resources"] = monitor.summary(result["requests"])
write_json(prefix.with_suffix(".json"), result)
return result
def payload_check(port, path, expected):
"""A full exact-body check outside the wrk measurement interval."""
try:
status, headers, body = http(port, "/" + path, headers={"Accept-Encoding": "identity"})
return {"status": status, "bytes": len(body), "sha256": hashlib.sha256(body).hexdigest(),
"expected_bytes": len(expected), "expected_sha256": hashlib.sha256(expected).hexdigest(),
"content_length": headers.get("content-length"), "content_encoding": headers.get("content-encoding"),
"approximate_header_bytes": 19 + sum(len(key) + len(value) + 4 for key, value in headers.items()),
"passed": status == 200 and body == expected}
except Exception as error:
return {"passed": False, "error": str(error), "expected_bytes": len(expected),
"expected_sha256": hashlib.sha256(expected).hexdigest()}
def transfer_sanity(result, payload_bytes, header_bytes, concurrency):
"""Check gross truncation/size anomalies; this is not per-response hashing."""
requests, wire_bytes = result["requests"], result["bytes"]
minimum = requests * payload_bytes
# wrk counts headers and partially received final responses in its wire bytes.
header_allowance = max(1024, 2 * header_bytes)
maximum = (requests + concurrency) * (payload_bytes + header_allowance)
return {"passed": requests > 0 and minimum <= wire_bytes <= maximum,
"wire_bytes_per_request": wire_bytes / requests if requests else None,
"expected_payload_bytes_per_request": payload_bytes,
"minimum_wire_bytes": minimum, "maximum_wire_bytes": maximum,
"header_allowance_per_response": header_allowance,
"inflight_response_allowance": concurrency}
def counter_values(value, prefix=""):
"""Retain numeric drop and rotation counters without session/config data."""
found = {}
if isinstance(value, dict):
for key, child in value.items():
path = f"{prefix}.{key}" if prefix else key
if isinstance(child, (int, float)) and any(part in key for part in ["dropped", "rotations", "generation", "requests_in_generation"]):
found[path] = child
found.update(counter_values(child, path))
elif isinstance(value, list):
for index, child in enumerate(value):
found.update(counter_values(child, f"{prefix}[{index}]"))
return found
def validate_drop_counters(values):
for name in DROP_COUNTER_FIELDS:
if type(values.get(name)) is not int or values[name] < 0:
raise RuntimeError(f"missing or invalid cumulative status counter: {name}")
return values
def status_counter_snapshot(rust):
if rust is None:
return None
status, _, body = rust.admin("/api/status")
if status != 200:
raise RuntimeError(f"{rust.label} status capture failed: HTTP {status}")
return counter_values(json.loads(body))
def drop_counter_report(before, after, requests):
if requests <= 0:
raise ValueError("drop-rate denominator must contain completed requests")
if (before is None) != (after is None):
raise RuntimeError("both status snapshots are required for a drop delta")
if before is not None:
validate_drop_counters(before)
validate_drop_counters(after)
deltas = {name: after[name] - before[name] if before is not None else None
for name in DROP_COUNTER_FIELDS}
if any(value is not None and value < 0 for value in deltas.values()):
raise RuntimeError("cumulative drop counter decreased during measured interval")
fields = {field: deltas[name] for name, field in DROP_COUNTER_FIELDS.items()}
fields["history_drop_percent_of_completed_requests"] = (
100 * deltas["logs_dropped"] / requests if before is not None else None)
return {"applicable": before is not None, "counter_deltas": deltas, "row_fields": fields,
"completed_requests_denominator": requests, "denominator_is_approximate": True,
"interval_note": DROP_INTERVAL_NOTE}
def analytics_counter_report(before, after, requests):
if requests <= 0 or (before is None) != (after is None):
raise ValueError("analytics delta requires paired snapshots and completed requests")
delta = traffic_analytics_delta(before, after) if before is not None else {
"queue_dropped_delta": None, "recorded_requests_delta": None, "interval_seconds": None}
fields = {"analytics_queue_dropped_delta": delta["queue_dropped_delta"],
"analytics_recorded_requests_delta": delta["recorded_requests_delta"],
"analytics_queue_drop_percent_of_completed_requests": (
100 * delta["queue_dropped_delta"] / requests if before is not None else None)}
return {"applicable": before is not None, **delta, "row_fields": fields,
"completed_requests_denominator": requests, "denominator_is_approximate": True,
"interval_note": DROP_INTERVAL_NOTE,
"recorded_requests_note": "Asynchronous analytics records; not HTTP success or completion counts."}
def persist_rows(out, rows):
write_json(out / "runs.json", rows)
if not rows:
return
with (out / "runs.csv").open("w", newline="") as stream:
writer = csv.DictWriter(stream, fieldnames=list(rows[0]))
writer.writeheader()
writer.writerows(rows)
groups = {}
for row in rows:
groups.setdefault((row["mode"], row["size"], row["concurrency"], row["variant"]), []).append(row)
summary = []
for (mode, size, concurrency, variant), group in sorted(groups.items()):
rps = [row["rps"] for row in group]
summary.append({"mode": mode, "size": size, "concurrency": concurrency, "variant": variant,
"samples": len(group), "rps_median": statistics.median(rps),
"rps_min": min(rps), "rps_max": max(rps),
"rps_cv_percent": statistics.stdev(rps) / statistics.mean(rps) * 100 if len(rps) > 1 else None,
"p50_us_median": statistics.median(row["p50_us"] for row in group),
"p95_us_median": statistics.median(row["p95_us"] for row in group),
"p99_us_median": statistics.median(row["p99_us"] for row in group),
"server_cpu_us_per_request_median": statistics.median(row["server_cpu_us_per_request"] for row in group),
"server_rss_peak_bytes": max(row["server_rss_peak_bytes"] for row in group),
"server_fd_peak": max(row["server_fd_peak"] for row in group),
"requests": sum(row["requests"] for row in group),
"cgroup_throttled_samples": sum(row["cgroup_nr_throttled_delta"] > 0 for row in group),
"cgroup_nr_throttled_delta": sum(row["cgroup_nr_throttled_delta"] for row in group),
"cgroup_throttled_usec_delta": sum(row["cgroup_throttled_usec_delta"] for row in group),
"validation_failures": sum(not (row["payload_before_passed"] and row["payload_after_passed"]
and row["transfer_sanity_passed"]) for row in group),
"errors": sum(row["error_total"] for row in group)})
write_json(out / "summary.json", summary)
with (out / "summary.csv").open("w", newline="") as stream:
writer = csv.DictWriter(stream, fieldnames=list(summary[0]))
writer.writeheader()
writer.writerows(summary)
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--binary", action="append", required=True, metavar="LABEL=/ABS/PATH")
parser.add_argument("--max-idle", action="append", default=[], metavar="LABEL=COUNT")
parser.add_argument("--nginx", type=Path, required=True)
parser.add_argument("--wrk", type=Path, required=True)
parser.add_argument("--out", type=Path, required=True)
parser.add_argument("--workers", type=int, default=4)
parser.add_argument("--origin-workers", type=int, default=4)
parser.add_argument("--client-threads", type=int, default=4)
parser.add_argument("--server-cpus")
parser.add_argument("--origin-cpus")
parser.add_argument("--client-cpus")
parser.add_argument("--duration", type=int, default=15)
parser.add_argument("--warmup", type=int, default=3)
parser.add_argument("--repeats", type=int, default=5)
parser.add_argument("--concurrency", default="64")
parser.add_argument("--sizes", default="small,large")
parser.add_argument("--modes", default="proxy,static")
parser.add_argument("--resource-interval", type=float, default=.5)
parser.add_argument("--exploratory", action="store_true", help="explicitly allow <5 repetitions or <15s samples")
parser.add_argument("--prepare-only", action="store_true", help="start processes and verify hashes; run no load generator")
args = parser.parse_args()
if (args.duration < 15 or args.repeats < 5) and not args.exploratory:
parser.error("acceptance requires >=15-second runs and >=5 repetitions; use --exploratory for labelled short diagnostics")
if args.out.exists() and any(args.out.iterdir()):
parser.error("--out must be a new or empty directory so raw evidence is never overwritten")
modes, sizes = args.modes.split(","), args.sizes.split(",")
concurrencies = [int(value) for value in args.concurrency.split(",")]
if any(mode not in ("proxy", "static") for mode in modes) or any(size not in SIZES for size in sizes):
parser.error("supported modes: proxy,static; sizes: small,large,huge")
if min(concurrencies) < 1 or max(concurrencies) > 4096:
parser.error("concurrency must be 1..4096")
if args.client_threads < 1:
parser.error("--client-threads must be positive")
if args.workers < 1 or args.origin_workers < 1:
parser.error("--workers and --origin-workers must be positive")
try:
cpu_sets = validate_cpu_sets({"server": args.server_cpus, "origin": args.origin_cpus,
"client": args.client_cpus}, set(os.sched_getaffinity(0)))
except ValueError as error:
parser.error(str(error))
if args.resource_interval <= 0:
parser.error("--resource-interval must be positive")
args.nginx, args.wrk, args.out = args.nginx.resolve(), args.wrk.resolve(), args.out.resolve()
variants = dict(item.split("=", 1) for item in args.binary)
if len(variants) != len(args.binary) or "nginx" in variants:
parser.error("variant names must be unique and must not be nginx")
max_idle = dict(item.split("=", 1) for item in args.max_idle)
args.out.mkdir(parents=True, exist_ok=True)
environment = {"kind": "same-host real HTTP/1.1 process benchmark", "simulation": False,
"exploratory": args.exploratory, "prepare_only": args.prepare_only,
"acceptance_sampling": args.duration >= 15 and args.repeats >= 5 and not args.prepare_only,
"platform": platform.platform(), "python": sys.version,
"started_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
"cpu_affinity_available": sorted(os.sched_getaffinity(0)),
"cpu_sets": {label: sorted(cpus) for label, cpus in cpu_sets.items()},
"resource_process_discovery": "PPid map from /proc/*/stat; root and configured nginx process counts must be readable",
"expected_nginx_processes": {"frontend": args.workers + 1, "origin": args.origin_workers + 1},
"generator_plan": [{"connections": concurrency, "requested_threads": args.client_threads,
"effective_threads": effective_threads(args.client_threads, concurrency)}
for concurrency in concurrencies],
"settings": {key: str(value) if isinstance(value, Path) else value for key, value in vars(args).items()},
"wrk_sha256": sha256(args.wrk), "nginx_sha256": sha256(args.nginx),
"limitations": ["Generator, upstream and frontend share one host and kernel; not production capacity",
"Closed-loop wrk latency includes its histogram correction; no open-loop arrival-rate SLA claim",
"RSS sums processes and may double-count shared pages in nginx",
"RSS and FD peaks are sampled, not absolute instantaneous peaks",
"Exact HTTP 200 and payload hashes are checked before/after each measured run, not for every response during load",
"wrk status errors count only status >399; aggregate bytes/request is a gross sanity check, not response integrity proof",
"cgroup CPU-throttling deltas cover all processes in the shared cgroup, not only the measured server",
DROP_INTERVAL_NOTE,
"No TLS/H2/H3 speed inference; no CPU cycles/allocation profiler claim",
"nginx has a smaller feature set; request logging/ACL/analytics work is not feature-matched"]}
for path in ["/sys/fs/cgroup/cpu.max", "/sys/fs/cgroup/memory.max", "/proc/version"]:
try:
environment[path] = Path(path).read_text().strip()
except OSError:
environment[path] = None
write_json(args.out / "environment.json", environment)
expected_samples = len(modes) * len(sizes) * len(concurrencies) * args.repeats * (len(variants) + 1)
write_json(args.out / "completion.json", {"completed": False, "samples": 0,
"expected_samples": expected_samples,
"simulation": False, "prepare_only": args.prepare_only})
rows, error = [], None
try:
with Fixture(args.out / "fixture", workers=args.workers, server_cpus=args.server_cpus,
origin_cpus=args.origin_cpus, origin_workers=args.origin_workers) as fixture:
nginx = fixture.nginx(args.nginx)
servers = {"nginx": {"process": nginx["process"], "proxy": nginx["reverse_port"],
"static": nginx["static_port"], "rust": None}}
for label, binary in variants.items():
rust = fixture.rust(Path(binary), label, max_idle=int(max_idle[label]) if label in max_idle else None)
rust.login()
servers[label] = {"process": rust.process, "proxy": rust.reverse_port, "static": rust.static_port, "rust": rust}
quiescence = {label: server["rust"].wait_quiescent()
for label, server in servers.items() if server["rust"] is not None}
write_json(args.out / "quiescence-preflight.json", quiescence)
initial_affinity = {"origin": affinity_snapshot(nginx["origin"].pid, cpu_sets["origin"], args.origin_workers + 1)}
initial_affinity.update({label: affinity_snapshot(server["process"].pid, cpu_sets["server"],
args.workers + 1 if label == "nginx" else 1)
for label, server in servers.items()})
write_json(args.out / "affinity-preflight.json", initial_affinity)
# The fixture has only master + configured workers. Refuse a
# master-only snapshot even if HTTP payload checks would succeed.
prepare_monitor = Monitor({"server": nginx["process"].pid, "origin": nginx["origin"].pid},
args.resource_interval,
{"server": args.workers + 1, "origin": args.origin_workers + 1})
prepare_monitor.sample()
write_json(args.out / "resources-preflight.json", prepare_monitor.samples)
telemetry_preflight = {label: {"status": status_counter_snapshot(server["rust"]),
"traffic": traffic_analytics_snapshot(server["rust"]) if server["rust"] else None}
for label, server in servers.items()}
write_json(args.out / "telemetry-preflight.json", telemetry_preflight)
for snapshot in telemetry_preflight.values():
if snapshot["status"] is not None:
validate_drop_counters(snapshot["status"])
preflight = []
for label, server in servers.items():
for mode, size in itertools.product(modes, sizes):
payload = (fixture.site / SIZES[size]).read_bytes()
check = payload_check(server[mode], SIZES[size], payload)
preflight.append({"variant": label, "mode": mode, "size": size, **check})
write_json(args.out / "preflight.json", preflight)
if not check["passed"]:
raise AssertionError(f"payload preflight failed: {label} {mode} {size}: {check}")
write_json(args.out / "preflight.json", preflight)
labels = [*variants, "nginx"]
schedule = []
for mode, size, concurrency in itertools.product(modes, sizes, concurrencies):
for repetition in range(args.repeats):
offset = repetition % len(labels)
order = labels[offset:] + labels[:offset]
if repetition % 2:
order.reverse()
for variant in order:
schedule.append({"sequence": len(schedule) + 1, "mode": mode, "size": size,
"concurrency": concurrency, "repetition": repetition + 1, "variant": variant})
write_json(args.out / "schedule.json", schedule)
if not args.prepare_only:
for item in schedule:
server = servers[item["variant"]]
expected_processes = {"origin": args.origin_workers + 1,
"server": args.workers + 1 if item["variant"] == "nginx" else 1}
stem = f"{item['sequence']:04}-{item['mode']}-{item['size']}-c{item['concurrency']}-{item['variant']}"
if args.warmup:
warmup = wrk_run(args, server[item["mode"]], SIZES[item["size"]], args.warmup,
item["concurrency"], args.out / "warmup" / stem)
if warmup["error_total"]:
raise RuntimeError(f"warmup errors in {stem}; raw evidence retained")
payload = (fixture.site / SIZES[item["size"]]).read_bytes()
before = payload_check(server[item["mode"]], SIZES[item["size"]], payload)
write_json(args.out / "checks" / (stem + "-before.json"), before)
if not before["passed"]:
raise RuntimeError(f"exact payload check failed before {stem}: {before}")
counters_before = status_counter_snapshot(server["rust"])
write_json(args.out / "status" / (stem + "-before.json"), counters_before)
if counters_before is not None:
validate_drop_counters(counters_before)
analytics_before = traffic_analytics_snapshot(server["rust"]) if server["rust"] else None
write_json(args.out / "status" / (stem + "-traffic-before.json"), analytics_before)
cpu_before = cgroup_cpu_stat()
try:
result = wrk_run(args, server[item["mode"]], SIZES[item["size"]], args.duration,
item["concurrency"], args.out / "raw" / stem,
{"server": server["process"].pid, "origin": nginx["origin"].pid},
expected_processes)
finally:
cpu_after = cgroup_cpu_stat()
cpu_delta = cgroup_cpu_delta(cpu_before, cpu_after)
throttling = {"before": cpu_before, "after": cpu_after, "delta": cpu_delta}
write_json(args.out / "checks" / (stem + "-cgroup-cpu.json"), throttling)
after = payload_check(server[item["mode"]], SIZES[item["size"]], payload)
write_json(args.out / "checks" / (stem + "-after.json"), after)
write_json(args.out / "checks" / (stem + "-affinity-after.json"), {
"server": affinity_snapshot(server["process"].pid, cpu_sets["server"], expected_processes.get("server")),
"origin": affinity_snapshot(nginx["origin"].pid, cpu_sets["origin"], expected_processes["origin"])})
counters_after = status_counter_snapshot(server["rust"])
write_json(args.out / "status" / (stem + ".json"), counters_after)
analytics_after = traffic_analytics_snapshot(server["rust"]) if server["rust"] else None
write_json(args.out / "status" / (stem + "-traffic-after.json"), analytics_after)
drops = drop_counter_report(counters_before, counters_after, result["requests"])
write_json(args.out / "status" / (stem + "-delta.json"), drops)
analytics = analytics_counter_report(analytics_before, analytics_after, result["requests"])
write_json(args.out / "status" / (stem + "-traffic-delta.json"), analytics)
sanity = transfer_sanity(result, len(payload), before["approximate_header_bytes"], item["concurrency"])
result.update({"payload_before": before, "payload_after": after, "transfer_sanity": sanity,
"cgroup_cpu": throttling, "observability": drops, "traffic_analytics": analytics})
write_json((args.out / "raw" / stem).with_suffix(".json"), result)
row = {**item, **{key: result[key] for key in ["requests", "bytes", "duration_us", "rps", "p50_us", "p95_us", "p99_us",
"client_threads_requested", "client_threads", "requested_connections", "effective_connections",
"error_total", "errors_connect", "errors_read", "errors_write", "errors_timeout", "errors_status", "generator_cpu_seconds"]}}
row.update({"payload_before_passed": before["passed"], "payload_after_passed": after["passed"],
"transfer_sanity_passed": sanity["passed"], "wire_bytes_per_request": sanity["wire_bytes_per_request"],
"cgroup_nr_throttled_delta": cpu_delta["nr_throttled"],
"cgroup_throttled_usec_delta": cpu_delta["throttled_usec"]})
row.update(drops["row_fields"])
row.update(analytics["row_fields"])
for group in ["server", "origin"]:
row.update({f"{group}_{key}": value for key, value in result["resources"][group].items()})
rows.append(row)
persist_rows(args.out, rows)
print(json.dumps({"sequence": item["sequence"], "total": len(schedule), "variant": item["variant"],
"scenario": f"{item['mode']}_{item['size']}_c{item['concurrency']}",
"rps": round(result["rps"], 2), "p95_ms": result["p95_us"] / 1000,
"errors": result["error_total"]}), flush=True)
if result["error_total"]:
raise RuntimeError(f"benchmark errors in {stem}; raw evidence retained")
if not after["passed"] or not sanity["passed"]:
raise RuntimeError(f"payload/byte-count validation failed in {stem}; raw evidence retained")
except (Exception, KeyboardInterrupt) as exc:
error = {"error": str(exc) or type(exc).__name__, "type": type(exc).__name__,
"interrupted": isinstance(exc, KeyboardInterrupt), "traceback": traceback.format_exc()}
write_json(args.out / "failure.json", error)
print(json.dumps(error), flush=True)
finally:
persist_rows(args.out, rows)
completed = error is None and (args.prepare_only or len(rows) == expected_samples)
write_json(args.out / "completion.json", {"completed": completed, "samples": len(rows),
"expected_samples": expected_samples,
"simulation": False, "prepare_only": args.prepare_only,
"finished_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())})
seal_results(args.out)
return (130 if error.get("interrupted") else 1) if error else (0 if completed else 1)
if __name__ == "__main__":
sys.exit(main())
scripts/performance/checks_cache.py
Executable
+180
View File
@@ -0,0 +1,180 @@
"""Cache and response-processing paths must survive disabled-path optimization."""
from concurrent.futures import ThreadPoolExecutor
import copy
import gzip
import io
import json
import time
import zlib
from common import http
def run(check, rust, peer):
port = rust.reverse_port
before = peer.count("GET", "/fresh")
results = [http(port, "/fresh") for _ in range(2)]
check("cache.global_off", all(s == 200 and h.get("x-cache") != "HIT" for s, h, b in results)
and peer.count("GET", "/fresh") - before == 2)
config = copy.deepcopy(rust.base)
config["cache"] = {"enabled": True, "expose_status": True, "expose_hits": True,
"expose_cache_status": True, "http_invalidation_allowed_cidrs": ["127.0.0.1/32"]}
config["web_services"][0]["cache_control"] = "public, max-age=60"
rust.apply(config, "cache-enabled")
first = http(port, "/fresh")
second = http(port, "/fresh")
check("cache.miss_then_hit", first[0] == 200 and first[1].get("x-cache") == "MISS"
and second[1].get("x-cache") == "HIT" and first[2] == second[2],
first=first[1].get("x-cache"), second=second[1].get("x-cache"))
results = {}
for language in ["en", "pl", "en", "pl"]:
response = http(port, "/vary", headers={"Accept-Language": language})
previous = results.get(language)
if previous:
check("cache.vary_hit." + language, response[1].get("x-cache") == "HIT" and response[2] == previous[2],
cache=response[1].get("x-cache"))
results[language] = response
check("cache.vary_isolation", results["en"][2] != results["pl"][2])
for path in ["/private", "/no-store", "/cookie"]:
before = peer.count("GET", path)
first, second = http(port, path), http(port, path)
check("cache.nonpublic_bypass." + path, first[0] == second[0] == 200
and first[1].get("x-cache") != "HIT" and second[1].get("x-cache") != "HIT"
and peer.count("GET", path) - before == 2,
first=first[1].get("x-cache"), second=second[1].get("x-cache"))
for name, headers in [("authorization", {"Authorization": "Bearer fixture-value"}),
("cookie", {"Cookie": "application_session=fixture"})]:
before = peer.count("GET", "/fresh")
s, h, b = http(port, "/fresh", headers=headers)
check("cache.request_bypass." + name, s == 200 and h.get("x-cache") != "HIT"
and peer.count("GET", "/fresh") - before == 1, cache=h.get("x-cache"))
before = peer.count("GET", "/fresh")
s, h, b = http(port, "/fresh", "GET", {"Content-Type": "application/octet-stream"}, b"nonempty-get-body")
check("cache.nonempty_get_bypass_and_preserve", s == 200 and h.get("x-cache") != "HIT"
and h.get("x-received-body-bytes") == "17" and peer.count("GET", "/fresh") - before == 1,
cache=h.get("x-cache"), origin_body_bytes=h.get("x-received-body-bytes"))
s, h, b = http(port, "/fresh", "PURGE")
after = http(port, "/fresh")
check("cache.http_purge", s in (200, 202, 204) and after[1].get("x-cache") == "MISS", status=s)
with ThreadPoolExecutor(max_workers=8) as pool:
results = list(pool.map(lambda _: http(port, "/concurrent"), range(8)))
check("cache.concurrent_fill", all(item[0] == 200 for item in results)
and len({item[2] for item in results}) == 1, origin_requests=peer.count("GET", "/concurrent"))
first = http(port, "/revalidate")
time.sleep(1.2)
second = http(port, "/revalidate")
check("cache.expired_validator_revalidation", first[0] == second[0] == 200 and first[2] == second[2]
and peer.count("GET", "/revalidate") >= 2, cache=second[1].get("x-cache"))
first, second = http(rust.static_port), http(rust.static_port)
check("cache.static_hit", first[0] == second[0] == 200 and second[1].get("x-cache") == "HIT"
and first[2] == second[2], cache=second[1].get("x-cache"))
empty_first, empty_second = http(rust.static_port, "/empty.txt"), http(rust.static_port, "/empty.txt")
check("cache.static_empty_hit", empty_first[0] == empty_second[0] == 200
and empty_second[1].get("x-cache") == "HIT" and empty_second[2] == b"",
first=empty_first[1].get("x-cache"), second=empty_second[1].get("x-cache"))
hs, hh, hb = http(rust.static_port, "/download.txt", "HEAD")
gs, gh, gb = http(rust.static_port, "/download.txt")
check("cache.static_head_cannot_fill_empty_get", hs == gs == 200 and hb == b""
and gb == (rust.fixture.site / "download.txt").read_bytes(), head_status=hs, get_status=gs)
s, h, b = http(rust.static_port, headers={"Range": "bytes=0-9"})
check("cache.static_range_survives", s == 206 and b == first[2][:10], status=s)
s, h, b = http(rust.static_port, headers={"If-None-Match": first[1]["etag"]})
check("cache.static_conditional_survives", s == 304 and b == b"", status=s)
config["vhosts"][0]["cache"] = {"enabled": False}
rust.apply(config, "cache-vhost-disabled")
first, second = http(port, "/fresh"), http(port, "/fresh")
check("cache.vhost_off", all(response[1].get("x-cache") != "HIT" for response in [first, second])
and first[2] != second[2])
config["vhosts"][0]["cache"] = {"enabled": True}
config["vhosts"][0]["routes"] = [{"id": "fresh-no-cache", "path_prefix": "/fresh", "cache": False}]
rust.apply(config, "cache-route-disabled")
first, second = http(port, "/fresh"), http(port, "/fresh")
check("cache.route_off", all(response[1].get("x-cache") != "HIT" for response in [first, second])
and first[2] != second[2])
config["cache"]["enabled"] = False
config["vhosts"][0]["routes"] = []
rust.apply(config, "cache-global-disabled-vhost-enabled")
first, second = http(port, "/fresh"), http(port, "/fresh")
check("cache.vhost_cannot_override_global_off", first[2] != second[2]
and second[1].get("x-cache") != "HIT")
def transforms(check, rust, peer):
config = copy.deepcopy(rust.base)
original = (rust.fixture.site / "app.json").read_bytes()
for port, source in [(rust.reverse_port, "proxy"), (rust.static_port, "static")]:
s, h, b = http(port, "/app.json")
check("transform.off." + source, s == 200 and b == original, status=s)
config["transform"] = {"enabled": True, "json": True, "html": True}
for vhost in config["vhosts"]:
vhost["transform"] = {"enabled": True, "json": True, "html": True}
rust.apply(config, "transform-enabled-cache-off")
for port, source in [(rust.reverse_port, "proxy"), (rust.static_port, "static")]:
s, h, b = http(port, "/app.json")
check("transform.on." + source, s == 200 and json.loads(b) == json.loads(original)
and len(b) < len(original), original_bytes=len(original), result_bytes=len(b), status=s)
s, h, b = http(port, "/large.bin")
check("transform.binary_unchanged." + source, s == 200 and b == (rust.fixture.site / "large.bin").read_bytes())
s, h, b = http(port, "/app.json", headers={"Range": "bytes=0-9"})
check("transform.range_exclusion." + source, s == 206 and b == original[:10], status=s)
config["cache"] = {"enabled": True, "expose_status": True}
rust.apply(config, "transform-enabled-cache-on")
first, second = http(rust.reverse_port, "/app.json"), http(rust.reverse_port, "/app.json")
check("transform.cache_combination", second[0] == 200 and second[1].get("x-cache") == "HIT"
and first[2] == second[2] and json.loads(second[2]) == json.loads(original),
cache=second[1].get("x-cache"))
rust.reset()
expected = (rust.fixture.site / "compress.txt").read_bytes()
for port, source in [(rust.reverse_port, "proxy"), (rust.static_port, "static")]:
s, h, b = http(port, "/compress.txt", headers={"Accept-Encoding": "gzip"})
check("compression.gzip." + source, s == 200 and h.get("content-encoding") == "gzip"
and gzip.decompress(b) == expected and len(b) < len(expected), bytes=len(b))
s, h, b = http(port, "/compress.txt", headers={"Accept-Encoding": "gzip;q=0, identity"})
check("compression.identity_qvalue." + source, s == 200 and "content-encoding" not in h and b == expected)
s, h, b = http(port, "/compress.txt", headers={"Accept-Encoding": "gzip", "Range": "bytes=0-9"})
check("compression.range_exclusion." + source, s == 206 and "content-encoding" not in h and b == expected[:10])
s, h, b = http(rust.reverse_port, "/no-transform", headers={"Accept-Encoding": "gzip"})
check("compression.no_transform_exclusion", s == 200 and "content-encoding" not in h)
config = copy.deepcopy(rust.base)
config["vhosts"][0]["compression"] = {"enabled": False}
rust.apply(config, "compression-vhost-disabled")
s, h, b = http(rust.reverse_port, "/compress.txt", headers={"Accept-Encoding": "gzip"})
check("compression.vhost_off", s == 200 and "content-encoding" not in h and b == expected)
config["compression"] = {"gzip": False, "br": False, "zstd": False, "deflate": False}
config["vhosts"][0]["compression"] = {"enabled": True, "gzip": True}
rust.apply(config, "compression-global-disabled")
s, h, b = http(rust.reverse_port, "/compress.txt", headers={"Accept-Encoding": "gzip"})
check("compression.global_codec_ceiling", s == 200 and "content-encoding" not in h and b == expected)
config = copy.deepcopy(rust.base)
config["compression"] = {"gzip": True, "br": True, "zstd": True, "deflate": True}
config["cache"] = {"enabled": True, "expose_status": True}
rust.apply(config, "compression-cache-enabled")
for codec in ["gzip", "deflate", "br", "zstd"]:
decoder = None
if codec == "gzip":
decoder = gzip.decompress
elif codec == "deflate":
decoder = zlib.decompress
elif codec == "br":
try:
import brotli
decoder = brotli.decompress
except ImportError:
pass
else:
try:
import zstandard
decoder = lambda data: zstandard.ZstdDecompressor().stream_reader(io.BytesIO(data)).read()
except ImportError:
pass
if decoder is None:
check("compression.codec." + codec, None, skip_reason="optional Python decoder is not installed")
continue
first = http(rust.reverse_port, "/compress.txt", headers={"Accept-Encoding": codec})
second = http(rust.reverse_port, "/compress.txt", headers={"Accept-Encoding": codec})
check("compression.codec." + codec, second[0] == 200 and second[1].get("content-encoding") == codec
and decoder(first[2]) == decoder(second[2]) == expected, cache=second[1].get("x-cache"))
scripts/performance/checks_policy.py
Executable
+184
View File
@@ -0,0 +1,184 @@
"""Live ACL, authentication, atomic reload and logging regressions."""
import base64
import copy
import json
import re
import time
from common import http, write_json
DENY = {"default": "deny", "rules": []}
ALLOW_LOCAL = {"default": "deny", "rules": [{"network": "127.0.0.1/32", "action": "allow"}]}
def run(check, rust, peer):
port = rust.reverse_port
config = copy.deepcopy(rust.base)
config["host_policies"] = [{"hosts": ["fixture.test"], "acl": DENY}]
rust.apply(config, "policy-global-host")
count = peer.count("GET", "/echo")
s, h, b = http(port, "/echo", headers={"Host": "fixture.test"})
check("acl.global_host_without_vhost_acl", s == 403 and peer.count("GET", "/echo") == count,
status=s, upstream_requests=peer.count("GET", "/echo") - count)
s, h, b = http(port, "/echo", headers={"Host": "localhost"})
check("acl.global_other_host_unchanged", s == 200, status=s)
config = copy.deepcopy(rust.base)
config["vhosts"][0]["acl"] = DENY
rust.apply(config, "policy-vhost-deny")
s, h, b = http(port)
check("acl.vhost_deny", s == 403, status=s)
s, h, b = http(rust.static_port)
check("acl.vhost_isolation", s == 200, status=s)
config["vhosts"][0]["acl"] = ALLOW_LOCAL
rust.apply(config, "policy-vhost-allow")
s, h, b = http(port)
check("acl.hot_enable_allow", s == 200, status=s)
config = copy.deepcopy(rust.base)
config["acl_profiles"] = [{"id": "restricted", "name": "Restricted", "acl": DENY}]
config["vhosts"][0]["acl_profile_id"] = "restricted"
config["vhosts"][0]["acl"] = ALLOW_LOCAL
rust.apply(config, "policy-profile-deny")
s, h, b = http(port)
check("acl.profile_and_vhost_stack", s == 403, status=s)
config = copy.deepcopy(rust.base)
config["vhosts"][0]["routes"] = [{"id": "denied", "path_prefix": "/echo", "acl": DENY}]
rust.apply(config, "policy-location-deny")
count = peer.count("GET", "/echo")
s, h, b = http(port, "/echo")
check("acl.location_before_upstream", s == 403 and peer.count("GET", "/echo") == count, status=s)
s, h, b = http(port)
check("acl.location_other_path_unchanged", s == 200, status=s)
config["vhosts"][0]["acl"] = DENY
config["vhosts"][0]["routes"][0]["acl"] = ALLOW_LOCAL
rust.apply(config, "policy-location-allow-vhost-deny")
s, h, b = http(port, "/echo")
check("acl.location_cannot_override_vhost_deny", s == 403, status=s)
config = copy.deepcopy(rust.base)
config["vhosts"][0]["acl"] = {"default": "deny", "rules": [{"network": "203.0.113.22/32", "action": "allow"}]}
rust.apply(config, "policy-untrusted-forwarded")
s, h, b = http(port, headers={"X-Forwarded-For": "203.0.113.22", "X-Real-IP": "203.0.113.22"})
check("acl.spoofed_forwarded_ip_denied", s == 403, status=s)
config["forwarding"] = {"trusted_proxies": ["127.0.0.1/32"]}
rust.apply(config, "policy-trusted-forwarded")
s, h, b = http(port, headers={"X-Forwarded-For": "203.0.113.22"})
check("acl.trusted_forwarded_ip_allowed", s == 200, status=s)
config = copy.deepcopy(rust.base)
config["vhosts"][0]["rate_limit"] = {"country_access": {"mode": "allow", "countries": ["PL"]}}
config["vhosts"][0]["cloudflare_client_ip"] = True
rust.apply(config, "policy-country")
s, h, b = http(port, headers={"CF-IPCountry": "PL", "CF-Connecting-IP": "203.0.113.22"})
check("acl.untrusted_country_header_cannot_allow", s == 403, status=s)
config = copy.deepcopy(rust.base)
config["vhosts"][0]["method_policy"] = {"mode": "allow", "methods": ["GET", "HEAD"]}
rust.apply(config, "policy-method")
s, h, b = http(port, "/echo", "POST", body=b"denied method")
check("policy.method_allowlist", s in (403, 405), status=s)
s, h, b = http(port)
check("policy.allowed_method_survives", s == 200, status=s)
config = copy.deepcopy(rust.base)
config["vhosts"][0]["rate_limit"] = {"enabled": True, "requests": 2, "window_seconds": 60, "burst": 0}
rust.apply(config, "policy-rate-limit")
statuses = [http(port)[0] for _ in range(3)]
check("policy.rate_limit", statuses[:2] == [200, 200] and statuses[2] == 429, statuses=statuses)
rust.reset()
s, h, b = http(port)
check("policy.hot_disable_restores_request", s == 200, status=s)
# Every route check uses the real admin socket and no bearer/backdoor token.
for path in ["/api/config", "/api/status", "/api/logs", "/api/events", "/api/openapi.json", "/assets/app.js"]:
s, h, b = http(rust.admin_port, path)
check("admin.private." + path, s in (401, 403), status=s)
check("admin.login", rust.login() == 200)
s, h, b = rust.admin("/api/config")
envelope = json.loads(b)
check("admin.authenticated_config", s == 200 and "config" in envelope and "revision" in envelope, status=s)
s, h, b = rust.admin("/api/config", "PUT", {"expected_revision": envelope["revision"], "config": rust.base}, csrf=False)
check("admin.csrf_required", s == 403, status=s)
s, h, b = rust.admin("/api/config", origin="http://foreign.example")
check("admin.foreign_origin_denied", s == 403, status=s)
# Generate the disposable Basic Auth hash through the existing secured API.
s, h, b = rust.admin("/api/basic-auth/hash", "POST", {"username": "fixtureuser", "password": rust.password})
hashed = json.loads(b)
check("basic.hash_api", s == 200 and "password_hash" in hashed, status=s)
config = copy.deepcopy(rust.base)
config["vhosts"][0]["basic_auth"] = {"enabled": True, "require_tls": False,
"users": [{"username": "fixtureuser", "password_hash": hashed["password_hash"]}]}
rust.apply(config, "policy-basic-auth")
s, h, b = http(port, "/echo")
check("basic.challenge", s == 401 and "Basic" in h.get("www-authenticate", ""), status=s)
credentials = base64.b64encode(("fixtureuser:" + rust.password).encode()).decode()
s, h, b = http(port, "/echo", headers={"Authorization": "Basic " + credentials})
check("basic.correct_password_and_strip", s == 200 and "authorization" not in json.loads(b)["headers"], status=s)
s, h, b = http(port, "/echo", headers={"Authorization": "Basic Zml4dHVyZXVzZXI6d3Jvbmc="})
check("basic.wrong_password_denied", s == 401, status=s)
config["vhosts"][0]["acl"] = DENY
rust.apply(config, "policy-acl-before-basic")
s, h, b = http(port, "/echo")
check("basic.acl_precedes_auth", s == 403, status=s)
rust.reset()
config = copy.deepcopy(rust.base)
config["vhosts"][0]["frontend"]["listener_ids"] = ["does-not-exist"]
rust.apply(config, "invalid-config-retains-snapshot", expected_success=False)
s, h, b = http(port)
check("reload.invalid_document_retains_previous_snapshot", s == 200, status=s)
s, h, b = http(rust.static_port)
check("reload.invalid_document_keeps_other_vhost", s == 200, status=s)
config = copy.deepcopy(rust.base)
config["vhosts"][0]["backends"][0]["allow_private"] = False
# Explicit private-address rejection is checked through apply, preserving the
# active configuration if compile-time validation rejects the backend.
rust.apply(config, "ssrf-private-origin-disallowed", expected_success=False)
s, h, b = http(port)
check("ssrf.rejected_origin_does_not_replace_snapshot", s == 200, status=s)
query = "/echo?observability=" + rust.label
s, h, b = http(port, query)
request_id = json.loads(b)["headers"]["x-request-id"]
log = None
for _ in range(20):
ls, lh, lb = rust.admin("/api/logs?vhost=reverse&limit=512")
logs = json.loads(lb)
log = next((entry for entry in logs.get("entries", []) if entry.get("request_id") == request_id), None)
if log:
break
time.sleep(.05)
check("observability.log_fields_and_body_bytes", log is not None and log.get("status") == 200
and log.get("response_bytes") == len(b) and log.get("request_bytes") == 0
and log.get("cache_effective_enabled") is False and log.get("transform_effective_enabled") is False
and bool(re.fullmatch("[0-9a-f]{64}", log.get("request_id", ""))),
recorded_status=log.get("status") if log else None,
recorded_response_bytes=log.get("response_bytes") if log else None, response_bytes=len(b))
if log:
write_json(rust.fixture.results / f"{rust.label}-sample-exchange.json", log)
config = copy.deepcopy(rust.base)
config["diagnostic_headers"] = {"request_id": True}
rust.apply(config, "observability-static-eos")
s, h, b = http(rust.static_port)
static_request_id = h.get("x-request-id")
static_log = None
for _ in range(20):
ls, lh, lb = rust.admin("/api/logs?vhost=static&limit=64")
static_log = next((entry for entry in json.loads(lb).get("entries", [])
if entry.get("request_id") == static_request_id), None)
if static_log:
break
time.sleep(.05)
check("observability.static_complete_after_content_length", s == 200 and static_log is not None
and static_log.get("outcome") == "complete" and static_log.get("response_bytes") == len(b),
outcome=static_log.get("outcome") if static_log else None,
recorded_response_bytes=static_log.get("response_bytes") if static_log else None,
response_bytes=len(b))
if static_log:
write_json(rust.fixture.results / f"{rust.label}-sample-static-exchange.json", static_log)
s, h, b = rust.admin("/api/status")
status = json.loads(b)
write_json(rust.fixture.results / f"{rust.label}-runtime-status.json", status)
check("observability.status_available", s == 200, status=s)
scripts/performance/checks_proxy.py
Executable
+135
View File
@@ -0,0 +1,135 @@
"""Proxy streaming, status, header, request-ID and WebSocket tests."""
import copy
import hashlib
from http.client import HTTPConnection, HTTPResponse
import json
import re
import socket
import time
from common import http
def run(check, rust, peer):
port = rust.reverse_port
for name in ["small.html", "large.bin", "huge.bin"]:
expected = (rust.fixture.site / name).read_bytes()
s, h, b = http(port, "/" + name)
check("proxy.get." + name, s == 200 and b == expected, status=s, bytes=len(b))
s, h, b = http(port, "/" + name, "HEAD")
check("proxy.head." + name, s == 200 and b == b"" and h.get("content-length") == str(len(expected)), status=s)
s, h, b = http(port, "/large.bin", headers={"Range": "bytes=0-127"})
check("proxy.range", s == 206 and b == bytes(range(128)), status=s, bytes=len(b))
s, h, b = http(port)
s, h, b = http(port, headers={"If-None-Match": h["etag"]})
check("proxy.conditional304", s == 304 and b == b"", status=s)
for status in [201, 400, 404, 500, 503]:
s, h, b = http(port, f"/status/{status}")
check(f"proxy.origin_status.{status}", s == status and b == b"origin status payload\n"
and h.get("x-origin-fixture") == "yes", status=s)
s, h, b = http(port, "/redirect")
check("proxy.redirect", s == 302 and h.get("location") == "/small.html", status=s)
payload = bytes(range(256)) * 4096
ids = []
for method in ["POST", "PUT"]:
s, h, b = http(port, "/echo?query=preserved", method,
{"Content-Type": "application/octet-stream", "X-Request-ID": "untrusted-client-id"}, payload)
data = json.loads(b)
request_id = data["headers"].get("x-request-id", "")
ids.append(request_id)
check("proxy.upload." + method, s == 200 and data["method"] == method
and data["path"] == "/echo?query=preserved" and data["body_bytes"] == len(payload)
and data["body_sha256"] == hashlib.sha256(payload).hexdigest(), status=s, bytes=data["body_bytes"])
check("proxy.request_id." + method, bool(re.fullmatch("[0-9a-f]{64}", request_id)),
valid_64_hex=bool(re.fullmatch("[0-9a-f]{64}", request_id)))
check("proxy.request_id_unique", len(set(ids)) == 2)
s, h, b = http(port, "/echo", "GET", {"Content-Type": "application/octet-stream"}, b"nonempty-get-body")
data = json.loads(b)
check("proxy.nonempty_get_body_preserved", s == 200 and data["body_bytes"] == 17
and data["body_sha256"] == hashlib.sha256(b"nonempty-get-body").hexdigest(), status=s)
peer.started_upload.clear()
first, last = b"first streamed chunk" * 64, b"final streamed chunk" * 64
with socket.create_connection(("127.0.0.1", port), timeout=5) as sock:
sock.sendall((f"POST /stream-upload HTTP/1.1\r\nHost: 127.0.0.1:{port}\r\n"
"Transfer-Encoding: chunked\r\nContent-Type: application/octet-stream\r\n\r\n").encode())
sock.sendall(f"{len(first):x}\r\n".encode() + first + b"\r\n")
forwarded_before_end = peer.started_upload.wait(2)
sock.sendall(f"{len(last):x}\r\n".encode() + last + b"\r\n0\r\n\r\n")
response = HTTPResponse(sock)
response.begin()
data = json.loads(response.read())
check("proxy.chunked_streaming_upload", forwarded_before_end and response.status == 200
and data["body_sha256"] == hashlib.sha256(first + last).hexdigest(),
first_chunk_arrived_before_end=forwarded_before_end, status=response.status)
peer.release_stream.clear()
connection = HTTPConnection("127.0.0.1", port, timeout=3)
connection.request("GET", "/stream-download")
started = time.monotonic()
response = connection.getresponse()
first = response.read(5)
first_seconds = time.monotonic() - started
peer.release_stream.set()
last = response.read()
connection.close()
check("proxy.streaming_response", response.status == 200 and first == b"first" and last == b"last"
and first_seconds < 2, first_chunk_seconds=first_seconds)
peer.release_stream.clear()
connection = HTTPConnection("127.0.0.1", port, timeout=3)
connection.request("GET", "/stream-download")
response = connection.getresponse()
response.read(5)
connection.close()
peer.release_stream.set()
s, h, b = http(port)
check("proxy.client_disconnect_survives", s == 200, status=s)
before = peer.count("POST", "/no-replay")
s, h, b = http(port, "/no-replay", "POST", {"Content-Type": "application/octet-stream"}, b"write-once")
after = peer.count("POST", "/no-replay")
check("proxy.failed_post_not_replayed", s in (502, 503, 504) and after - before == 1,
status=s, upstream_writes=after - before)
config = copy.deepcopy(rust.base)
config["headers"] = {"request": [{"op": "set", "name": "X-Fixture-Global", "value": "global"}],
"response": [{"op": "set", "name": "X-Fixture-Global", "value": "response"}]}
config["vhosts"][0]["headers"] = {"request": [{"op": "set", "name": "X-Fixture-Vhost", "value": "vhost"}]}
config["vhosts"][0]["routes"] = [{"id": "echo", "path_prefix": "/echo", "headers": {
"request": [{"op": "set", "name": "X-Fixture-Route", "value": "route"}],
"response": [{"op": "set", "name": "X-Fixture-Route", "value": "response"}]}}]
rust.apply(config, "proxy-headers")
s, h, b = http(port, "/echo", headers={"Connection": "keep-alive, x-hop-fixture", "X-Hop-Fixture": "must-be-stripped"})
observed = json.loads(b)["headers"]
check("proxy.header_scopes", s == 200 and observed.get("x-fixture-global") == "global"
and observed.get("x-fixture-vhost") == "vhost" and observed.get("x-fixture-route") == "route"
and h.get("x-fixture-global") == "response" and h.get("x-fixture-route") == "response", status=s)
check("proxy.hop_by_hop_removed", "x-hop-fixture" not in observed)
s, h, b = http(port, "/echo", headers={"X-Forwarded-For": "203.0.113.22", "X-Real-IP": "203.0.113.22"})
observed = json.loads(b)["headers"]
check("proxy.untrusted_forwarded_ip", observed.get("x-real-ip") == "127.0.0.1"
and "203.0.113.22" not in observed.get("x-forwarded-for", ""),
observed_x_real_ip=observed.get("x-real-ip"), observed_x_forwarded_for=observed.get("x-forwarded-for"))
rust.reset()
with socket.create_connection(("127.0.0.1", port), timeout=4) as sock:
sock.sendall((f"GET /ws HTTP/1.1\r\nHost: 127.0.0.1:{port}\r\nUpgrade: websocket\r\n"
"Connection: Upgrade\r\nSec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==\r\n"
"Sec-WebSocket-Version: 13\r\n\r\n").encode())
buffer = b""
while b"\r\n\r\n" not in buffer:
piece = sock.recv(4096)
if not piece:
break
buffer += piece
upgraded = buffer.startswith(b"HTTP/1.1 101")
echoed = b""
if upgraded:
message, mask = b"hello", b"\x11\x22\x33\x44"
sock.sendall(bytes([0x81, 0x80 | len(message)]) + mask +
bytes(c ^ mask[i % 4] for i, c in enumerate(message)))
frame = sock.recv(2)
if len(frame) == 2:
echoed = sock.recv(frame[1] & 127)
check("proxy.websocket_echo", upgraded and echoed == b"hello", upgraded=upgraded, bytes=len(echoed))
scripts/performance/checks_static.py
Executable
+132
View File
@@ -0,0 +1,132 @@
"""Live static-file regression cases against RustProxy's network listener."""
import copy
from email.parser import BytesParser
from email.policy import default
import gzip
from common import http
def run(check, rust, peer):
port, fixture = rust.static_port, rust.fixture
for name in ["small.html", "large.bin", "huge.bin", "empty.txt"]:
expected = (fixture.site / name).read_bytes()
status, headers, body = http(port, "/" + name)
check(f"static.get.{name}", status == 200 and body == expected,
status=status, bytes=len(body))
head_status, head_headers, head_body = http(port, "/" + name, "HEAD")
check(f"static.head.{name}", head_status == 200 and head_body == b""
and head_headers.get("content-length") == str(len(expected)),
status=head_status, content_length=head_headers.get("content-length"))
status, headers, body = http(port)
tag = headers.get("etag")
modified = headers.get("last-modified")
check("static.validators_present", bool(tag and modified), etag=tag, last_modified=modified)
for label, condition in [("etag", {"If-None-Match": tag}),
("modified", {"If-Modified-Since": modified})]:
s, h, b = http(port, headers=condition)
check("static.conditional304." + label, s == 304 and b == b"", status=s, bytes=len(b))
s, h, b = http(port, headers={"If-None-Match": '"missing"'})
check("static.conditional_miss", s == 200 and b == body, status=s)
data = (fixture.site / "large.bin").read_bytes()
cases = [("prefix", "bytes=0-127", data[:128], "bytes 0-127/65536"),
("suffix", "bytes=-37", data[-37:], "bytes 65499-65535/65536"),
("open", "bytes=65500-", data[65500:], "bytes 65500-65535/65536")]
for name, range_header, expected, content_range in cases:
s, h, b = http(port, "/large.bin", headers={"Range": range_header})
check("static.range." + name, s == 206 and b == expected
and h.get("content-range") == content_range,
status=s, bytes=len(b), content_range=h.get("content-range"))
s, h, b = http(port, "/large.bin", headers={"Range": "bytes=999999-"})
check("static.range.unsatisfiable", s == 416 and h.get("content-range") == "bytes */65536",
status=s, content_range=h.get("content-range"))
s, h, b = http(port, "/large.bin", headers={"Range": "bytes=0-15,100-115"})
mime = BytesParser(policy=default).parsebytes(
b"Content-Type: " + h.get("content-type", "").encode() + b"\r\n\r\n" + b)
parts = list(mime.iter_parts())
check("static.range.multipart", s == 206 and len(parts) == 2
and parts[0].get_payload(decode=True) == data[:16]
and parts[1].get_payload(decode=True) == data[100:116], status=s, parts=len(parts))
for name, value, expected_status in [("match", tag, 206), ("mismatch", '"other"', 200)]:
s, h, b = http(port, headers={"Range": "bytes=0-15", "If-Range": value})
expected = body[:16] if expected_status == 206 else body
check("static.if_range." + name, s == expected_status and b == expected,
status=s, bytes=len(b))
s, h, b = http(port, "/directory")
check("static.directory_slash", s in (301, 308) and h.get("location", "").endswith("/directory/"),
status=s, location=h.get("location"))
s, h, b = http(port, "/directory/")
check("static.directory_index", s == 200 and b == b"directory index\n", status=s)
s, h, b = http(port, "/listing/")
check("static.directory_listing_denied", s == 403, status=s)
for path in ["/escape-link.txt", "/inside-link.txt"]:
s, h, b = http(port, path)
check("static.symlink_deny." + path, s == 403 and b"MUST-NOT" not in b, status=s)
for path in ["/%2e%2e/outside-secret.txt", "/%2fetc/passwd", "/%252e%252e/outside-secret.txt",
"/%00", "/..%5coutside-secret.txt"]:
s, h, b = http(port, path)
check("static.invalid_path." + path, s in (400, 403, 404) and b"MUST-NOT" not in b,
status=s)
config = copy.deepcopy(rust.base)
service = config["web_services"][0]
service["symlink_policy"] = "inside_root"
service["directory_listing"] = True
rust.apply(config, "static-inside-root")
s, h, b = http(port, "/inside-link.txt")
check("static.symlink_inside_root_allowed", s == 200 and b == b"download fixture\n", status=s)
s, h, b = http(port, "/escape-link.txt")
check("static.symlink_inside_root_escape_denied", s == 403 and b"MUST-NOT" not in b, status=s)
s, h, b = http(port, "/listing/")
check("static.directory_listing_enabled", s == 200 and b"visible.txt" in b, status=s)
# Alternate a symlink target between valid and outside-root files. The file
# read must stay confined even when a pathname changes between requests.
link = fixture.site / "changing-link.txt"
race_statuses = []
escaped = False
for index in range(12):
if link.is_symlink():
link.unlink()
link.symlink_to(fixture.outside if index % 2 else fixture.site / "download.txt")
s, h, b = http(port, "/changing-link.txt")
race_statuses.append(s)
escaped = escaped or b"MUST-NOT" in b
check("static.changing_symlink_confinement", not escaped and all(s in (200, 403) for s in race_statuses),
statuses=race_statuses, note="Sequential target replacement; not a concurrent rename-race proof")
config = copy.deepcopy(rust.base)
config["web_services"][0]["precompressed"] = {"enabled": True, "gzip": True, "brotli": False}
rust.apply(config, "static-precompressed")
s, h, b = http(port, headers={"Accept-Encoding": "gzip"})
sidecar = (fixture.site / "small.html.gz").read_bytes()
check("static.precompressed_gzip", s == 200 and h.get("content-encoding") == "gzip"
and b == sidecar and gzip.decompress(b) == body, status=s, bytes=len(b))
s, h, b = http(port, headers={"Accept-Encoding": "gzip", "Range": "bytes=0-9"})
check("static.precompressed_range", s == 206 and b == sidecar[:10], status=s, bytes=len(b))
config = copy.deepcopy(rust.base)
config["web_services"][0]["try_files"] = ["$uri", "/small.html"]
config["web_services"][0]["rewrites"] = [{"pattern": "^/old$", "replacement": "/small.html", "status": 302}]
config["web_services"][0]["download_extensions"] = ["txt"]
config["web_services"][0]["mime_types"] = {"bin": "application/x-fixture"}
rust.apply(config, "static-routing")
s, h, b = http(port, "/spa/deep/link")
check("static.try_files_spa", s == 200 and b == body, status=s)
s, h, b = http(port, "/old")
check("static.rewrite_redirect", s == 302 and h.get("location") == "/small.html", status=s)
s, h, b = http(port, "/download.txt")
check("static.download_header", s == 200 and "attachment" in h.get("content-disposition", ""), status=s)
s, h, b = http(port, "/large.bin")
check("static.mime_override", s == 200 and h.get("content-type") == "application/x-fixture", status=s)
config = copy.deepcopy(rust.base)
(fixture.site / ".htaccess").write_text("Header set X-Fixture-Htaccess yes\nDirectoryIndex small.html\n")
config["web_services"][0]["htaccess"] = {"enabled": True, "strict": True}
rust.apply(config, "static-htaccess")
s, h, b = http(port)
check("static.htaccess_header", s == 200 and h.get("x-fixture-htaccess") == "yes", status=s)
s, h, b = http(port, "/.htaccess")
check("static.htaccess_not_served", s in (403, 404) and b"Header set" not in b, status=s)
scripts/performance/common.py
Executable
+448
View File
@@ -0,0 +1,448 @@
#!/usr/bin/env python3
"""Disposable, loopback-only fixtures for testing actual RustProxy processes.
Runtime databases and credentials live in TemporaryDirectory, never in results.
No pre-existing SQLite database, fixed port, or historical /mnt/data path is used.
"""
from __future__ import annotations
import copy
import gzip
import hashlib
from http.client import HTTPConnection, HTTPException
import json
import os
from pathlib import Path
import pwd
import secrets
import shutil
import signal
import socket
import subprocess
import tempfile
import time
def sha256(path: Path) -> str:
result = hashlib.sha256()
with path.open("rb") as stream:
for block in iter(lambda: stream.read(1024 * 1024), b""):
result.update(block)
return result.hexdigest()
def write_json(path: Path, value) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(value, indent=2, sort_keys=True) + "\n")
def seal_results(directory: Path) -> None:
"""Checksums cover every regular evidence file, except the checksum manifest."""
paths = sorted(p for p in directory.rglob("*") if p.is_file()
and p.name != "SHA256SUMS")
(directory / "SHA256SUMS").write_text("".join(
f"{sha256(path)} {path.relative_to(directory)}\n" for path in paths))
def http(port: int, path="/small.html", method="GET", headers=None, body=None,
timeout=10, source_address=None):
connection = HTTPConnection("127.0.0.1", port, timeout=timeout,
source_address=source_address)
try:
connection.request(method, path, body, headers or {})
response = connection.getresponse()
payload = response.read()
return response.status, {k.lower(): v for k, v in response.getheaders()}, payload
finally:
connection.close()
def free_port() -> int:
with socket.socket() as sock:
sock.bind(("127.0.0.1", 0))
return sock.getsockname()[1]
def wait_ready(port: int, process: subprocess.Popen, timeout=30):
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if process.poll() is not None:
raise RuntimeError(f"process {process.pid} exited with {process.returncode}")
try:
http(port, "/", timeout=.5)
return
except (OSError, HTTPException):
time.sleep(.05)
raise RuntimeError(f"loopback listener {port} did not start")
class Fixture:
def __init__(self, results: Path, workers=4, temp_parent: Path | None = None,
server_cpus=None, origin_cpus=None, origin_workers=None):
self.results = results.resolve()
self.results.mkdir(parents=True, exist_ok=True)
# Keep live state outside the evidence tree. Even an interrupted run or
# workspace synchronization must never put databases/keys in results.
self.temp = tempfile.TemporaryDirectory(prefix="rustproxy-perf-", dir=temp_parent)
self.root = Path(self.temp.name)
self.root.chmod(0o755) # nginx workers must traverse the public fixture root
self.site = self.root / "site"
self.site.mkdir(mode=0o755)
self.workers = workers
self.server_cpus = server_cpus
self.origin_cpus = origin_cpus
self.origin_workers = origin_workers or workers
self.children = []
self.handles = []
self.ports = set()
self.origin_port = self.port()
self.make_site()
def port(self):
while True:
port = free_port()
if port not in self.ports:
self.ports.add(port)
return port
def make_site(self):
# Exact, reproducible payload sizes match the original report's cases.
small = b"<!doctype html><html><body>RustProxy benchmark "
small += b"x" * (1040 - len(small) - len(b"</body></html>"))
small += b"</body></html>"
files = {"small.html": small,
"large.bin": bytes(range(256)) * 256,
"huge.bin": bytes(range(256)) * 4096,
"empty.txt": b"",
"app.json": (b'{\n "message": "stable fixture",\n "values": [1, 2, 3]\n}\n'),
"app.css": b"/* fixture comment */\nbody { color: red; padding: 0px; }\n",
"compress.txt": (b"compressible RustProxy text fixture. " * 4096),
"download.txt": b"download fixture\n"}
for name, body in files.items():
(self.site / name).write_bytes(body)
(self.site / "small.html.gz").write_bytes(gzip.compress(small, mtime=0))
(self.site / "directory").mkdir()
(self.site / "directory" / "index.html").write_bytes(b"directory index\n")
(self.site / "listing").mkdir()
(self.site / "listing" / "visible.txt").write_bytes(b"listing fixture\n")
(self.site / "inside-link.txt").symlink_to(self.site / "download.txt")
self.outside = self.root / "outside-secret.txt"
self.outside.write_bytes(b"MUST-NOT-BE-SERVED-OUTSIDE-ROOT\n")
(self.site / "escape-link.txt").symlink_to(self.outside)
write_json(self.results / "payloads.json", {
name: {"bytes": len(body), "sha256": hashlib.sha256(body).hexdigest()}
for name, body in files.items()})
def spawn(self, argv, name, env=None, affinity=None):
log = (self.results / f"{name}.log").open("w")
self.handles.append(log)
if affinity:
argv = ["taskset", "-c", affinity, *argv]
child = subprocess.Popen([str(x) for x in argv], stdout=log,
stderr=subprocess.STDOUT, env=env, start_new_session=True)
self.children.append(child)
return child
@staticmethod
def stop(process):
if process.poll() is None:
os.killpg(process.pid, signal.SIGTERM)
try:
process.wait(timeout=12)
except subprocess.TimeoutExpired:
os.killpg(process.pid, signal.SIGKILL)
process.wait(timeout=3)
def close(self):
for process in reversed(self.children):
self.stop(process)
for handle in self.handles:
handle.close()
self.temp.cleanup()
write_json(self.results / "fixture-cleanup.json", {
"temporary_state_removed": not self.root.exists(),
"state_was_outside_results": not self.root.is_relative_to(self.results)})
if self.root.exists():
raise RuntimeError("temporary runtime state was not removed; do not package the fixture directory")
def __enter__(self):
return self
def __exit__(self, *_):
self.close()
def base_config(self, reverse_port, static_port):
endpoints = []
for label, port in [("reverse", reverse_port), ("static", static_port)]:
endpoints.append({"id": label, "name": label, "scope": "internal",
"enabled": True, "binding": {
"address": f"127.0.0.1:{port}", "tls": False,
"http2": False, "http3": None}})
return {
"cache": {"enabled": False},
"geoip_updates": {"enabled": False},
"listeners": {"endpoints": endpoints},
"web_services": [{"id": "fixture-static", "name": "Fixture static",
"document_root": str(self.site), "index": ["index.html", "small.html"],
"directory_listing": False}],
"vhosts": [{"id": "reverse", "frontend": {
"hosts": ["127.0.0.1", "localhost", "fixture.test"],
"listener_ids": ["reverse"]},
"backends": [{"id": "origin", "url": f"http://127.0.0.1:{self.origin_port}",
"allow_private": True, "preserve_host": False}],
"load_balancing": {"health_check": {"mode": "off"}}, "routes": []},
{"id": "static", "frontend": {
"hosts": ["127.0.0.1", "localhost", "fixture.test"],
"listener_ids": ["static"]},
"source": {"type": "web_server", "web_service_id": "fixture-static"},
"routes": []}]
}
def rust(self, binary: Path, label: str, max_idle=None):
return RustInstance(self, binary.resolve(), label, max_idle=max_idle)
def nginx(self, executable: Path, frontend=True):
"""Use a separate nginx master for the shared origin and comparator."""
executable = executable.resolve()
result = subprocess.run([str(executable), "-V"], capture_output=True, text=True, timeout=10)
(self.results / "nginx-version.txt").write_text(result.stdout + result.stderr)
origin_conf = self.nginx_config("nginx-origin", f"""
server {{ listen 127.0.0.1:{self.origin_port}; server_name _;
root "{self.site}"; location / {{ add_header Cache-Control "public, max-age=60"; }}
}}
""", workers=self.origin_workers)
origin = self.spawn([executable, "-p", self.root, "-c", origin_conf], "nginx-origin",
affinity=self.origin_cpus)
wait_ready(self.origin_port, origin)
result = {"origin": origin, "origin_port": self.origin_port}
if frontend:
reverse_port, static_port = self.port(), self.port()
conf = self.nginx_config("nginx-frontend", f"""
upstream fixture_origin {{ server 127.0.0.1:{self.origin_port}; keepalive 64; }}
server {{ listen 127.0.0.1:{reverse_port}; server_name _;
location / {{ proxy_pass http://fixture_origin; proxy_http_version 1.1;
proxy_set_header Connection ""; proxy_buffering off; }}
}}
server {{ listen 127.0.0.1:{static_port}; server_name _; root "{self.site}"; }}
""")
process = self.spawn([executable, "-p", self.root, "-c", conf], "nginx-frontend",
affinity=self.server_cpus)
wait_ready(reverse_port, process)
wait_ready(static_port, process)
result.update({"process": process, "reverse_port": reverse_port,
"static_port": static_port})
return result
def nginx_config(self, name, servers, workers=None):
path = self.root / f"{name}.conf"
text = f"""daemon off;
master_process on;
user {pwd.getpwuid(os.geteuid()).pw_name};
worker_processes {workers or self.workers};
pid {self.root}/{name}.pid;
error_log stderr warn;
events {{ worker_connections 8192; }}
http {{
access_log off;
sendfile on;
tcp_nopush on;
keepalive_timeout 65;
keepalive_requests 1000000;
default_type application/octet-stream;
types {{ text/html html; text/css css; application/json json; text/plain txt; }}
client_body_temp_path {self.root}/{name}-body;
proxy_temp_path {self.root}/{name}-proxy;
{servers}
}}
"""
path.write_text(text)
(self.results / path.name).write_text(text)
return path
class RustInstance:
def __init__(self, fixture, binary, label, max_idle=None):
self.fixture, self.binary, self.label = fixture, binary, label
if not os.access(binary, os.X_OK):
target = fixture.root / f"{label}-rust-proxy"
shutil.copyfile(binary, target)
target.chmod(0o700)
self.binary = target
self.admin_port, self.reverse_port, self.static_port = (fixture.port() for _ in range(3))
self.data = fixture.root / f"{label}-data"
self.base = fixture.base_config(self.reverse_port, self.static_port)
if max_idle is not None:
self.base["vhosts"][0]["load_balancing"]["keepalive"] = {"max_idle_per_host": max_idle}
self.password = "Fixture!" + secrets.token_hex(20)
self.username = "performance"
self.env = os.environ.copy()
self.env["TOKIO_WORKER_THREADS"] = str(fixture.workers)
self.env["RUST_PROXY_PROXY_LISTEN"] = f"127.0.0.1:{self.reverse_port}"
self.env["RUST_PROXY_UPDATE_CHECK_INTERVAL_SECS"] = "0"
self.env.pop("RUST_PROXY_TLS_LISTEN", None)
self.applies = []
self.cli("admin", "init", "--username", self.username, "--password-stdin",
input=self.password + "\n")
self.start("bootstrap")
self.apply(self.base, "base")
fixture.stop(self.process)
self.start("runtime")
wait_ready(self.static_port, self.process)
version = self.cli("--version").stdout.strip()
write_json(fixture.results / f"{label}-binary.json", {
"label": label, "path": str(binary), "sha256": sha256(binary),
"version": version, "worker_threads": fixture.workers,
"pid": self.process.pid, "protocol": "HTTP/1.1 plaintext loopback",
"cpu_affinity": fixture.server_cpus,
"max_idle_per_host_override": max_idle})
def cli(self, *args, input=None, check=True):
result = subprocess.run([str(self.binary), "--data-dir", str(self.data),
"--admin-listen", f"127.0.0.1:{self.admin_port}", *args],
input=input, text=True, capture_output=True,
env=self.env, timeout=40)
if check and result.returncode:
raise RuntimeError(f"{self.label} CLI {' '.join(args[:2])}: "
f"{result.stdout[-2000:]} {result.stderr[-3000:]}")
return result
def start(self, phase):
self.process = self.fixture.spawn([
self.binary, "--data-dir", self.data, "--admin-listen", f"127.0.0.1:{self.admin_port}",
"--update-check-interval-secs", "0", "serve", "--proxy-listen",
f"127.0.0.1:{self.reverse_port}", "--max-connections", "4096", "--max-inflight", "4096"],
f"{self.label}-{phase}", env=self.env, affinity=self.fixture.server_cpus)
wait_ready(self.admin_port, self.process)
wait_ready(self.reverse_port, self.process)
def apply(self, config, name, expected_success=True):
config_path = self.fixture.root / f"{self.label}-{name}.json"
write_json(config_path, config)
result = self.cli("config", "apply", str(config_path), check=False)
# Configurations with ephemeral test passwords/hashes are intentionally not copied.
safe = copy.deepcopy(config)
for vhost in safe.get("vhosts", []):
if "basic_auth" in vhost:
vhost["basic_auth"]["users"] = "REDACTED fixture users"
write_json(self.fixture.results / "configs" / f"{self.label}-{name}.json", safe)
event = {"name": name, "sha256": sha256(config_path), "returncode": result.returncode,
"expected_success": expected_success, "stdout": result.stdout[-2000:],
"stderr": result.stderr[-3000:]}
self.applies.append(event)
write_json(self.fixture.results / f"{self.label}-config-applies.json", self.applies)
if (result.returncode == 0) != expected_success:
raise RuntimeError(f"config {name}: {result.stdout} {result.stderr}")
return result
def reset(self):
return self.apply(self.base, "restored-base")
def login(self):
status, headers, body = http(self.admin_port, "/api/auth/login", "POST",
{"Origin": f"http://127.0.0.1:{self.admin_port}", "Content-Type": "application/json",
"X-RustProxy-Request": "1"},
json.dumps({"username": self.username, "password": self.password}))
if status != 200:
raise RuntimeError(f"fixture login failed: HTTP {status}: {body[:200]!r}")
self.cookie = headers["set-cookie"].split(";", 1)[0]
self.csrf = json.loads(body)["csrf_token"]
return status
def wait_quiescent(self, timeout=180, quiet_seconds=5, poll_seconds=.5):
"""Wait for the real, one-time ACME bootstrap before any timed work.
This does not disable ACME or change the process environment. The
private fixture has no issuers/jobs; a terminal unavailable state is
valid for HTTP tests, but does not establish ACME issuance support.
"""
from resources import descendants, process_stats
if timeout <= 0 or quiet_seconds <= 0 or poll_seconds <= 0 or timeout <= quiet_seconds:
raise ValueError("quiescence timeout must exceed the positive quiet period")
started = time.monotonic()
deadline = started + timeout
output = self.fixture.results / f"{self.label}-quiescence.json"
report = {"completed": False, "label": self.label, "pid": self.process.pid,
"timeout_seconds": timeout, "required_quiet_seconds": quiet_seconds,
"poll_seconds": poll_seconds, "observations": [],
"acme_issuance_tested": False}
def read_admin(path):
remaining = deadline - time.monotonic()
if remaining <= 0:
raise TimeoutError(f"{self.label} did not become quiescent within {timeout}s")
if self.process.poll() is not None:
raise RuntimeError(f"{self.label} exited during quiescence")
status, _, body = self.admin(path, timeout=min(1, remaining))
if status != 200:
raise RuntimeError(f"{self.label} quiescence {path}: HTTP {status}")
return json.loads(body)
try:
root = process_stats(self.process.pid)
if root is None:
raise RuntimeError(f"{self.label} has no live root process")
identity = (root["pid"], root["start_ticks"])
report["start_ticks"] = root["start_ticks"]
envelope = read_admin("/api/config")
if not isinstance(envelope, dict) or not isinstance(envelope.get("config"), dict):
raise RuntimeError("quiescence requires the actual configuration envelope")
certificates = envelope["config"].get("tls", {}).get("certificates", [])
if not isinstance(certificates, list) or any(not isinstance(cert, dict) for cert in certificates):
raise RuntimeError("unexpected certificate configuration during quiescence")
report["configured_issuers"] = sum(cert.get("issuer") is not None for cert in certificates)
if report["configured_issuers"]:
raise RuntimeError("performance fixture must have no configured ACME issuers")
quiet_started, previous_terminal = None, None
while True:
payload = read_admin("/api/acme")
if not isinstance(payload, dict) or not {"runtime", "job", "renewals"} <= payload.keys():
raise RuntimeError("quiescence requires explicit ACME runtime/job/renewal state")
if payload["job"] is not None or payload["renewals"] != {}:
raise RuntimeError("performance fixture has an unexpected ACME job or renewal")
runtime = payload["runtime"]
if not isinstance(runtime, dict) or runtime.get("status") not in {
"checking", "initializing", "ready", "unavailable"}:
raise RuntimeError("unexpected ACME runtime state during quiescence")
pids = descendants(self.process.pid)
root = process_stats(self.process.pid)
if root is None or (root["pid"], root["start_ticks"]) != identity:
raise RuntimeError(f"{self.label} root process identity changed during quiescence")
now = time.monotonic()
if now >= deadline:
raise TimeoutError(f"{self.label} did not become quiescent within {timeout}s")
terminal = tuple(runtime.get(key) for key in ("status", "mode", "message"))
quiet = runtime["status"] in {"ready", "unavailable"} and pids == [self.process.pid]
if not quiet:
quiet_started, previous_terminal = None, None
elif quiet_started is None or terminal != previous_terminal:
quiet_started, previous_terminal = now, terminal
elapsed_quiet = now - quiet_started if quiet_started is not None else 0
report["observations"].append({
"elapsed_seconds": now - started, "process_count": len(pids),
"acme_status": runtime["status"], "acme_mode": runtime.get("mode"),
"acme_message": runtime.get("message"), "quiet_seconds": elapsed_quiet})
if elapsed_quiet >= quiet_seconds:
report.update({"completed": True, "elapsed_seconds": now - started,
"terminal_status": runtime["status"], "process_count": len(pids)})
write_json(output, report)
return report
write_json(output, report)
time.sleep(min(poll_seconds, max(0, deadline - time.monotonic())))
except (Exception, KeyboardInterrupt) as error:
report.update({"elapsed_seconds": time.monotonic() - started,
"error_type": type(error).__name__, "error": str(error)})
write_json(output, report)
raise
def admin(self, path, method="GET", body=None, csrf=True, cookie=True, origin=None,
timeout=10):
headers = {"Origin": origin or f"http://127.0.0.1:{self.admin_port}", "X-RustProxy-Request": "1"}
if cookie:
headers["Cookie"] = self.cookie
if csrf:
headers["X-CSRF-Token"] = self.csrf
if body is not None:
headers["Content-Type"] = "application/json"
body = json.dumps(body)
return http(self.admin_port, path, method, headers, body, timeout=timeout)
scripts/performance/concurrency_regression.py
Executable
+138
View File
@@ -0,0 +1,138 @@
#!/usr/bin/env python3
"""Finite live HTTP concurrency correctness stress; no throughput/latency claim."""
from __future__ import annotations
import argparse
from concurrent.futures import ThreadPoolExecutor
import hashlib
import json
from pathlib import Path
import threading
import time
import traceback
from common import Fixture, http, seal_results, sha256, write_json
from resources import process_stats
PAYLOADS = ["small.html", "large.bin", "huge.bin"]
def active_backends(rust):
status, _, body = rust.admin("/api/status")
if status != 200:
return {"error": f"HTTP {status}"}
values = []
def walk(value):
if isinstance(value, dict):
if "id" in value and "active" in value and "status" in value and "requests" in value:
values.append({key: value[key] for key in ("id", "active", "status", "requests")})
for child in value.values():
walk(child)
elif isinstance(value, list):
for child in value:
walk(child)
walk(json.loads(body))
return values
def batch(rust, port, name, expected, concurrency):
count = max(32, concurrency * 2)
expected_sha = hashlib.sha256(expected).hexdigest()
ready = threading.Barrier(concurrency, timeout=15)
lock = threading.Lock()
running = peak = 0
def fetch(index):
nonlocal running, peak
with lock:
running += 1
peak = max(peak, running)
try:
if index < concurrency:
ready.wait()
status, headers, body = http(port, "/" + name, timeout=15)
digest = hashlib.sha256(body).hexdigest()
return {"index": index, "status": status, "bytes": len(body), "sha256": digest,
"passed": status == 200 and len(body) == len(expected) and digest == expected_sha}
except Exception as error:
return {"index": index, "passed": False, "error": str(error)}
finally:
with lock:
running -= 1
before = process_stats(rust.process.pid)
started = time.monotonic()
with ThreadPoolExecutor(max_workers=concurrency) as executor:
responses = list(executor.map(fetch, range(count)))
elapsed = time.monotonic() - started
# Completion means every client consumed or failed its full response. Give
# asynchronous server bookkeeping a bounded interval to release its leases.
for _ in range(40):
active = active_backends(rust)
if isinstance(active, list) and active and all(row["active"] == 0 for row in active):
break
time.sleep(.05)
after = process_stats(rust.process.pid)
return {"configured_concurrent_clients": concurrency, "peak_running_client_tasks": peak,
"client_tasks_include_barrier": True, "requests": count, "elapsed_seconds": elapsed,
"expected_bytes": len(expected), "expected_sha256": expected_sha,
"response_failures": sum(not response["passed"] for response in responses),
"passed": all(response["passed"] for response in responses)
and isinstance(active, list) and bool(active) and all(row["active"] == 0 for row in active),
"responses": responses, "active_backends_after_drain": active,
"server_before": before, "server_after_drain": after}
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--binary", action="append", required=True, metavar="LABEL=/ABS/PATH")
parser.add_argument("--nginx", type=Path, required=True)
parser.add_argument("--out", type=Path, required=True)
args = parser.parse_args()
if args.out.exists() and any(args.out.iterdir()):
parser.error("--out must be new or empty")
reports = {}
for specification in args.binary:
label, binary = specification.split("=", 1)
cases, error = [], None
try:
with Fixture(args.out / label, workers=4, origin_workers=2) as fixture:
fixture.nginx(args.nginx, frontend=False)
rust = fixture.rust(Path(binary), label)
rust.login()
for source, port in [("proxy", rust.reverse_port), ("static", rust.static_port)]:
for name in PAYLOADS:
expected = (fixture.site / name).read_bytes()
for concurrency in [1, 16, 128]:
result = batch(rust, port, name, expected, concurrency)
case = {"case": f"concurrency.{source}.{name}.c{concurrency}", **result}
cases.append(case)
write_json(args.out / label / "cases" / f"{source}-{name}-c{concurrency}.json", case)
print(f"{label} {'PASS' if case['passed'] else 'FAIL'} {case['case']}", flush=True)
if rust.process.poll() is not None:
raise RuntimeError("RustProxy exited during concurrency regression")
except Exception as exc:
error = {"error": str(exc), "traceback": traceback.format_exc()}
write_json(args.out / label / "failure.json", error)
report = {"kind": "live HTTP/1.1 finite concurrency functional stress", "simulation": False,
"binary_sha256": sha256(Path(binary)), "throughput_measurement": False,
"passed_cases": sum(case["passed"] for case in cases),
"failed_cases": sum(not case["passed"] for case in cases),
"requests": sum(case["requests"] for case in cases),
"response_failures": sum(case["response_failures"] for case in cases),
"completed": error is None and len(cases) == 18,
"cases": [{key: case[key] for key in ("case", "passed", "requests", "response_failures")} for case in cases],
"limitations": ["configured client concurrency is not proof of simultaneous active server exchanges",
"Python client is not a throughput or latency benchmark",
"FD/RSS are before/after snapshots, not peaks or a long-term leak test"]}
reports[label] = report
write_json(args.out / label / "report.json", report)
write_json(args.out / "comparison.json", reports)
seal_results(args.out)
return 0 if all(report["completed"] and not report["failed_cases"] for report in reports.values()) else 1
if __name__ == "__main__":
raise SystemExit(main())
scripts/performance/lb_regression.py
Executable
+223
View File
@@ -0,0 +1,223 @@
#!/usr/bin/env python3
"""Small live two-origin failover and active-exchange regression, not a benchmark."""
from __future__ import annotations
import argparse
from collections import Counter
import copy
import hashlib
from http.client import HTTPConnection
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
from pathlib import Path
import threading
import time
import traceback
from common import Fixture, http, seal_results, write_json
class Origin:
def __init__(self, port, name):
self.name, self.port = name, port
self.fail_get = False
self.counts, self.post_hashes = Counter(), []
self.release, self.lock = threading.Event(), threading.Lock()
origin = self
class Handler(BaseHTTPRequestHandler):
protocol_version = "HTTP/1.1"
def log_message(self, *_):
pass
def do_GET(self):
self.respond()
def do_POST(self):
self.respond()
def respond(self):
body = self.rfile.read(int(self.headers.get("Content-Length", "0")))
with origin.lock:
origin.counts[(self.command, self.path)] += 1
if self.command == "POST":
origin.post_hashes.append(hashlib.sha256(body).hexdigest())
hold = self.path == "/hold"
status = 503 if self.command == "POST" or origin.fail_get else 200
payload = b"firstlast" if hold else origin.name.encode()
try:
self.send_response(status)
self.send_header("Content-Length", str(len(payload)))
self.send_header("Content-Type", "application/octet-stream")
self.send_header("Cache-Control", "no-store")
# Fresh connections make refused-listener tests independent
# of already accepted upstream keepalive connections.
self.send_header("Connection", "close")
self.end_headers()
if hold:
self.wfile.write(payload[:5])
self.wfile.flush()
origin.release.wait(15)
self.wfile.write(payload[5:])
else:
self.wfile.write(payload)
self.wfile.flush()
except (BrokenPipeError, ConnectionResetError):
pass
self.close_connection = True
self.server = ThreadingHTTPServer(("127.0.0.1", port), Handler)
self.thread = threading.Thread(target=self.server.serve_forever, daemon=True)
self.thread.start()
self.closed = False
def close(self):
self.release.set()
if not self.closed:
self.server.shutdown()
self.server.server_close()
self.thread.join(timeout=3)
self.closed = True
def backend_views(rust):
status, _, body = rust.admin("/api/status")
if status != 200:
raise RuntimeError(f"status unavailable: {status}")
found = {}
def walk(value):
if isinstance(value, dict):
if value.get("id") in ("a", "b") and "active" in value and "status" in value:
found[value["id"]] = {key: value[key] for key in value
if key in ("id", "status", "active", "requests", "errors", "bytes_out")
or key.startswith("retry_") or key.startswith("connect_errors")}
for child in value.values():
walk(child)
elif isinstance(value, list):
for child in value:
walk(child)
walk(json.loads(body))
return found
def run(check, fixture, binary, label):
a, b = Origin(fixture.origin_port, "a"), Origin(fixture.port(), "b")
try:
rust = fixture.rust(binary, label)
rust.login()
config = copy.deepcopy(rust.base)
vhost = config["vhosts"][0]
vhost["backends"] = [{"id": origin.name, "url": f"http://127.0.0.1:{origin.port}",
"allow_private": True, "preserve_host": False} for origin in [a, b]]
vhost["load_balancing"] = {"algorithm": "round_robin", "health_check": {"mode": "off"},
"slow_start_seconds": 0, "passive_health": {"enabled": False},
"retry": {"max_retries": 1, "retry_statuses": [503]}}
vhost["routes"] = [{"id": "hold-a", "path_prefix": "/hold", "backend_id": "a"}]
rust.apply(config, "lb-two-origins")
replies = [http(rust.reverse_port, "/rr") for _ in range(8)]
counts = Counter(body.decode() for status, _, body in replies if status == 200)
check("lb.round_robin_both_origins", counts == {"a": 4, "b": 4}, counts=dict(counts))
a.fail_get = True
replies = [http(rust.reverse_port, "/retry-status") for _ in range(6)]
check("lb.get_503_failover", all(status == 200 and body == b"b" for status, _, body in replies)
and a.counts[("GET", "/retry-status")] > 0,
failed_origin_attempts=a.counts[("GET", "/retry-status")])
views = backend_views(rust)
check("lb.retry_observations_recorded", sum(view.get("retry_attempts_total", 0) for view in views.values()) > 0,
backends=views)
a.fail_get = False
payload = b"one POST must be attempted once"
replies = [http(rust.reverse_port, "/post-no-replay", "POST", body=payload) for _ in range(4)]
attempts = sum(origin.counts[("POST", "/post-no-replay")] for origin in [a, b])
check("lb.post_503_never_replayed", all(status == 503 for status, _, _ in replies) and attempts == 4,
client_requests=4, origin_attempts=attempts)
check("lb.post_body_preserved", len(a.post_hashes + b.post_hashes) == 4 and
all(digest == hashlib.sha256(payload).hexdigest() for digest in a.post_hashes + b.post_hashes))
for mode in ["drain", "disable"]:
vhost["backends"][0].update({"enabled": True, "drain": False})
rust.apply(config, f"lb-{mode}-before")
a.release.clear()
connection = HTTPConnection("127.0.0.1", rust.reverse_port, timeout=10)
try:
connection.request("GET", "/hold")
response = connection.getresponse()
first = response.read(5)
before = backend_views(rust)
check(f"lb.{mode}.lease_spans_response_body", response.status == 200 and first == b"first"
and before.get("a", {}).get("active") == 1, backends=before)
vhost["backends"][0].update({"drain": mode == "drain", "enabled": mode != "disable"})
rust.apply(config, f"lb-{mode}-active")
replies = [http(rust.reverse_port, "/new") for _ in range(4)]
check(f"lb.{mode}.new_requests_use_healthy_member",
all(status == 200 and body == b"b" for status, _, body in replies))
during = backend_views(rust)
check(f"lb.{mode}.active_lease_survives_hot_edit", during.get("a", {}).get("active") == 1,
backends=during)
a.release.set()
last = response.read()
check(f"lb.{mode}.admitted_body_finishes", first + last == b"firstlast")
for _ in range(20):
after = backend_views(rust)
if after.get("a", {}).get("active") == 0:
break
time.sleep(.05)
check(f"lb.{mode}.lease_released_after_body", after.get("a", {}).get("active") == 0,
backends=after)
finally:
a.release.set()
connection.close()
vhost["backends"][0].update({"enabled": True, "drain": False})
rust.apply(config, "lb-connect-refused")
a.close()
replies = [http(rust.reverse_port, "/retry-connect") for _ in range(6)]
views = backend_views(rust)
check("lb.connect_refused_failover", all(status == 200 and body == b"b" for status, _, body in replies)
and views.get("a", {}).get("connect_errors_total", 0) > 0, backends=views)
rust.reset()
check("lb.static_vhost_survives_reload", http(rust.static_port)[0] == 200)
finally:
a.close()
b.close()
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--binary", action="append", required=True, metavar="LABEL=/ABS/PATH")
parser.add_argument("--out", type=Path, required=True)
args = parser.parse_args()
if args.out.exists() and any(args.out.iterdir()):
parser.error("--out must be new or empty")
reports = {}
for specification in args.binary:
label, binary = specification.split("=", 1)
cases = []
def check(case, passed, **details):
cases.append({"case": case, "passed": passed, **details})
print(f"{label} {'PASS' if passed else 'FAIL'} {case}", flush=True)
write_json(args.out / label / "checks.json", cases)
try:
with Fixture(args.out / label) as fixture:
run(check, fixture, Path(binary), label)
except Exception as exc:
check("lb.runner", False, error=str(exc), traceback=traceback.format_exc())
report = {"passed": sum(case["passed"] for case in cases),
"failed": sum(not case["passed"] for case in cases), "cases": cases,
"simulation": False, "kind": "live RustProxy two-origin HTTP/1.1 regression",
"not_covered": ["sticky affinity", "active health probes", "long-running connection soak",
"TLS origin failures", "DNS rotation"]}
reports[label] = report
write_json(args.out / label / "report.json", report)
write_json(args.out / "comparison.json", reports)
seal_results(args.out)
return 1 if any(report["failed"] for report in reports.values()) else 0
if __name__ == "__main__":
raise SystemExit(main())
scripts/performance/peer.py
Executable
+223
View File
@@ -0,0 +1,223 @@
#!/usr/bin/env python3
"""A protocol peer, not a RustProxy emulator; used only as the test upstream.
Performance comparisons use nginx as the upstream, never this Python fixture.
"""
from __future__ import annotations
import base64
import collections
import hashlib
import json
import socket
import struct
import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import urlsplit, parse_qs
class Peer:
def __init__(self, port, site):
self.site = site
self.counts = collections.Counter()
self.received = {}
self.lock = threading.Lock()
self.started_upload = threading.Event()
self.release_stream = threading.Event()
peer = self
class Handler(BaseHTTPRequestHandler):
protocol_version = "HTTP/1.1"
def log_message(self, *_):
pass
def emit(self, status, body=b"", headers=None):
self.send_response(status)
self.send_header("Content-Length", str(len(body)))
if "Content-Type" not in (headers or {}):
self.send_header("Content-Type", "application/octet-stream")
for key, value in (headers or {}).items():
self.send_header(key, value)
self.end_headers()
if self.command != "HEAD" and body:
self.wfile.write(body)
def upload(self):
chunks = []
if self.headers.get("Transfer-Encoding", "").lower() == "chunked":
while True:
length = int(self.rfile.readline().split(b";", 1)[0], 16)
if length == 0:
while self.rfile.readline().strip():
pass
break
payload = self.rfile.read(length)
self.rfile.read(2)
chunks.append(payload)
peer.started_upload.set()
else:
remaining = int(self.headers.get("Content-Length", 0))
while remaining:
payload = self.rfile.read(min(remaining, 1024))
if not payload:
break
chunks.append(payload)
remaining -= len(payload)
peer.started_upload.set()
return b"".join(chunks)
def do_POST(self):
self.handle_method()
def do_PUT(self):
self.handle_method()
def do_HEAD(self):
self.handle_method()
def do_GET(self):
self.handle_method()
def handle_method(self):
try:
self.route()
except (BrokenPipeError, ConnectionResetError, socket.timeout):
self.close_connection = True
def route(self):
parsed = urlsplit(self.path)
path, query = parsed.path, parse_qs(parsed.query)
with peer.lock:
peer.counts[(self.command, path)] += 1
count = peer.counts[(self.command, path)]
body = self.upload() if self.command in ("POST", "PUT") or self.headers.get("Content-Length") or self.headers.get("Transfer-Encoding") else b""
if path == "/no-replay":
self.close_connection = True
self.connection.shutdown(socket.SHUT_RDWR)
self.connection.close()
return
if path in ("/echo", "/stream-upload"):
data = {"method": self.command, "path": self.path,
"headers": {key.lower(): value for key, value in self.headers.items()},
"body_bytes": len(body), "body_sha256": hashlib.sha256(body).hexdigest()}
with peer.lock:
peer.received[path] = data
self.emit(200, json.dumps(data).encode(), {"Content-Type": "application/json",
"Cache-Control": "no-store", "X-Origin-Fixture": "yes"})
return
if path == "/stream-download":
self.send_response(200)
self.send_header("Transfer-Encoding", "chunked")
self.send_header("Content-Type", "application/octet-stream")
self.send_header("Cache-Control", "no-store")
self.end_headers()
self.wfile.write(b"5\r\nfirst\r\n")
self.wfile.flush()
peer.release_stream.wait(5)
self.wfile.write(b"4\r\nlast\r\n0\r\n\r\n")
self.wfile.flush()
return
if path == "/ws":
key = self.headers.get("Sec-WebSocket-Key", "")
accept = base64.b64encode(hashlib.sha1(
(key + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11").encode()).digest()).decode()
self.send_response(101)
self.send_header("Upgrade", "websocket")
self.send_header("Connection", "Upgrade")
self.send_header("Sec-WebSocket-Accept", accept)
self.end_headers()
self.wfile.flush()
frame = self.rfile.read(2)
if len(frame) == 2:
length = frame[1] & 127
if length == 126:
length = struct.unpack("!H", self.rfile.read(2))[0]
mask = self.rfile.read(4) if frame[1] & 128 else b"\0\0\0\0"
data = self.rfile.read(length)
data = bytes(byte ^ mask[i % 4] for i, byte in enumerate(data))
self.wfile.write(bytes([0x81, len(data)]) + data)
self.wfile.flush()
self.close_connection = True
return
if path.startswith("/status/"):
self.emit(int(path.rsplit("/", 1)[1]), b"origin status payload\n",
{"Cache-Control": "no-store", "X-Origin-Fixture": "yes"})
return
if path == "/redirect":
self.emit(302, b"redirect body\n", {"Location": "/small.html", "Cache-Control": "no-store"})
return
if path in ("/private", "/cookie", "/no-store", "/vary", "/fresh", "/revalidate", "/concurrent"):
cache_control = "public, max-age=60"
headers = {"Content-Type": "text/plain"}
if body:
headers["X-Received-Body-Bytes"] = str(len(body))
if path == "/private":
cache_control = "private, max-age=60"
if path == "/no-store":
cache_control = "no-store"
if path == "/cookie":
headers["Set-Cookie"] = "application-session=fixture; Path=/"
if path == "/vary":
headers["Vary"] = "Accept-Language"
if path == "/concurrent":
time.sleep(.08)
if path == "/revalidate":
cache_control = "public, max-age=1, stale-if-error=10"
headers["ETag"] = '"fixture-v1"'
if self.headers.get("If-None-Match") == headers["ETag"]:
self.emit(304, b"", {**headers, "Cache-Control": cache_control})
return
data = f"{path}:count={count}:lang={self.headers.get('Accept-Language', '')}".encode()
if path == "/revalidate":
data = b"stable revalidated object"
self.emit(200, data, {**headers, "Cache-Control": cache_control})
return
if path == "/no-transform":
self.emit(200, b"unchanged compression fixture " * 1024,
{"Content-Type": "text/plain", "Cache-Control": "no-transform, public, max-age=60"})
return
file = peer.site / path.lstrip("/")
if not file.is_file() or ".." in file.parts:
self.emit(404, b"fixture not found")
return
payload = file.read_bytes()
tag = '"' + hashlib.sha256(payload).hexdigest() + '"'
headers = {"Cache-Control": "public, max-age=60", "ETag": tag,
"Accept-Ranges": "bytes", "X-Origin-Fixture": "yes"}
headers["Content-Type"] = {".html": "text/html", ".txt": "text/plain",
".css": "text/css", ".json": "application/json"}.get(file.suffix, "application/octet-stream")
if self.headers.get("If-None-Match") == tag:
self.emit(304, b"", headers)
return
range_value = self.headers.get("Range")
if range_value and self.headers.get("If-Range", tag) == tag:
try:
first, last = range_value.removeprefix("bytes=").split("-", 1)
start = int(first) if first else max(0, len(payload) - int(last))
end = min(int(last), len(payload) - 1) if first and last else len(payload) - 1
if start >= len(payload) or end < start:
self.emit(416, b"", {**headers, "Content-Range": f"bytes */{len(payload)}"})
else:
self.emit(206, payload[start:end + 1], {**headers,
"Content-Range": f"bytes {start}-{end}/{len(payload)}"})
return
except ValueError:
pass
self.emit(200, payload, headers)
self.server = ThreadingHTTPServer(("127.0.0.1", port), Handler)
self.server.daemon_threads = True
self.thread = threading.Thread(target=self.server.serve_forever, daemon=True)
self.thread.start()
def count(self, method, path):
with self.lock:
return self.counts[(method, path)]
def close(self):
self.release_stream.set()
self.server.shutdown()
self.server.server_close()
self.thread.join(timeout=2)
scripts/performance/profile_syscalls.py
Executable
+284
View File
@@ -0,0 +1,284 @@
#!/usr/bin/env python3
"""Profile selected server syscalls separately from throughput measurements.
Runs each RustProxy binary sequentially against a disposable loopback fixture.
The measured operation is one persistent HTTP/1.1 connection requesting the
64-KiB static file with cache disabled and Accept-Encoding: identity.
Every response is validated; syscall counts include background server work.
strace elapsed/CPU times must not be interpreted as uninstrumented performance.
"""
from __future__ import annotations
import argparse
import hashlib
from http.client import HTTPConnection
import json
import os
from pathlib import Path
import platform
import re
import shutil
import signal
import subprocess
import sys
import time
import traceback
from common import Fixture, seal_results, sha256, write_json
from resources import descendants
SYSCALLS = ("read", "pread64", "write", "writev", "openat", "close", "fcntl", "lseek", "futex")
REQUEST_PATH = "/large.bin"
def thread_tracers(pid: int) -> dict[int, int]:
result = {}
for task in Path(f"/proc/{pid}/task").iterdir():
try:
status = (task / "status").read_text()
except FileNotFoundError:
continue # A thread may exit between the directory read and status.
match = re.search(r"^TracerPid:\s+(\d+)$", status, re.MULTILINE)
if match:
result[int(task.name)] = int(match.group(1))
return result
def wait_attached(tracer, server, timeout=10):
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if tracer.poll() is not None:
raise RuntimeError("strace exited before attachment; see strace diagnostic log (ptrace may be unavailable)")
if server.poll() is not None:
raise RuntimeError("server exited during strace attachment")
traced = thread_tracers(server.pid)
if traced and all(traced.values()):
return sorted(traced)
time.sleep(.01)
raise RuntimeError("strace did not attach to every server thread within 10 seconds")
def detach(tracer, server):
# Do not signal the process group: only the independently started tracer.
if tracer.poll() is None:
tracer.send_signal(signal.SIGINT)
try:
tracer.wait(timeout=10)
except subprocess.TimeoutExpired:
tracer.terminate()
try:
tracer.wait(timeout=3)
except subprocess.TimeoutExpired:
tracer.kill()
tracer.wait(timeout=3)
raise RuntimeError("strace failed to finish on SIGINT; profile is incomplete")
if server.poll() is not None:
raise RuntimeError("server exited while the tracer was detached")
traced = thread_tracers(server.pid)
if not traced or any(traced.values()):
raise RuntimeError("server still has traced threads after strace exited")
return {"signal": "SIGINT to strace PID only", "strace_returncode": tracer.returncode,
"server_alive": True, "remaining_traced_threads": 0}
def parse_summary(path: Path, requests: int):
rows = {name: {"calls": 0, "errors": 0, "calls_per_request": 0.0}
for name in SYSCALLS}
total = None
for line in path.read_text().splitlines():
fields = line.split()
if len(fields) not in (5, 6) or fields[-1] not in (*SYSCALLS, "total"):
continue
percent, seconds, usecs, calls = map(float, fields[:4])
errors = int(fields[4]) if len(fields) == 6 else 0
row = {"calls": int(calls), "errors": errors,
"calls_per_request": int(calls) / requests,
"strace_system_seconds": seconds, "strace_percent_time": percent,
"strace_us_per_call": usecs}
if fields[-1] == "total":
total = row
else:
rows[fields[-1]] = row
if total is None:
raise RuntimeError(f"no complete strace summary found in {path}")
if sum(row["calls"] for row in rows.values()) != total["calls"]:
raise RuntimeError("parsed syscall counts do not match the raw strace total")
return {"selected_syscalls": rows, "selected_total": total}
def request_checked(connection, expected_socket, expected_sha, phase, sequence, evidence):
if connection.sock is not expected_socket:
raise RuntimeError("keepalive connection was replaced before the next request")
connection.request("GET", REQUEST_PATH,
headers={"Accept-Encoding": "identity", "Connection": "keep-alive"})
response = connection.getresponse()
payload = response.read()
digest = hashlib.sha256(payload).hexdigest()
headers = {key.lower(): value for key, value in response.getheaders()}
valid = (response.status == 200 and len(payload) == 65536 and digest == expected_sha
and headers.get("content-length") == "65536"
and headers.get("content-encoding", "identity") == "identity"
and connection.sock is expected_socket and not response.will_close)
result = {"phase": phase, "sequence": sequence, "status": response.status,
"bytes": len(payload), "sha256": digest,
"content_length": headers.get("content-length"),
"content_encoding": headers.get("content-encoding"),
"connection_reused": connection.sock is expected_socket,
"will_close": response.will_close, "passed": valid}
evidence.write(json.dumps(result, sort_keys=True) + "\n")
evidence.flush()
if not valid:
raise RuntimeError(f"response verification failed in {phase} request {sequence}: {result}")
def profile_variant(fixture, label, binary, args):
rust = fixture.rust(binary, label)
rust.login()
rust.wait_quiescent()
raw = args.out / "raw"
raw.mkdir(exist_ok=True)
summary_path = raw / f"{label}.strace.txt"
diagnostic_path = raw / f"{label}.strace.log"
responses_path = raw / f"{label}.responses.jsonl"
expected_sha = sha256(fixture.site / "large.bin")
quiescence_path = fixture.results / f"{label}-quiescence.json"
config_path = fixture.results / "configs" / f"{label}-base.json"
config_sha = sha256(config_path)
if config_sha != rust.applies[-1]["sha256"]:
raise RuntimeError("saved fixture configuration differs from the applied configuration")
command = [str(args.strace), "-f", "-c", "-e", "trace=" + ",".join(SYSCALLS),
"-p", str(rust.process.pid), "-o", str(summary_path)]
connection = HTTPConnection("127.0.0.1", rust.static_port, timeout=args.timeout)
tracer = None
result = {"variant": label, "binary": str(binary), "binary_sha256": sha256(binary),
"config": str(config_path.relative_to(args.out)), "config_sha256": config_sha,
"quiescence": str(quiescence_path.relative_to(args.out)),
"quiescence_sha256": sha256(quiescence_path),
"pid": rust.process.pid, "request_path": REQUEST_PATH, "payload_bytes": 65536,
"payload_sha256": expected_sha, "cache_enabled": False,
"request_encoding": "identity", "concurrency": 1,
"warmup_requests": args.warmup, "profile_requests": args.requests,
"strace_command": command, "raw_summary": str(summary_path.relative_to(args.out)),
"raw_diagnostics": str(diagnostic_path.relative_to(args.out)),
"responses": str(responses_path.relative_to(args.out)), "passed": False}
try:
with responses_path.open("w") as responses, diagnostic_path.open("w") as diagnostics:
connection.connect()
expected_socket = connection.sock
result["client_local_address"] = expected_socket.getsockname()
for index in range(args.warmup):
request_checked(connection, expected_socket, expected_sha, "warmup", index + 1, responses)
result["processes_before_attach"] = descendants(rust.process.pid)
if result["processes_before_attach"] != [rust.process.pid]:
raise RuntimeError("server spawned a child after quiescence; refusing contaminated syscall trace")
environment = os.environ.copy()
environment["LC_ALL"] = "C"
tracer = subprocess.Popen(command, stdout=diagnostics, stderr=diagnostics,
env=environment, start_new_session=True)
result["initial_traced_threads"] = wait_attached(tracer, rust.process)
started = time.monotonic()
try:
for index in range(args.requests):
request_checked(connection, expected_socket, expected_sha, "profile", index + 1, responses)
if tracer.poll() is not None:
raise RuntimeError("strace exited before all profile requests completed")
finally:
result["instrumented_request_loop_seconds"] = time.monotonic() - started
result["detach"] = detach(tracer, rust.process)
tracer = None
# An actual HTTP response after detaching proves the server stayed usable.
request_checked(connection, expected_socket, expected_sha, "after_detach", 1, responses)
result["processes_after_detach"] = descendants(rust.process.pid)
if result["processes_after_detach"] != [rust.process.pid]:
raise RuntimeError("server has a child after profiling; syscall trace needs review")
result.update(parse_summary(summary_path, args.requests))
result["verified_responses"] = args.requests + args.warmup + 1
result["passed"] = True
return result
finally:
if tracer is not None:
detach(tracer, rust.process)
connection.close()
fixture.stop(rust.process)
write_json(args.out / f"{label}.json", result)
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--binary", action="append", required=True, metavar="LABEL=/ABS/PATH")
parser.add_argument("--nginx", type=Path, required=True, help="nginx executable for the shared fixture origin")
parser.add_argument("--strace", type=Path, default=shutil.which("strace"))
parser.add_argument("--out", type=Path, required=True)
parser.add_argument("--requests", type=int, default=500)
parser.add_argument("--warmup", type=int, default=20)
parser.add_argument("--workers", type=int, default=4)
parser.add_argument("--origin-workers", type=int, default=4)
parser.add_argument("--server-cpus")
parser.add_argument("--origin-cpus")
parser.add_argument("--timeout", type=float, default=20)
args = parser.parse_args()
if args.requests < 1 or args.warmup < 0 or args.workers < 1 or args.origin_workers < 1 or args.timeout <= 0:
parser.error("requests/workers/timeout must be positive, warmup must be nonnegative")
if args.strace is None:
parser.error("strace is required; pass --strace /absolute/path")
if args.out.exists() and (not args.out.is_dir() or any(args.out.iterdir())):
parser.error("--out must be a new or empty directory; existing evidence is never overwritten")
variants = {}
for value in args.binary:
label, separator, filename = value.partition("=")
if not separator or not re.fullmatch(r"[A-Za-z][A-Za-z0-9_-]{0,48}", label) or label in variants:
parser.error("each --binary must use a unique filesystem-safe LABEL=/ABS/PATH")
binary = Path(filename).resolve()
if not binary.is_file():
parser.error(f"binary does not exist: {binary}")
variants[label] = binary
args.out, args.nginx, args.strace = args.out.resolve(), args.nginx.resolve(), args.strace.resolve()
for executable in (args.nginx, args.strace):
if not executable.is_file() or not os.access(executable, os.X_OK):
parser.error(f"executable is missing or is not executable: {executable}")
args.out.mkdir(parents=True, exist_ok=True)
version = subprocess.run([str(args.strace), "-V"], capture_output=True, text=True, timeout=10)
environment = {"kind": "selected syscall counts with ptrace instrumentation", "simulation": False,
"platform": platform.platform(), "python": sys.version, "strace_version": version.stdout + version.stderr,
"strace_sha256": sha256(args.strace), "nginx_sha256": sha256(args.nginx),
"variant_order": list(variants), "selected_syscalls": SYSCALLS,
"started_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
"settings": {key: str(value) if isinstance(value, Path) else value for key, value in vars(args).items()},
"limitations": ["strace changes scheduling and latency; do not use instrumented times as RPS evidence",
"Counts cover selected syscalls in every traced server thread, including background work",
"Normalized counts divide the whole trace by verified profile requests; they do not isolate per-request causality",
"This is a warm-file-cache, c1, HTTP/1.1 plaintext static-file scenario; no general capacity inference",
"Syscall counts do not directly count Tokio task dispatches or memory allocations"]}
for path in ("/proc/sys/kernel/yama/ptrace_scope", "/sys/fs/cgroup/cpu.max", "/sys/fs/cgroup/memory.max"):
try:
environment[path] = Path(path).read_text().strip()
except OSError:
environment[path] = None
write_json(args.out / "environment.json", environment)
rows, error = [], None
try:
with Fixture(args.out / "fixture", workers=args.workers, server_cpus=args.server_cpus,
origin_cpus=args.origin_cpus, origin_workers=args.origin_workers) as fixture:
fixture.nginx(args.nginx, frontend=False)
for label, binary in variants.items():
result = profile_variant(fixture, label, binary, args)
rows.append(result)
write_json(args.out / "profiles.json", rows)
print(json.dumps({"variant": label, "requests": args.requests, "passed": result["passed"],
"selected_syscall_calls": result["selected_total"]["calls"]}), flush=True)
except Exception as exc:
error = {"error": str(exc), "traceback": traceback.format_exc()}
write_json(args.out / "failure.json", error)
print(json.dumps(error), flush=True)
finally:
write_json(args.out / "profiles.json", rows)
write_json(args.out / "completion.json", {"completed": error is None, "profiles": len(rows),
"simulation": False, "throughput_measurement": False,
"finished_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())})
seal_results(args.out)
return 1 if error else 0
if __name__ == "__main__":
sys.exit(main())
scripts/performance/regression.py
Executable
+107
View File
@@ -0,0 +1,107 @@
#!/usr/bin/env python3
"""Run black-box acceptance tests against one or more real RustProxy binaries.
Example:
python3 scripts/performance/regression.py --binary before=/abs/before \
--binary after=/abs/after --out /abs/results/regression
"""
from __future__ import annotations
import argparse
import json
import os
from pathlib import Path
import platform
import sys
import time
import traceback
from common import Fixture, seal_results, write_json
from peer import Peer
import checks_cache
import checks_policy
import checks_proxy
import checks_static
GROUPS = {"static": checks_static.run, "proxy": checks_proxy.run,
"policy": checks_policy.run, "cache": checks_cache.run,
"transform": checks_cache.transforms}
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--binary", action="append", required=True, metavar="LABEL=/ABS/PATH")
parser.add_argument("--out", required=True, type=Path)
parser.add_argument("--workers", type=int, default=4)
parser.add_argument("--groups", default=",".join(GROUPS))
args = parser.parse_args()
groups = args.groups.split(",")
if any(group not in GROUPS for group in groups):
parser.error(f"groups must be selected from {','.join(GROUPS)}")
args.out = args.out.resolve()
args.out.mkdir(parents=True, exist_ok=True)
metadata = {"kind": "live RustProxy black-box HTTP acceptance", "simulation": False,
"python": sys.version, "platform": platform.platform(),
"cpu_affinity": sorted(os.sched_getaffinity(0)) if hasattr(os, "sched_getaffinity") else None,
"started_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
"groups": groups,
"not_covered": ["TLS/ALPN", "HTTP/2", "HTTP/3", "PHP-FPM",
"Tinyauth verifier", "real DNS rebinding", "24-hour soak",
"concurrent filesystem rename-race proof"]}
write_json(args.out / "environment.json", metadata)
reports = {}
for binary_arg in args.binary:
label, separator, name = binary_arg.partition("=")
if not separator or not label.replace("-", "").replace("_", "").isalnum():
parser.error("--binary must be LABEL=/ABS/PATH with a simple unique label")
if label in reports:
parser.error("binary labels must be unique")
cases = []
started = time.monotonic()
output = args.out / label
def check(case, passed, **details):
row = {"case": case, "passed": passed, **details}
cases.append(row)
print(f"{label} {'SKIP' if passed is None else 'PASS' if passed else 'FAIL'} {case}", flush=True)
write_json(output / "checks.json", cases)
try:
with Fixture(output, workers=args.workers) as fixture:
peer = Peer(fixture.origin_port, fixture.site)
try:
rust = fixture.rust(Path(name), label)
check("runtime.started", True)
for group in groups:
try:
rust.reset()
GROUPS[group](check, rust, peer)
except Exception as exc:
check(group + ".runner", False, error=str(exc), traceback=traceback.format_exc())
finally:
rust.reset()
check("runtime.still_alive", rust.process.poll() is None)
finally:
peer.close()
except Exception as exc:
check("runtime.runner", False, error=str(exc), traceback=traceback.format_exc())
report = {"label": label, "kind": metadata["kind"], "simulation": False,
"seconds": time.monotonic() - started, "passed": sum(c["passed"] is True for c in cases),
"failed": sum(c["passed"] is False for c in cases),
"skipped": sum(c["passed"] is None for c in cases), "cases": cases}
reports[label] = report
write_json(output / "report.json", report)
print(f"{label}: {report['passed']} passed, {report['failed']} failed, {report['skipped']} skipped", flush=True)
comparison = {"variants": {name: {key: report[key] for key in ["passed", "failed", "skipped", "seconds"]}
for name, report in reports.items()},
"cases": [{"case": case, **{name: next((entry["passed"] for entry in report["cases"]
if entry["case"] == case), "not_run") for name, report in reports.items()}}
for case in sorted({entry["case"] for report in reports.values() for entry in report["cases"]})]}
write_json(args.out / "comparison.json", comparison)
seal_results(args.out)
return 1 if any(report["failed"] for report in reports.values()) else 0
if __name__ == "__main__":
sys.exit(main())
scripts/performance/resources.py
Executable
+228
View File
@@ -0,0 +1,228 @@
"""Linux /proc sampling outside the server process; no synthetic resource data."""
from __future__ import annotations
import os
from pathlib import Path
import threading
import time
def cgroup_cpu_stat(path=Path("/sys/fs/cgroup/cpu.stat")):
values = {key: int(value) for key, value in (line.split() for line in path.read_text().splitlines())}
if not {"nr_throttled", "throttled_usec"} <= values.keys():
raise RuntimeError(f"{path} lacks cgroup v2 throttling counters")
return {"path": str(path), "observed_monotonic": time.monotonic(), "values": values}
def cgroup_cpu_delta(before, after):
delta = {key: after["values"][key] - value for key, value in before["values"].items()
if key in after["values"]}
if any(value < 0 for value in delta.values()):
raise RuntimeError("cgroup CPU counters reset during the measured run")
return delta
def parse_cpu_set(value):
if not value:
raise ValueError("an explicit, nonempty CPU set is required")
result = set()
for part in value.split(","):
bounds = part.strip().split("-")
if len(bounds) not in (1, 2) or any(not item.isascii() or not item.isdigit() for item in bounds):
raise ValueError(f"invalid CPU set {value!r}; use comma-separated numbers or ranges")
start, end = int(bounds[0]), int(bounds[-1])
if start > end or end - start > 65536:
raise ValueError(f"invalid CPU range {part!r}")
result.update(range(start, end + 1))
return result
def validate_cpu_sets(values, allowed):
groups = {label: parse_cpu_set(value) for label, value in values.items()}
for label, cpus in groups.items():
if not cpus <= allowed:
raise ValueError(f"{label} CPUs {sorted(cpus - allowed)} are outside the allowed affinity")
for other, other_cpus in groups.items():
if label < other and cpus & other_cpus:
raise ValueError(f"{label} and {other} CPU sets overlap: {sorted(cpus & other_cpus)}")
return groups
def affinity_snapshot(root, expected, expected_processes=None):
"""Verify every currently visible thread in the root process and children."""
tasks = []
for pid in descendants(root):
try:
threads = list(Path(f"/proc/{pid}/task").iterdir())
except FileNotFoundError:
continue
for thread in threads:
try:
actual = set(os.sched_getaffinity(int(thread.name)))
except ProcessLookupError:
continue
if actual != expected:
raise RuntimeError(f"affinity mismatch for PID {pid} TID {thread.name}: "
f"expected {sorted(expected)}, got {sorted(actual)}")
tasks.append({"pid": pid, "tid": int(thread.name), "cpus": sorted(actual)})
if not tasks or not any(task["pid"] == root for task in tasks):
raise RuntimeError(f"cannot inspect affinity of live process {root}")
processes = len({task["pid"] for task in tasks})
if expected_processes is not None and processes != expected_processes:
raise RuntimeError(f"process count mismatch for PID {root}: "
f"expected {expected_processes}, inspected {processes}")
return {"root_pid": root, "processes": processes,
"threads": len(tasks), "tasks": tasks}
def wait_for_affinity(process, expected, timeout=2):
"""Allow taskset to execute before inspecting the generator's inherited mask."""
deadline = time.monotonic() + timeout
while True:
if process.poll() is not None:
raise RuntimeError(f"generator exited before affinity verification: {process.returncode}")
try:
return affinity_snapshot(process.pid, expected)
except RuntimeError:
if time.monotonic() >= deadline:
raise
time.sleep(.01)
def descendants(pid, proc_root=Path("/proc")):
"""Find the process tree through stat PPids, including nginx workers.
/proc/PID/task/PID/children is optional on some Linux kernels and may also
omit children created by another thread. Missing that file is not evidence
that a live process is a leaf. A single PPid-map scan avoids either case.
"""
children, seen = {}, set()
for entry in proc_root.iterdir():
if not entry.name.isdecimal():
continue
current = int(entry.name)
try:
fields = (entry / "stat").read_text().rsplit(")", 1)[1].split()
parent = int(fields[1])
except (FileNotFoundError, ProcessLookupError):
# A non-root process may finish while the snapshot is assembled.
continue
except (OSError, ValueError, IndexError) as error:
raise RuntimeError(f"cannot discover process tree from PID {current}: {error}") from error
if fields[0] in {"Z", "X", "x"}:
# A dead/zombie worker cannot satisfy the configured live count.
continue
seen.add(current)
children.setdefault(parent, []).append(current)
if pid not in seen:
raise RuntimeError(f"no readable live root process for PID {pid}")
result, pending = [], [pid]
while pending:
current = pending.pop()
if current in result:
continue
result.append(current)
pending.extend(sorted(children.get(current, []), reverse=True))
return result
def process_stats(pid):
try:
data = Path(f"/proc/{pid}/stat").read_text().rsplit(")", 1)[1].split()
if data[0] in {"Z", "X", "x"}:
return None
ticks = os.sysconf("SC_CLK_TCK")
pages = os.sysconf("SC_PAGE_SIZE")
return {"pid": pid, "start_ticks": int(data[19]),
"cpu_seconds": (int(data[11]) + int(data[12])) / ticks,
"rss_bytes": int(data[21]) * pages,
"fd": len(list(Path(f"/proc/{pid}/fd").iterdir())),
"threads": int(data[17])}
except (FileNotFoundError, ProcessLookupError):
return None
except (OSError, ValueError, IndexError) as error:
raise RuntimeError(f"cannot read resource counters for PID {pid}: {error}") from error
class Monitor:
def __init__(self, roots, interval=.5, expected_processes=None):
self.roots = roots
self.expected_processes = expected_processes or {}
self.process_identities = {}
self.interval = interval
if interval <= 0:
raise ValueError("resource interval must be positive")
self.samples = []
self.errors = []
self.stop_event = threading.Event()
self.started = time.monotonic()
self.thread = threading.Thread(target=self.run, daemon=True)
def sample(self):
groups = {}
for label, root in self.roots.items():
processes = [stats for pid in descendants(root) if (stats := process_stats(pid)) is not None]
if not any(process["pid"] == root for process in processes):
raise RuntimeError(f"no readable live root process for {label} (PID {root})")
expected = self.expected_processes.get(label)
if expected is not None and len(processes) != expected:
raise RuntimeError(f"resource process count mismatch for {label} (PID {root}): "
f"expected {expected}, read {len(processes)}")
identities = {(process["pid"], process["start_ticks"]) for process in processes}
if expected is not None:
original = self.process_identities.setdefault(label, identities)
if identities != original:
raise RuntimeError(f"process identities changed while measuring {label}: "
f"expected {sorted(original)}, read {sorted(identities)}")
groups[label] = {"processes": processes,
"rss_bytes": sum(p["rss_bytes"] for p in processes),
"fd": sum(p["fd"] for p in processes),
"threads": sum(p["threads"] for p in processes)}
self.samples.append({"seconds": time.monotonic() - self.started, "groups": groups})
def run(self):
while not self.stop_event.wait(self.interval):
try:
self.sample()
except Exception as error:
self.errors.append(str(error))
self.stop_event.set()
def start(self):
self.sample()
self.thread.start()
def stop(self):
self.stop_event.set()
self.thread.join(timeout=3)
if self.thread.is_alive():
self.errors.append("resource sampler did not stop")
try:
self.sample()
except Exception as error:
self.errors.append(str(error))
def summary(self, requests):
if self.errors:
raise RuntimeError("resource sampling failed: " + "; ".join(self.errors))
result = {}
for label in self.roots:
observations = [sample["groups"][label] for sample in self.samples]
totals = {}
for group in observations:
for process in group["processes"]:
key = (process["pid"], process["start_ticks"])
low, high = totals.get(key, (process["cpu_seconds"], process["cpu_seconds"]))
totals[key] = (min(low, process["cpu_seconds"]), max(high, process["cpu_seconds"]))
if not totals:
raise RuntimeError(f"no readable process samples for {label}; refusing zero resource metrics")
seconds = sum(high - low for low, high in totals.values())
result[label] = {"cpu_seconds": seconds,
"cpu_us_per_request": seconds * 1_000_000 / requests if requests else None,
"rss_peak_bytes": max(group["rss_bytes"] for group in observations),
"rss_start_bytes": observations[0]["rss_bytes"],
"rss_end_bytes": observations[-1]["rss_bytes"],
"fd_peak": max(group["fd"] for group in observations),
"fd_start": observations[0]["fd"], "fd_end": observations[-1]["fd"],
"processes_seen": len(totals)}
return result
scripts/performance/soak.py
Executable
+325
View File
@@ -0,0 +1,325 @@
#!/usr/bin/env python3
"""Fixed-rate mixed traffic, controlled client closes and cache hot-toggle soak.
The default is a 30-minute live run. Short invocations must be explicitly marked
--exploratory and cannot satisfy the release soak gate. No run is auto-scheduled.
"""
from __future__ import annotations
import argparse
from collections import Counter
from concurrent.futures import ThreadPoolExecutor, wait, FIRST_COMPLETED
import copy
import hashlib
from http.client import HTTPConnection
import json
import os
from pathlib import Path
import platform
import statistics
import sys
import time
import traceback
from benchmark import counter_values
from common import Fixture, http, seal_results, sha256, write_json
from resources import Monitor
from telemetry import traffic_analytics_snapshot, traffic_analytics_delta
PAYLOADS = ["small.html", "large.bin", "huge.bin"]
def fetch(index, rust, expected, abort_every):
name = PAYLOADS[index % len(PAYLOADS)]
source = "proxy" if (index // len(PAYLOADS)) % 2 else "static"
port = rust.reverse_port if source == "proxy" else rust.static_port
started = time.monotonic()
case = {"index": index, "source": source, "file": name, "method": "GET", "intentional_abort": False}
try:
if abort_every and index % abort_every == 0:
case.update({"file": "huge.bin", "intentional_abort": True})
connection = HTTPConnection("127.0.0.1", port, timeout=10)
try:
connection.request("GET", "/huge.bin", headers={"Accept-Encoding": "identity"})
response = connection.getresponse()
body = response.read(128)
case.update({"status": response.status, "bytes": len(body),
"cache": response.getheader("X-Cache"),
"passed": response.status == 200 and body == expected["huge.bin"][:128]})
finally:
connection.close()
else:
headers = {"Accept-Encoding": "identity"}
method = "HEAD" if index % 29 == 0 else "GET"
if method == "GET" and index % 37 == 0:
headers["Range"] = "bytes=0-127"
expected_status, expected_body = 206, expected[name][:128]
else:
expected_status, expected_body = 200, b"" if method == "HEAD" else expected[name]
status, response_headers, body = http(port, "/" + name, method, headers)
case.update({"method": method, "status": status, "bytes": len(body),
"cache": response_headers.get("x-cache"),
"passed": status == expected_status and body == expected_body})
if not case["passed"]:
case.update({"expected_status": expected_status, "expected_bytes": len(expected_body),
"body_sha256": hashlib.sha256(body).hexdigest()})
except Exception as exc:
case.update({"passed": False, "error": str(exc)})
case["seconds"] = time.monotonic() - started
return case
def probe_cache_mode(rust, expected, enabled, phase, out, observations):
"""Record two exact GETs per source, separate from the paced workload."""
phase_observations = []
for source, port in [("proxy", rust.reverse_port), ("static", rust.static_port)]:
replies = []
for attempt in [1, 2]:
try:
status, headers, body = http(port, "/small.html?soak_cache_probe=1",
headers={"Accept-Encoding": "identity"})
replies.append({"attempt": attempt, "status": status, "cache": headers.get("x-cache"),
"bytes": len(body), "sha256": hashlib.sha256(body).hexdigest(),
"body_passed": status == 200 and body == expected["small.html"]})
except Exception as error:
replies.append({"attempt": attempt, "body_passed": False, "error": str(error)})
mode_passed = replies[1].get("cache") == "HIT" if enabled else all(
reply.get("cache") is None for reply in replies)
phase_observations.append({"phase": phase, "source": source, "cache_enabled": enabled,
"responses": replies,
"passed": all(reply["body_passed"] for reply in replies) and mode_passed})
observations.extend(phase_observations)
write_json(out / "cache-probes.json", observations)
if not all(observation["passed"] for observation in phase_observations):
raise RuntimeError(f"cache mode/body probe failed after {phase}; see cache-probes.json")
def backend_drain_snapshot(rust):
status, _, body = rust.admin("/api/status")
if status != 200:
return {"passed": False, "error": f"admin status HTTP {status}"}
backends = []
def walk(value):
if isinstance(value, dict):
if "id" in value and "active" in value and "status" in value and "requests" in value:
backends.append({key: value[key] for key in ("id", "active", "status", "requests")})
for child in value.values():
walk(child)
elif isinstance(value, list):
for child in value:
walk(child)
walk(json.loads(body))
return {"passed": bool(backends) and all(backend["active"] == 0 for backend in backends),
"backends": backends}
def memory_trend(samples, label="server"):
observations = [(sample["seconds"], sample["groups"][label]["rss_bytes"], sample["groups"][label]["fd"])
for sample in samples if sample["seconds"] >= 60]
if len(observations) < 3:
return {"available": False, "reason": "less than three samples after the first minute"}
tail_seconds = observations[-1][0]
early = [row[1] for row in observations if row[0] < observations[0][0] + 60]
late = [row[1] for row in observations if row[0] >= tail_seconds - 60]
x_mean = statistics.mean(row[0] for row in observations)
y_mean = statistics.mean(row[1] for row in observations)
denominator = sum((row[0] - x_mean) ** 2 for row in observations)
slope = sum((row[0] - x_mean) * (row[1] - y_mean) for row in observations) / denominator if denominator else 0
return {"available": True, "rss_first_minute_after_warmup_median": statistics.median(early),
"rss_last_minute_median": statistics.median(late),
"rss_growth_bytes": statistics.median(late) - statistics.median(early),
"rss_linear_bytes_per_hour": slope * 3600, "fd_first": observations[0][2],
"fd_last": observations[-1][2], "fd_peak": max(row[2] for row in observations),
"interpretation": "Descriptive only: cache toggles and allocator retention can change RSS without a leak"}
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--binary", type=Path, required=True)
parser.add_argument("--label", default="after")
parser.add_argument("--nginx", type=Path, required=True)
parser.add_argument("--out", type=Path, required=True)
parser.add_argument("--duration", type=int, default=1800)
parser.add_argument("--rate", type=float, default=100, help="offered requests per second, not a saturation benchmark")
parser.add_argument("--clients", type=int, default=8)
parser.add_argument("--workers", type=int, default=4)
parser.add_argument("--origin-workers", type=int, default=2)
parser.add_argument("--server-cpus")
parser.add_argument("--origin-cpus")
parser.add_argument("--toggle-seconds", type=int, default=120)
parser.add_argument("--sample-seconds", type=int, default=5)
parser.add_argument("--abort-every", type=int, default=101)
parser.add_argument("--cooldown-seconds", type=int, default=15)
parser.add_argument("--exploratory", action="store_true")
args = parser.parse_args()
if args.duration < 1800 and not args.exploratory:
parser.error("the soak gate requires >=1800 seconds; label short diagnostics with --exploratory")
if args.rate <= 0 or not 1 <= args.clients <= 128 or args.toggle_seconds < 1 or args.sample_seconds < 1:
parser.error("rate, sample/toggle intervals and clients must be positive; clients <=128")
args.out = args.out.resolve()
if args.out.exists() and any(args.out.iterdir()):
parser.error("--out must be a new or empty evidence directory")
args.out.mkdir(parents=True, exist_ok=True)
settings = {key: str(value) if isinstance(value, Path) else value for key, value in vars(args).items()}
write_json(args.out / "environment.json", {
"kind": "actual RustProxy fixed-rate mixed-traffic resource soak", "simulation": False,
"settings": settings, "platform": platform.platform(), "binary_sha256": sha256(args.binary),
"started_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
"frontend_connection_policy": "A new HTTP/1.1 TCP connection for each workload request and probe",
"controlled_close_interpretation": "Client closes after reading 128 bytes; the server may already have sent the full body. No server-cancellation count is inferred.",
"traffic_analytics_observation_policy": "Global counters are read before traffic, periodically with status diagnostics, and after final drain. Deltas include cache-mode probes and cooldown, and are separate from HTTP errors.",
"not_covered": ["24-hour soak", "TLS/H2/H3 soak", "multi-backend failures", "PHP-FPM",
"real application write load", "origin DNS/TLS rotation", "production traffic",
"long-lived frontend keepalive connections"]})
totals = {"submitted": 0, "completed": 0, "passed": 0, "failed": 0, "controlled_client_closes": 0,
"response_bytes": 0, "scheduler_late_events": 0, "peak_pending": 0}
failure_examples, progress, toggles, counter_samples = [], [], [], []
traffic_samples, traffic_delta = [], None
cache_probes, cache_status_counts = [], {"proxy": Counter(), "static": Counter()}
drain_snapshot, configuration_restored = None, False
samples, resources = [], {}
fatal, runtime_seconds = None, 0
try:
with Fixture(args.out / "fixture", workers=args.workers, server_cpus=args.server_cpus,
origin_cpus=args.origin_cpus, origin_workers=args.origin_workers) as fixture:
nginx = fixture.nginx(args.nginx, frontend=False)
rust = fixture.rust(args.binary, args.label)
rust.login()
rust.wait_quiescent()
expected = {name: (fixture.site / name).read_bytes() for name in PAYLOADS}
config = copy.deepcopy(rust.base)
config["cache"].update({"expose_status": True})
config["web_services"][0]["cache_control"] = "public, max-age=60"
rust.apply(config, "soak-initial")
probe_cache_mode(rust, expected, False, "initial-off", args.out, cache_probes)
traffic_before = traffic_analytics_snapshot(rust)
traffic_samples.append({"phase": "before", "seconds": 0, "snapshot": traffic_before})
write_json(args.out / "traffic-analytics.json", traffic_samples)
monitor = Monitor({"server": rust.process.pid, "origin": nginx["origin"].pid}, args.sample_seconds)
monitor.start()
start = time.monotonic()
deadline = start + args.duration
next_toggle, next_snapshot, next_submit = start + args.toggle_seconds, start, start
pending = set()
def consume(done):
for future in done:
result = future.result()
totals["completed"] += 1
totals["passed" if result["passed"] else "failed"] += 1
totals["controlled_client_closes"] += int(result["intentional_abort"])
totals["response_bytes"] += result.get("bytes", 0)
cache_status = (result.get("cache") or "absent") if "status" in result else "no_response"
cache_status_counts[result["source"]][cache_status] += 1
if not result["passed"] and len(failure_examples) < 100:
failure_examples.append(result)
try:
with ThreadPoolExecutor(max_workers=args.clients) as pool:
while time.monotonic() < deadline:
now = time.monotonic()
done = {future for future in pending if future.done()}
consume(done)
pending -= done
if now >= next_toggle:
config["cache"]["enabled"] = not config["cache"]["enabled"]
rust.apply(config, f"soak-toggle-{len(toggles) + 1}")
toggles.append({"seconds": time.monotonic() - start, "cache_enabled": config["cache"]["enabled"]})
write_json(args.out / "cache-toggles.json", toggles)
probe_cache_mode(rust, expected, config["cache"]["enabled"],
f"toggle-{len(toggles)}", args.out, cache_probes)
next_toggle += args.toggle_seconds
if now >= next_snapshot:
status, headers, body = rust.admin("/api/status")
if status != 200:
raise RuntimeError(f"admin status became unavailable: HTTP {status}")
counter_samples.append({"seconds": time.monotonic() - start,
"cache_enabled": config["cache"]["enabled"],
"counters": counter_values(json.loads(body))})
traffic_now = traffic_analytics_snapshot(rust)
traffic_delta = traffic_analytics_delta(traffic_before, traffic_now)
traffic_samples.append({"phase": "during", "seconds": time.monotonic() - start,
"snapshot": traffic_now, "delta_from_start": traffic_delta})
row = {"seconds": time.monotonic() - start, **totals,
"cache_enabled": config["cache"]["enabled"], "pending": len(pending),
"observed_cache_status_counts": {source: dict(counts)
for source, counts in cache_status_counts.items()}}
progress.append(row)
write_json(args.out / "progress.json", progress)
write_json(args.out / "status-counters.json", counter_samples)
write_json(args.out / "traffic-analytics.json", traffic_samples)
write_json(args.out / "resources.json", list(monitor.samples))
print(json.dumps(row), flush=True)
next_snapshot = time.monotonic() + args.sample_seconds
now = time.monotonic()
if len(pending) >= args.clients * 2:
done, pending = wait(pending, timeout=.1, return_when=FIRST_COMPLETED)
consume(done)
continue
if now < next_submit:
time.sleep(min(.02, next_submit - now))
continue
if now > next_submit + .25:
totals["scheduler_late_events"] += 1
next_submit = now
totals["submitted"] += 1
pending.add(pool.submit(fetch, totals["submitted"], rust, expected, args.abort_every))
totals["peak_pending"] = max(totals["peak_pending"], len(pending))
next_submit += 1 / args.rate
done, pending = wait(pending, timeout=30)
consume(done)
if pending:
raise RuntimeError("requests failed to settle during bounded soak drain")
runtime_seconds = time.monotonic() - start
rust.reset()
configuration_restored = True
cooldown_until = time.monotonic() + args.cooldown_seconds
while time.monotonic() < cooldown_until:
time.sleep(max(0, min(.5, cooldown_until - time.monotonic())))
probe_cache_mode(rust, expected, False, "restored-off", args.out, cache_probes)
for _ in range(40):
drain_snapshot = backend_drain_snapshot(rust)
if drain_snapshot["passed"]:
break
time.sleep(.05)
write_json(args.out / "backend-drain.json", drain_snapshot)
if not drain_snapshot["passed"]:
raise RuntimeError("backend active exchanges did not drain to zero")
traffic_after = traffic_analytics_snapshot(rust)
traffic_delta = traffic_analytics_delta(traffic_before, traffic_after)
traffic_samples.append({"phase": "after_drain", "seconds": time.monotonic() - start,
"snapshot": traffic_after, "delta_from_start": traffic_delta})
finally:
monitor.stop()
samples = monitor.samples
resources = monitor.summary(totals["completed"])
write_json(args.out / "resources.json", samples)
write_json(args.out / "status-counters.json", counter_samples)
write_json(args.out / "traffic-analytics.json", traffic_samples)
write_json(args.out / "failure-examples.json", failure_examples)
except Exception as exc:
fatal = {"error": str(exc), "traceback": traceback.format_exc()}
write_json(args.out / "failure.json", fatal)
report = {"completed": fatal is None, "simulation": False, "exploratory": args.exploratory,
"duration_seconds": runtime_seconds, "soak_duration_gate": runtime_seconds >= 1800,
"totals": totals, "resources": resources, "memory_trend": memory_trend(samples) if samples else None,
"cache_toggles": len(toggles), "errors": failure_examples,
"traffic_analytics": {"observations": len(traffic_samples), "delta": traffic_delta,
"includes_cache_mode_probes_and_cooldown": True},
"cache_mode_probes": {"source_checks": len(cache_probes),
"requests": sum(len(probe["responses"]) for probe in cache_probes),
"passed": bool(cache_probes) and all(probe["passed"] for probe in cache_probes)},
"observed_cache_status_counts": cache_status_counts,
"configuration_restored": configuration_restored, "backend_drain": drain_snapshot,
"actual_completed_rps": totals["completed"] / runtime_seconds if runtime_seconds else None,
"passed": fatal is None and totals["failed"] == 0 and runtime_seconds >= args.duration}
write_json(args.out / "report.json", report)
seal_results(args.out)
print(json.dumps(report), flush=True)
return 0 if report["passed"] else 1
if __name__ == "__main__":
sys.exit(main())
scripts/performance/telemetry.py
Executable
+47
View File
@@ -0,0 +1,47 @@
"""Selected counters from existing authenticated APIs, without traffic records."""
from __future__ import annotations
import json
import time
def traffic_analytics_snapshot(rust, timeout=2):
"""Read global analytics queue loss; never retain paths, IPs or user agents.
queue_dropped is process-global even when the API's vhost query is used.
recorded_requests is asynchronous analytics work, not an HTTP success count.
Call outside a timed load interval so this control-plane read is not charged
to the request path being measured.
"""
status, _, body = rust.admin("/api/traffic", timeout=timeout)
if status != 200:
raise RuntimeError(f"traffic analytics counter capture failed: HTTP {status}")
payload = json.loads(body)
if not isinstance(payload, dict) or payload.get("scope") != "*" or payload.get("since") != "process_start":
raise RuntimeError("traffic analytics requires the global process-start scope")
analytics = payload.get("analytics")
if not isinstance(analytics, dict) or analytics.get("scope") != "*":
raise RuntimeError("traffic analytics response lacks the global queue view")
data = analytics.get("data")
if not isinstance(data, dict):
raise RuntimeError("traffic analytics response lacks the recorded counters")
values = {"queue_capacity": analytics.get("queue_capacity"),
"queue_dropped": analytics.get("queue_dropped"),
"recorded_requests": data.get("requests")}
if any(type(value) is not int or value < 0 for value in values.values()) or not values["queue_capacity"]:
raise RuntimeError("traffic analytics queue counters are absent or malformed")
return {"endpoint": "/api/traffic", "scope": "*", "since": "process_start",
"pid": rust.process.pid, "observed_monotonic": time.monotonic(), **values}
def traffic_analytics_delta(before, after):
"""Keep telemetry loss separate from failed HTTP requests and log rotation."""
for key in ["pid", "endpoint", "scope", "since", "queue_capacity"]:
if before[key] != after[key]:
raise RuntimeError(f"traffic analytics identity changed during capture: {key}")
result = {"queue_dropped_delta": after["queue_dropped"] - before["queue_dropped"],
"recorded_requests_delta": after["recorded_requests"] - before["recorded_requests"],
"interval_seconds": after["observed_monotonic"] - before["observed_monotonic"]}
if any(value < 0 for value in result.values()):
raise RuntimeError("traffic analytics counters or clock reset during capture")
return result
scripts/performance/transport_clients.py
Executable
+191
View File
@@ -0,0 +1,191 @@
"""Strict H2/TLS and H3/QUIC clients for local RustProxy acceptance tests.
The generated fixture certificate is explicitly trusted. Neither transport
silently falls back to HTTP/1.1 or disables certificate verification.
"""
from __future__ import annotations
import asyncio
from contextlib import asynccontextmanager
from dataclasses import dataclass, field
from pathlib import Path
import ssl
import httpx
from aioquic.asyncio import QuicConnectionProtocol, connect
from aioquic.h3.connection import H3Connection, MessageError
from aioquic.h3.events import DataReceived, HeadersReceived
from aioquic.quic.configuration import QuicConfiguration
from aioquic.quic.events import ConnectionTerminated, ProtocolNegotiated, StreamReset
@dataclass
class Response:
status: int
headers: dict[str, str]
body: bytes
protocol: str
alpn: str | None
stream_id: int | None = None
class H2Client:
def __init__(self, client: httpx.AsyncClient, port: int):
self.client, self.port = client, port
async def request(self, method, path, headers=None, body=None):
async def chunks():
for offset in range(0, len(body), 16384):
yield body[offset:offset + 16384]
await asyncio.sleep(0)
response = await self.client.request(
method, f"https://127.0.0.1:{self.port}{path}",
headers=headers or {}, content=chunks() if body is not None else None,
)
network = response.extensions.get("network_stream")
tls = network.get_extra_info("ssl_object") if network else None
alpn = tls.selected_alpn_protocol() if tls else None
if response.http_version != "HTTP/2" or alpn != "h2":
raise AssertionError(f"H2 required, got {response.http_version}, ALPN={alpn!r}")
return Response(response.status_code, dict(response.headers), response.content,
"HTTP/2", alpn, response.extensions.get("stream_id"))
@dataclass
class Pending:
future: asyncio.Future
status: int = 0
headers: dict[str, str] = field(default_factory=dict)
body: bytearray = field(default_factory=bytearray)
class HeadAwareH3Connection(H3Connection):
"""Supply request-method context missing from aioquic 1.7.0's length check.
RFC 9110 section 8.6 permits HEAD's Content-Length to describe a GET body.
Still reject any actual HEAD DATA; keep aioquic's exact length check for all
other requests. This changes only the test client, never the target server.
"""
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.head_streams = set()
def _check_content_length(self, stream):
if stream.stream_id in self.head_streams:
if stream.content_length:
raise MessageError("HEAD response contained DATA")
return
super()._check_content_length(stream)
def validate_head_adapter():
"""Keep adapter checks separate from the real-server acceptance totals."""
from types import SimpleNamespace
connection = HeadAwareH3Connection.__new__(HeadAwareH3Connection)
connection.head_streams = {0}
for stream_id, declared, received, valid in [(0, 1040, 0, True), (0, 1040, 1, False),
(4, 1040, 1040, True), (4, 1040, 0, False)]:
stream = SimpleNamespace(stream_id=stream_id, expected_content_length=declared,
content_length=received)
try:
connection._check_content_length(stream)
accepted = True
except MessageError:
accepted = False
assert accepted == valid, "H3 fixture HEAD adapter changed ordinary framing validation"
return {"cases": 4, "passed": 4, "kind": "test-client self-test, not RustProxy coverage"}
class H3Client(QuicConnectionProtocol):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.http = HeadAwareH3Connection(self._quic)
self.pending: dict[int, Pending] = {}
self.alpn = None
self.authority = None
def quic_event_received(self, event):
if isinstance(event, ProtocolNegotiated):
self.alpn = event.alpn_protocol
if isinstance(event, ConnectionTerminated):
for item in self.pending.values():
if not item.future.done():
item.future.set_exception(ConnectionError(
f"QUIC closed code={event.error_code}: {event.reason_phrase}"))
if isinstance(event, StreamReset):
item = self.pending.get(event.stream_id)
if item is not None and not item.future.done():
item.future.set_exception(ConnectionError(
f"H3 stream reset code={event.error_code}"))
for http_event in self.http.handle_event(event):
if not isinstance(http_event, (HeadersReceived, DataReceived)):
continue
item = self.pending.get(http_event.stream_id)
if item is None or item.future.done():
continue
if isinstance(http_event, HeadersReceived):
for name, value in http_event.headers:
if name == b":status":
item.status = int(value)
elif not name.startswith(b":"):
item.headers[name.decode("ascii").lower()] = value.decode("latin-1")
else:
item.body.extend(http_event.data)
if len(item.body) > 4 * 1024 * 1024:
item.future.set_exception(ValueError("fixture response exceeded 4 MiB bound"))
continue
if http_event.stream_ended:
item.future.set_result(Response(item.status, item.headers, bytes(item.body),
"HTTP/3", self.alpn, http_event.stream_id))
async def request(self, method, path, headers=None, body=None):
if self.alpn != "h3":
raise AssertionError(f"H3 required, got ALPN={self.alpn!r}")
stream_id = self._quic.get_next_available_stream_id()
item = Pending(asyncio.get_running_loop().create_future())
self.pending[stream_id] = item
if method == "HEAD":
self.http.head_streams.add(stream_id)
fields = [(b":method", method.encode("ascii")), (b":scheme", b"https"),
(b":authority", self.authority.encode("ascii")),
(b":path", path.encode("ascii"))]
fields += [(name.lower().encode("ascii"), value.encode("latin-1"))
for name, value in (headers or {}).items()]
try:
self.http.send_headers(stream_id, fields, end_stream=body is None)
self.transmit()
if body is not None:
if not body:
self.http.send_data(stream_id, b"", end_stream=True)
self.transmit()
for offset in range(0, len(body), 16384):
end = min(offset + 16384, len(body))
self.http.send_data(stream_id, body[offset:end], end_stream=end == len(body))
self.transmit()
await asyncio.sleep(0)
return await asyncio.wait_for(item.future, timeout=10)
finally:
self.pending.pop(stream_id, None)
self.http.head_streams.discard(stream_id)
@asynccontextmanager
async def client(protocol: str, port: int, certificate: Path):
if protocol == "h2":
context = ssl.create_default_context(cafile=str(certificate))
async with httpx.AsyncClient(http2=True, verify=context, trust_env=False,
timeout=10, limits=httpx.Limits(max_connections=1)) as session:
yield H2Client(session, port)
elif protocol == "h3":
config = QuicConfiguration(is_client=True, alpn_protocols=["h3"],
server_name="localhost", idle_timeout=10)
config.verify_mode = ssl.CERT_REQUIRED
config.load_verify_locations(cafile=str(certificate))
async with connect("127.0.0.1", port, configuration=config,
create_protocol=H3Client) as session:
session.authority = f"127.0.0.1:{port}"
yield session
else:
raise ValueError(f"unsupported fixture protocol {protocol!r}")
scripts/performance/transport_regression.py
Executable
+281
View File
@@ -0,0 +1,281 @@
#!/usr/bin/env python3
"""Real, bounded RustProxy H2/TLS, H3/QUIC and H1 WebSocket acceptance.
Install Python dependencies aioquic, httpx[http2] and websockets. Example:
python3 scripts/performance/transport_regression.py --binary before=/abs/binary \
--binary after=/abs/binary --out /abs/results --python-deps /abs/optional/pydeps
Only fresh temporary databases and loopback listeners are used. Certificates
are generated by RustProxy and verified by clients; no protocol fallback is used.
This is an interoperability/functional suite, not a throughput benchmark.
"""
from __future__ import annotations
import argparse
import asyncio
import copy
import hashlib
import importlib.metadata
import json
from pathlib import Path
import platform
import socket
import sys
import time
import traceback
from common import Fixture, seal_results, sha256, write_json
from peer import Peer
def response_details(response):
return {"status": response.status, "protocol": response.protocol,
"alpn": response.alpn, "stream_id": response.stream_id,
"bytes": len(response.body), "sha256": hashlib.sha256(response.body).hexdigest(),
"cache": response.headers.get("x-cache"),
"cache_reason": response.headers.get("x-cache-reason")}
def tls_port(fixture):
while True:
port = fixture.port()
with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as udp:
try:
udp.bind(("127.0.0.1", port))
return port
except OSError:
continue
def transport_config(rust, reverse_tls, static_tls, websocket_port, websocket_origin):
config = copy.deepcopy(rust.base)
for source, port in [("reverse", reverse_tls), ("static", static_tls)]:
identifier = f"{source}-tls"
config["listeners"]["endpoints"].append({
"id": identifier, "name": identifier, "scope": "internal", "enabled": True,
"binding": {"address": f"127.0.0.1:{port}", "tls": True, "http2": True,
"http3": {"max_concurrent_connections": 16,
"max_concurrent_bidistream": 16,
"max_concurrent_unistream": 16, "max_idle_timeout": 10}}})
vhost = next(v for v in config["vhosts"] if v["id"] == source)
vhost["frontend"]["listener_ids"].append(identifier)
vhost["http2_enabled"] = True
vhost["http3_enabled"] = True
config["listeners"]["endpoints"].append({
"id": "websocket", "name": "WebSocket", "scope": "internal", "enabled": True,
"binding": {"address": f"127.0.0.1:{websocket_port}", "tls": False,
"http2": False, "http3": None}})
config["vhosts"].append({
"id": "websocket", "frontend": {"hosts": ["127.0.0.1", "localhost"],
"listener_ids": ["websocket"]},
"websockets": True,
"backends": [{"id": "echo", "url": f"http://127.0.0.1:{websocket_origin}",
"allow_private": True}],
"load_balancing": {"health_check": {"mode": "off"}}, "routes": []})
config["cache"].update({"enabled": False, "expose_status": True, "expose_hits": True,
"expose_cache_status": True, "expose_only_hits": False})
config["compression"] = {"gzip": False, "br": False, "zstd": False, "deflate": False}
config["web_services"][0]["cache_control"] = "public, max-age=60"
return config
async def http_cases(check, session, protocol, source, cache, fixture):
prefix = f"{protocol}.{source}.cache_{'on' if cache else 'off'}"
expected = (fixture.site / "small.html").read_bytes()
path = f"/transport-{protocol}.html"
first = await session.request("GET", path)
check(prefix + ".get", first.status == 200 and first.body == expected,
**response_details(first))
second = await session.request("GET", path)
check(prefix + ".cache_delivery", second.status == 200 and second.body == expected
and ((first.headers.get("x-cache") == "MISS" and second.headers.get("x-cache") == "HIT")
if cache else second.headers.get("x-cache") not in ("MISS", "HIT")),
first_cache=first.headers.get("x-cache"), **response_details(second))
head = await session.request("HEAD", path)
check(prefix + ".head", head.status == 200 and head.body == b""
and head.headers.get("content-length") == str(len(expected)), **response_details(head))
etag = second.headers.get("etag")
check(prefix + ".validator", bool(etag), etag=etag)
ranged = await session.request("GET", path, {"range": "bytes=11-40", "if-range": etag or ""})
check(prefix + ".range", ranged.status == 206 and ranged.body == expected[11:41]
and ranged.headers.get("content-range") == f"bytes 11-40/{len(expected)}",
**response_details(ranged))
mismatch = await session.request("GET", path, {"range": "bytes=11-40", "if-range": '"different"'})
check(prefix + ".if_range_mismatch", mismatch.status == 200 and mismatch.body == expected,
**response_details(mismatch))
conditional = await session.request("GET", path, {"if-none-match": etag or ""})
check(prefix + ".conditional_304", conditional.status == 304 and conditional.body == b"",
**response_details(conditional))
invalid = await session.request("GET", path, {"range": "bytes=999999-"})
check(prefix + ".range_416", invalid.status == 416 and invalid.body == b""
and invalid.headers.get("content-range") == f"bytes */{len(expected)}",
**response_details(invalid))
for name in ["large.bin", "huge.bin"]:
expected_file = (fixture.site / name).read_bytes()
response = await session.request("GET", "/" + name)
check(prefix + ".get." + name, response.status == 200 and response.body == expected_file,
**response_details(response))
parallel = await asyncio.gather(*(session.request("GET", "/large.bin") for _ in range(4)))
expected_large = (fixture.site / "large.bin").read_bytes()
check(prefix + ".multiplex_four_streams", all(r.status == 200 and r.body == expected_large for r in parallel)
and len({r.stream_id for r in parallel}) == 4,
stream_ids=[r.stream_id for r in parallel], response_bytes=[len(r.body) for r in parallel])
if source == "reverse":
for payload in [b"", (bytes(range(256)) * 512) + b"streamed-end"]:
response = await session.request("POST", "/echo?transport=preserved",
{"content-type": "application/octet-stream"}, payload)
data = json.loads(response.body) if response.status == 200 else {}
check(prefix + f".post_stream_{len(payload)}", response.status == 200
and data.get("method") == "POST" and data.get("body_bytes") == len(payload)
and data.get("body_sha256") == hashlib.sha256(payload).hexdigest()
and data.get("path") == "/echo?transport=preserved",
upload_bytes=len(payload), **response_details(response))
else:
response = await session.request("POST", path, body=b"static-method-gate")
check(prefix + ".post_static_method_gate", response.status == 405, **response_details(response))
async def websocket_cases(check, port, cache):
import websockets
prefix = f"h1.websocket.cache_{'on' if cache else 'off'}"
async with websockets.connect(f"ws://127.0.0.1:{port}/echo", open_timeout=5,
close_timeout=3, ping_interval=None, compression=None) as session:
text = "RustProxy echo: Zażółć gęślą jaźń"
await session.send(text)
returned = await asyncio.wait_for(session.recv(), timeout=5)
check(prefix + ".text_echo", returned == text)
binary = bytes(range(256)) * 256
await session.send(binary)
returned = await asyncio.wait_for(session.recv(), timeout=5)
check(prefix + ".binary_echo", returned == binary, bytes=len(returned))
pong = await session.ping(b"rustproxy-transport")
await asyncio.wait_for(pong, timeout=5)
check(prefix + ".ping_pong", True)
await session.close(code=1000, reason="fixture complete")
check(prefix + ".normal_close", session.close_code == 1000, close_code=session.close_code)
async def run_variant(binary, label, output, workers):
import websockets
from transport_clients import client
cases = []
started = time.monotonic()
def check(case, passed, **details):
cases.append({"case": case, "passed": bool(passed), **details})
print(f"{label} {'PASS' if passed else 'FAIL'} {case}", flush=True)
write_json(output / "checks.json", cases)
try:
with Fixture(output, workers=workers) as fixture:
plain = (fixture.site / "small.html").read_bytes()
for name in ["transport-h2.html", "transport-h3.html", "transport-shared.html"]:
(fixture.site / name).write_bytes(plain)
peer = Peer(fixture.origin_port, fixture.site)
websocket_origin, websocket_port = fixture.port(), fixture.port()
async def echo(session, _path=None):
async for message in session:
await session.send(message)
try:
async with websockets.serve(echo, "127.0.0.1", websocket_origin,
ping_interval=None, compression=None, close_timeout=3):
rust = fixture.rust(binary, label)
reverse_tls, static_tls = tls_port(fixture), tls_port(fixture)
config = transport_config(rust, reverse_tls, static_tls, websocket_port, websocket_origin)
rust.apply(config, "transport-listeners")
fixture.stop(rust.process)
rust.start("transport")
certificate = rust.data / "certs" / "default.pem"
write_json(output / "transport.json", {
"certificate_sha256": sha256(certificate), "certificate_verification": True,
"h2_alpn_required": "h2", "h3_alpn_required": "h3",
"reverse_tls_tcp_udp": reverse_tls, "static_tls_tcp_udp": static_tls,
"websocket_h1_tcp": websocket_port, "fallback_allowed": False})
for cache in [False, True]:
if cache:
config["cache"]["enabled"] = True
rust.apply(config, "transport-cache-on")
for protocol in ["h2", "h3"]:
for source, port in [("reverse", reverse_tls), ("static", static_tls)]:
group = f"{protocol}.{source}.cache_{'on' if cache else 'off'}"
try:
async with client(protocol, port, certificate) as session:
await http_cases(check, session, protocol, source, cache, fixture)
except Exception as exc:
check(group + ".runner", False, error=str(exc), traceback=traceback.format_exc())
try:
await websocket_cases(check, websocket_port, cache)
except Exception as exc:
check(f"h1.websocket.cache_{'on' if cache else 'off'}.runner", False,
error=str(exc), traceback=traceback.format_exc())
for source, port in [("reverse", reverse_tls), ("static", static_tls)]:
try:
async with client("h2", port, certificate) as h2:
warm = await h2.request("GET", "/transport-shared.html")
async with client("h3", port, certificate) as h3:
hit = await h3.request("GET", "/transport-shared.html")
check(f"cache_cross_protocol.{source}.h2_to_h3",
warm.status == 200 and hit.status == 200 and hit.body == plain
and hit.headers.get("x-cache") == "HIT",
first_cache=warm.headers.get("x-cache"), **response_details(hit))
except Exception as exc:
check(f"cache_cross_protocol.{source}.runner", False,
error=str(exc), traceback=traceback.format_exc())
check("runtime.still_alive", rust.process.poll() is None)
finally:
peer.close()
except Exception as exc:
check("runtime.runner", False, error=str(exc), traceback=traceback.format_exc())
report = {"label": label, "simulation": False, "seconds": time.monotonic() - started,
"passed": sum(case["passed"] for case in cases),
"failed": sum(not case["passed"] for case in cases), "cases": cases}
write_json(output / "report.json", report)
print(f"{label}: {report['passed']} passed, {report['failed']} failed", flush=True)
return report
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--binary", action="append", required=True, metavar="LABEL=/ABS/PATH")
parser.add_argument("--out", type=Path, required=True)
parser.add_argument("--workers", type=int, default=4)
parser.add_argument("--python-deps", type=Path)
args = parser.parse_args()
if args.python_deps:
sys.path.insert(0, str(args.python_deps.resolve()))
variants = {}
for value in args.binary:
label, separator, path = value.partition("=")
if not separator or not label.replace("-", "").replace("_", "").isalnum() or label in variants:
parser.error("--binary must be unique SIMPLE_LABEL=/ABS/PATH")
variants[label] = Path(path).resolve()
args.out = args.out.resolve()
args.out.mkdir(parents=True, exist_ok=True)
from transport_clients import validate_head_adapter
write_json(args.out / "environment.json", {
"kind": "real RustProxy transport acceptance", "simulation": False,
"python": sys.version, "platform": platform.platform(),
"dependencies": {name: importlib.metadata.version(name) for name in ["aioquic", "httpx", "h2", "websockets"]},
"aioquic_head_adapter": validate_head_adapter(),
"started_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
"not_covered": ["production certificates", "TLS upstreams", "H3 upstreams", "H2/H3 WebSocket extended CONNECT",
"network loss or migration", "24-hour soak", "real-browser navigation"]})
reports = {}
for label, binary in variants.items():
reports[label] = asyncio.run(run_variant(binary, label, args.out / label, args.workers))
write_json(args.out / "comparison.json", {
"variants": {label: {key: report[key] for key in ["passed", "failed", "seconds"]}
for label, report in reports.items()},
"cases": [{"case": case, **{label: next((row["passed"] for row in report["cases"] if row["case"] == case), "not_run")
for label, report in reports.items()}}
for case in sorted({row["case"] for report in reports.values() for row in report["cases"]})]})
seal_results(args.out)
return 1 if any(report["failed"] for report in reports.values()) else 0
if __name__ == "__main__":
sys.exit(main())
scripts/performance/wrk_report.lua
+14
View File
@@ -0,0 +1,14 @@
-- No response hook: benchmarks must not be limited by Lua body processing.
-- Known payload/status/headers are verified separately before measurement.
done = function(summary, latency, requests)
io.write("RUSTPROXY_BENCH_JSON {")
io.write(string.format('"requests":%d,"duration_us":%d,"bytes":%d,',
summary.requests, summary.duration, summary.bytes))
io.write(string.format('"p50_us":%.3f,"p95_us":%.3f,"p99_us":%.3f,',
latency:percentile(50), latency:percentile(95), latency:percentile(99)))
io.write(string.format('"latency_mean_us":%.3f,"latency_max_us":%.3f,', latency.mean, latency.max))
io.write(string.format('"errors_connect":%d,"errors_read":%d,"errors_write":%d,"errors_timeout":%d,"errors_status":%d',
summary.errors.connect, summary.errors.read, summary.errors.write,
summary.errors.timeout, summary.errors.status))
io.write("}\n")
end
scripts/tests/upgrade_performance_test.py
Executable
+80
View File
@@ -0,0 +1,80 @@
#!/usr/bin/env python3
"""Run source-upgrade preservation/refusal regressions in disposable directories."""
from pathlib import Path
import shutil
import sqlite3
import sys
import tempfile
ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(ROOT))
from scripts.deploy.upgrade import upgrade
def fixture(parent):
target = parent / "installed"
target.mkdir()
(target / "Cargo.toml").write_text('[package]\nname = "rust-proxy"\nversion = "0.33.7"\n')
(target / "src").mkdir()
(target / "src" / "obsolete.rs").write_text("// old managed source\n")
(target / "data").mkdir()
with sqlite3.connect(target / "data" / "fixture.sqlite") as db:
db.execute("CREATE TABLE preserved(value TEXT)")
db.execute("INSERT INTO preserved VALUES ('configuration and preferences')")
(target / ".env").write_text("LOCAL_FIXTURE=1\n")
(target / "langs").mkdir()
(target / "langs" / "custom.json").write_text('{"custom": true}\n')
return target
def main():
with tempfile.TemporaryDirectory(prefix="rustproxy-upgrade-test-", dir=ROOT.parent) as temporary:
parent = Path(temporary)
target = fixture(parent)
try:
upgrade(ROOT, target, stopped=False)
except ValueError:
pass
else:
raise AssertionError("upgrade must require stopped confirmation")
assert (target / "src" / "obsolete.rs").is_file()
print("PASS upgrade refuses missing --stopped without modifying source")
if sys.platform != "win32":
import fcntl
with (target / "data" / "process.lock").open("w+") as lock:
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
try:
upgrade(ROOT, target, stopped=True)
except ValueError:
pass
else:
raise AssertionError("upgrade must refuse a live process lock")
print("PASS upgrade refuses a data directory locked by a live process")
backup = upgrade(ROOT, target, stopped=True)
assert (target / ".env").read_text() == "LOCAL_FIXTURE=1\n"
assert (target / "langs" / "custom.json").read_text() == '{"custom": true}\n'
with sqlite3.connect(target / "data" / "fixture.sqlite") as db:
assert db.execute("SELECT value FROM preserved").fetchone() == ("configuration and preferences",)
assert not (target / "src" / "obsolete.rs").exists()
assert (backup / "src" / "obsolete.rs").is_file()
assert (target / "Cargo.toml").read_bytes() == (ROOT / "Cargo.toml").read_bytes()
assert (target / "src" / "proxy" / "cache_plan.rs").is_file()
print("PASS upgrade preserves SQLite, environment and custom language; backs up old source and replaces managed tree")
if sys.platform != "win32":
shutil.rmtree(target / "src")
(target / "src").symlink_to(backup / "src", target_is_directory=True)
try:
upgrade(ROOT, target, stopped=True)
except ValueError:
pass
else:
raise AssertionError("upgrade must refuse a managed symlink")
assert (backup / "src" / "obsolete.rs").is_file()
print("PASS upgrade refuses managed symlinks without changing their targets")
if __name__ == "__main__":
main()
src/acme/providers/tests.rs
+1
View File
@@ -7,6 +7,7 @@ mod challenge_tests {
async fn fixture() -> (Arc<App>, tempfile::TempDir) {
let directory = tempfile::tempdir().unwrap();
let certs = directory.path().join("certs");
crate::tls::initialize_default(&certs).unwrap();
let root = certs.join("acme-http01/.well-known/acme-challenge");
std::fs::create_dir_all(&root).unwrap();
std::fs::write(root.join("sample_token"), b"sample_token.thumbprint").unwrap();
src/admin/certificates.rs
+7 -6
View File
@@ -4,12 +4,6 @@ async fn certificate_deploy(
certificate_api_access: Option<Extension<CertificateApiAccess>>,
Json(input): Json<CertificateDeploy>,
) -> Result<Json<Value>, ApiError> {
let directory = state.policy.cert_dir.clone().ok_or_else(|| {
ApiError(
StatusCode::CONFLICT,
"certificate storage is unavailable; configure the certificate directory".into(),
)
})?;
let CertificateDeploy {
certificate_id,
cert_pem,
@@ -30,6 +24,13 @@ async fn certificate_deploy(
));
}
}
// Check the credential scope before disclosing certificate storage state.
let directory = state.policy.cert_dir.clone().ok_or_else(|| {
ApiError(
StatusCode::CONFLICT,
"certificate storage is unavailable; configure the certificate directory".into(),
)
})?;
let snapshot = state.config.load_full();
let expected_revision = snapshot.envelope.revision;
src/admin/tests.rs
+69 -37
View File
@@ -901,13 +901,19 @@ mod tests {
.oneshot(request(path).body(Body::empty()).unwrap())
.await
.unwrap();
assert_eq!(public.status(), StatusCode::OK, "anonymous {path}");
let public_body = axum::body::to_bytes(public.into_body(), 100_000)
.await
.unwrap();
assert!(std::str::from_utf8(&public_body)
.unwrap()
.contains(&format!("/assets/login_{}.js", super::asset_build_id())));
if path == "/logs" {
// The operational log route is private even on browser refresh.
assert_eq!(public.status(), StatusCode::UNAUTHORIZED);
assert_eq!(json_body(public).await, json!({"error":"auth.required"}));
} else {
assert_eq!(public.status(), StatusCode::OK, "anonymous {path}");
let public_body = axum::body::to_bytes(public.into_body(), 100_000)
.await
.unwrap();
assert!(std::str::from_utf8(&public_body)
.unwrap()
.contains(&format!("/assets/login_{}.js", super::asset_build_id())));
}
let private = router
.clone()
@@ -1314,6 +1320,13 @@ mod tests {
.unwrap();
assert_eq!(response.status(), StatusCode::FORBIDDEN);
// This fixture has no certificate directory. An out-of-scope credential
// must see only the grant error, never the storage availability error.
assert_eq!(
json_body(response).await,
json!({"error":"certificate_api.certificate_forbidden"})
);
let missing_key = router
.clone()
.oneshot(
@@ -1821,6 +1834,32 @@ mod tests {
"backend_table_sort":"requests",
"backend_table_descending":true
});
// Older clients may omit fields added since their preferences were
// saved. The API returns and stores the typed, defaulted document.
let normalized = serde_json::to_value(
serde_json::from_value::<AdminPreferences>(selected.clone()).unwrap(),
)
.unwrap();
for (field, value) in selected.as_object().unwrap() {
assert_eq!(&normalized[field], value, "selected field {field}");
}
assert_eq!(normalized["dashboard_widget_sizes"], Value::Null);
assert_eq!(normalized["file_log_sort"], "time");
assert_eq!(normalized["file_log_descending"], true);
assert_eq!(normalized["vhost_list_sort"], "id");
assert_eq!(normalized["docker_vhost_sort"], "service");
assert_eq!(normalized["stream_list_sort"], "name");
assert_eq!(normalized["redirect_list_sort"], "id");
assert_eq!(normalized["notfound_list_sort"], "id");
for field in [
"vhost_list_descending",
"docker_vhost_descending",
"stream_list_descending",
"redirect_list_descending",
"notfound_list_descending",
] {
assert_eq!(normalized[field], false, "default field {field}");
}
let response = router
.clone()
.oneshot(
@@ -1837,12 +1876,12 @@ mod tests {
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(json_body(response).await, selected);
assert_eq!(json_body(response).await, normalized);
let saved: Value =
serde_json::from_str(&store.admin_preferences("owner").await.unwrap().unwrap())
.unwrap();
assert_eq!(saved, selected);
assert_eq!(saved, normalized);
// Preferences must also survive a subsequent authenticated read.
let persisted = router
.clone()
@@ -1854,35 +1893,22 @@ mod tests {
)
.await
.unwrap();
let mut upgraded = selected.clone();
for id in [
"stream_overview",
"stream_traffic",
"stream_sessions",
"stream_backend_health",
"backend_health",
"backend_activity",
"backend_errors",
"backend_latency",
"backend_traffic",
"backend_protocols",
"backend_requests",
"backend_capacity",
"backend_http5xx",
"backend_timeouts",
"backend_network",
"backend_retries",
"backend_queue",
"backend_rotations",
] {
let mut upgraded = normalized.clone();
for widget in DASHBOARD_WIDGETS.iter() {
let id = json!(widget.id);
if selected["dashboard_widgets"].as_array().unwrap().contains(&id) {
continue;
}
upgraded["dashboard_widgets"]
.as_array_mut()
.unwrap()
.push(json!(id));
upgraded["dashboard_hidden_widgets"]
.as_array_mut()
.unwrap()
.push(json!(id));
.push(id.clone());
if widget.default_hidden {
upgraded["dashboard_hidden_widgets"]
.as_array_mut()
.unwrap()
.push(id);
}
}
assert_eq!(json_body(persisted).await, upgraded);
// Wrong allowlist value is rejected even if the rest is valid.
@@ -2192,7 +2218,10 @@ mod tests {
.await
.unwrap();
assert_eq!(response.status(), StatusCode::UNAUTHORIZED, "{path}");
assert_eq!(response.headers()["cache-control"], "no-store");
assert_eq!(
response.headers()["cache-control"],
"private, no-store, no-cache, no-transform"
);
assert!(!response
.headers()
.contains_key("access-control-allow-origin"));
@@ -2294,7 +2323,10 @@ mod tests {
.unwrap();
assert_eq!(response.status(), StatusCode::OK, "{path}");
assert_eq!(response.headers()["x-frame-options"], "DENY");
assert_eq!(response.headers()["cache-control"], "no-store");
assert_eq!(
response.headers()["cache-control"],
"private, no-store, no-cache, no-transform"
);
}
let response = router
.clone()
src/backup/tests.rs
+15 -1
View File
@@ -7,12 +7,26 @@ mod tests {
let input = dir.path().join("input");
let encrypted = dir.path().join("backup.rpb");
let output = dir.path().join("output");
let wrong_output = dir.path().join("wrong-password-output");
let tampered = dir.path().join("tampered.rpb");
let tampered_output = dir.path().join("tampered-output");
fs::write(&input, b"private backup payload").unwrap();
let crypto = new_crypto("correct horse battery staple").unwrap();
encrypt_archive(&input, &encrypted, &crypto).unwrap();
assert!(decrypt_archive(&encrypted, &output, "wrong password").is_err());
let error = decrypt_archive(&encrypted, &wrong_output, "wrong password").unwrap_err();
assert!(error.to_string().contains("incorrect backup password or corrupted archive"));
decrypt_archive(&encrypted, &output, "correct horse battery staple").unwrap();
assert_eq!(fs::read(&output).unwrap(), b"private backup payload");
let mut bytes = fs::read(&encrypted).unwrap();
// Corrupt the last authentication-tag byte, before the four-byte EOF marker.
let tag_byte = bytes.len() - 5;
bytes[tag_byte] ^= 1;
fs::write(&tampered, bytes).unwrap();
let error = decrypt_archive(&tampered, &tampered_output, "correct horse battery staple")
.unwrap_err();
assert!(error.to_string().contains("incorrect backup password or corrupted archive"));
assert_eq!(fs::read(&output).unwrap(), b"private backup payload");
}
#[test]
src/cache/dashboard.rs
Vendored
+3 -4
View File
@@ -323,7 +323,9 @@ impl Cache {
continue;
};
let age = e.initial_age.saturating_add(e.received.elapsed()).as_secs();
if age >= e.ttl || !scope.is_none_or(|s| s == e.scope) {
// This is an inventory of retained objects, not a freshness lookup.
// Expired representations still occupy RAM and may be revalidated.
if !scope.is_none_or(|s| s == e.scope) {
continue;
}
let row = groups
@@ -373,9 +375,6 @@ impl Cache {
continue;
}
let age = cold_age(e);
if age >= e.ttl && !e.requires_revalidation {
continue;
}
let row = groups
.entry((e.scope.clone(), e.domain.clone()))
.or_insert_with(|| DomainView {
src/cache/lookup/tiered.rs
Vendored
+7
View File
@@ -205,6 +205,13 @@ impl Cache {
_permit: permit,
_fill_leader: fill_leader,
};
// A genuinely finished body (including an empty local file) may never
// be polled by Hyper. EOS is the proof here, not Content-Length alone;
// finish still enforces expected length, privacy and ticket validity.
if body.is_end_stream() {
candidate.finish();
return Response::from_parts(parts, body);
}
Response::from_parts(
parts,
Body::new(FillBody {
src/cache/tests/cold_tier_tests.rs
Vendored
+31
View File
@@ -134,6 +134,37 @@ async fn lru_demotion_persists_and_survives_restart() {
restart_shutdown.cancel();
}
#[tokio::test]
async fn dashboard_lists_retained_expired_cold_objects_without_permitting_fresh_hits() {
let dir = tempfile::tempdir().unwrap();
let settings = cold_settings();
let shutdown = CancellationToken::new();
let cache = Cache::start(11, Some(dir.path().to_path_buf()), settings.clone(), shutdown.clone());
let first = fill(&cache, &settings, "/one", "first-payload").await;
fill(&cache, &settings, "/two", "second-payload").await;
wait_for_cold(&cache, 1).await;
{
let mut cold = cache.cold.lock().unwrap();
for entry in cold.entries.values_mut() {
entry.stored_at = SystemTime::now() - Duration::from_secs(2 * 60 * 60);
}
}
let dashboard = cache.dashboard(Some("cold-test"), 0, 50);
assert_eq!(dashboard.matching_objects, 2);
let row = dashboard.objects.iter().find(|row| row.path == "/one").unwrap();
assert_eq!(row.tier, "cold");
assert_eq!(row.remaining_ttl_seconds, 0);
assert!(row.age_seconds >= 2 * 60 * 60);
assert_eq!(row.storage_bytes, cache.view().cold_bytes);
assert!(cache.get_tiered(&first, &settings).await.is_none());
let mut stale = first;
stale.request_headers.insert("cache-control", HeaderValue::from_static("max-stale"));
let hit = cache.get_tiered(&stale, &settings).await.expect("explicit max-stale can reuse the retained disk object");
assert!(hit.stale);
assert_eq!(hit.response.into_body().collect().await.unwrap().to_bytes(), "first-payload");
shutdown.cancel();
}
#[tokio::test]
async fn cold_restart_preserves_transformed_validators_stale_metadata_and_no_transform() {
let dir = tempfile::tempdir().unwrap();
src/cache/tests/dashboard_tests.rs
Vendored
+9 -2
View File
@@ -78,7 +78,7 @@ mod dashboard_tests {
#[tokio::test]
async fn dashboard_retains_expired_entries_for_client_max_stale() {
let cache = Cache::new(7);
fill(&cache, "a", "a.example", "/").await;
let ticket = fill(&cache, "a", "a.example", "/").await;
{
let mut store = cache.store.lock().unwrap();
for entry in store.entries.values_mut() {
@@ -86,9 +86,16 @@ mod dashboard_tests {
}
store.last_sweep = None;
}
assert_eq!(cache.dashboard(None, 0, 50).matching_objects, 1);
let dashboard = cache.dashboard(None, 0, 50);
assert_eq!(dashboard.matching_objects, 1);
assert_eq!(dashboard.objects[0].remaining_ttl_seconds, 0);
assert!(dashboard.objects[0].age_seconds >= 2 * 60 * 60);
assert_eq!(cache.view().entries, 1);
assert_eq!(cache.view().expired, 0);
assert!(cache.get(&ticket).is_none(), "inventory visibility cannot permit unconditional stale delivery");
let mut stale = ticket;
stale.request_headers.insert("cache-control", HeaderValue::from_static("max-stale"));
assert!(cache.get(&stale).unwrap().stale);
}
#[test]
fn bypass_reasons_are_fixed_and_private_inputs_are_not_echoed() {
src/cache/tests/tests.rs
Vendored
+74 -13
View File
@@ -23,10 +23,8 @@ mod tests {
for control in [
"private, max-age=60",
"no-store, max-age=60",
"max-age=0",
"max-age=invalid",
"max-age=60,max-age=90",
"public",
"must-understand, max-age=60",
"max-age=\"60",
"max-age=60\"",
@@ -53,13 +51,6 @@ mod tests {
let mut validator_only = headers("no-cache");
validator_only.insert("etag", HeaderValue::from_static("\"validator\""));
assert!(plan(StatusCode::OK, &validator_only, &s, Duration::ZERO).is_some());
assert!(plan(
StatusCode::OK,
&headers("max-age=60,s-maxage=0"),
&s,
Duration::ZERO
)
.is_none());
let shared = plan(
StatusCode::OK,
&headers("public, max-age=600, s-maxage=30"),
@@ -75,6 +66,24 @@ mod tests {
h.insert("set-cookie", "session=secret".parse().unwrap());
assert!(plan(StatusCode::OK, &h, &s, Duration::ZERO).is_none());
}
#[tokio::test]
async fn zero_freshness_is_retained_but_never_served_as_an_unconditional_hit() {
for control in ["max-age=0", "public", "max-age=60,s-maxage=0"] {
let cache = Cache::new(1);
let settings = settings();
let ticket = ticket(&cache, &HeaderMap::new());
let mut response = Response::new(Body::from("zero-freshness"));
*response.headers_mut() = headers(control);
let plan = plan(StatusCode::OK, response.headers(), &settings, Duration::ZERO)
.expect("valid zero freshness is distinct from a no-store response");
assert_eq!(plan.ttl, 0, "{control}");
cache.wrap(response, ticket.clone(), plan, &settings)
.into_body().collect().await.unwrap();
assert_eq!(cache.view().entries, 1, "{control}");
assert!(cache.get(&ticket).is_none(), "zero-TTL {control} cannot be an unconditional HIT");
}
}
#[test]
fn request_policy_uses_conditional_cache_and_revalidation() {
for name in [
@@ -1362,21 +1371,23 @@ mod tests {
&a,
)
.unwrap();
let other_transport = cache
// Compare unrelated headers/member choice within the same frontend
// scheme. HTTP and HTTPS intentionally have distinct cache identities.
let other_member = cache
.ticket(
1,
"https",
"http",
"example.test",
"pool/member-b",
"/asset.js?x=1",
&b,
)
.unwrap();
assert_eq!(base.key, other_transport.key);
assert_eq!(base.key, other_member.key);
let other_url = cache
.ticket(
1,
"https",
"http",
"example.test",
"pool/member-b",
"/asset.js?x=2",
@@ -1479,4 +1490,54 @@ mod tests {
assert_eq!(cache.view().fills_active, 0);
}
}
#[test]
fn empty_complete_body_is_cached_without_requiring_a_consumer_poll() {
let cache = Cache::new(1);
let settings = settings();
let ticket = ticket(&cache, &HeaderMap::new());
let mut response = Response::new(Body::empty());
*response.headers_mut() = headers("public, max-age=600");
response.headers_mut().insert("content-length", HeaderValue::from_static("0"));
let plan = plan(StatusCode::OK, response.headers(), &settings, Duration::ZERO).unwrap();
let response = cache.wrap(response, ticket.clone(), plan, &settings);
assert!(response.body().is_end_stream());
// HTTP/1.1 Content-Length: 0 does not ask the response body for frames.
drop(response);
assert!(cache.get(&ticket).is_some());
assert_eq!(cache.view().entries, 1);
assert_eq!(cache.view().fills_active, 0);
assert_eq!(cache.view().rejected_fills, 0);
}
#[test]
fn empty_body_with_nonzero_content_length_cannot_publish_a_fill() {
let cache = Cache::new(1);
let settings = settings();
let ticket = ticket(&cache, &HeaderMap::new());
let mut response = Response::new(Body::empty());
*response.headers_mut() = headers("public, max-age=600");
response.headers_mut().insert("content-length", HeaderValue::from_static("6"));
let plan = plan(StatusCode::OK, response.headers(), &settings, Duration::ZERO).unwrap();
drop(cache.wrap(response, ticket.clone(), plan, &settings));
assert!(cache.get(&ticket).is_none());
assert_eq!(cache.view().fills_active, 0);
assert_eq!(cache.view().rejected_fills, 1);
}
#[test]
fn empty_body_completion_still_obeys_purge_barrier() {
let cache = Cache::new(1);
let settings = settings();
let ticket = ticket(&cache, &HeaderMap::new());
let mut response = Response::new(Body::empty());
*response.headers_mut() = headers("public, max-age=600");
response.headers_mut().insert("content-length", HeaderValue::from_static("0"));
let plan = plan(StatusCode::OK, response.headers(), &settings, Duration::ZERO).unwrap();
cache.purge(None);
drop(cache.wrap(response, ticket.clone(), plan, &settings));
assert!(cache.get(&ticket).is_none());
assert_eq!(cache.view().entries, 0);
assert_eq!(cache.view().fills_active, 0);
}
}
src/cache/tests/v022_cache_tests.rs
Vendored
+17 -6
View File
@@ -556,6 +556,13 @@ mod v022_cache_tests {
#[tokio::test]
async fn selective_purge_exact_prefix_tag_cache_key_and_query_are_precise() {
fn current_ticket(cache: &Cache, old: &Ticket) -> Ticket {
// Each post-PURGE request obtains the current publication generation.
// Reusing an old ticket would reject unrelated entries too and make
// assertions about removed objects pass without testing the selector.
cache.ticket_scoped(22, &old.scope, &old.scheme, &old.authority,
"origin@127.0.0.1:8080", &old.target, &old.request_headers).unwrap()
}
let cache = Cache::new(22);
let one = store(
&cache,
@@ -587,8 +594,9 @@ mod v022_cache_tests {
))
.unwrap();
assert_eq!(result.affected_entries, 1);
assert!(cache.get(&one).is_none());
assert!(cache.get(&two).is_some());
assert!(cache.get(&current_ticket(&cache, &one)).is_none());
let survivor = cache.get(&current_ticket(&cache, &two)).expect("unrelated query remains stored");
assert_eq!(survivor.response.into_body().collect().await.unwrap().to_bytes(), "two");
let three = store(
&cache,
@@ -623,8 +631,9 @@ mod v022_cache_tests {
.affected_entries,
2
);
assert!(cache.get(&three).is_none());
assert!(cache.get(&four).is_none());
assert!(cache.get(&current_ticket(&cache, &three)).is_none());
assert!(cache.get(&current_ticket(&cache, &four)).is_none());
assert!(cache.get(&current_ticket(&cache, &two)).is_some());
let tagged = store(
&cache,
@@ -649,7 +658,8 @@ mod v022_cache_tests {
.affected_entries,
1
);
assert!(cache.get(&tagged).is_none());
assert!(cache.get(&current_ticket(&cache, &tagged)).is_none());
assert!(cache.get(&current_ticket(&cache, &two)).is_some());
let keyed = store(
&cache,
@@ -675,7 +685,8 @@ mod v022_cache_tests {
.affected_entries,
1
);
assert!(cache.get(&keyed).is_none());
assert!(cache.get(&current_ticket(&cache, &keyed)).is_none());
assert!(cache.get(&current_ticket(&cache, &two)).is_some());
}
#[tokio::test]
src/compression.rs
+13
View File
@@ -173,6 +173,19 @@ pub fn no_transform(headers: &HeaderMap) -> bool {
})
}
/// Preserve exactly the request inputs used by encoding after forwarding has
/// consumed the request. Repeated field values and opaque bytes stay intact;
/// unrelated cookies, authorization and custom headers do not need a copy.
pub(crate) fn request_headers(headers: &HeaderMap) -> HeaderMap {
let mut selected = HeaderMap::new();
for name in ["accept-encoding", "range", "cache-control"] {
for value in headers.get_all(name) {
selected.append(name, value.clone());
}
}
selected
}
fn eligible(
response: &Response,
request: &HeaderMap,
src/compression/tests.rs
+32
View File
@@ -24,6 +24,38 @@ mod tests {
assert!(settings.enabled());
}
#[tokio::test]
async fn narrowed_request_headers_preserve_encoding_and_skip_decisions() {
for accept in ["gzip", "br;q=0.8, gzip;q=0.5", "zstd", "identity", "*;q=0"] {
for method in [Method::GET, Method::HEAD, Method::POST] {
for skip in [None, Some("range"), Some("cache-control")] {
let mut headers = HeaderMap::new();
headers.append("accept-encoding", accept.parse().unwrap());
headers.append("accept-encoding", "deflate;q=0".parse().unwrap());
headers.append("cache-control", "max-age=0".parse().unwrap());
if let Some(name) = skip {
headers.append(name, if name == "range" { "bytes=0-3" } else { "no-transform" }.parse().unwrap());
}
headers.insert("cookie", "session=private".parse().unwrap());
headers.insert("authorization", "Bearer test".parse().unwrap());
headers.insert("x-large-custom", "x".repeat(4096).parse().unwrap());
let selected = request_headers(&headers);
for name in ["accept-encoding", "range", "cache-control"] {
assert_eq!(headers.get_all(name).iter().collect::<Vec<_>>(), selected.get_all(name).iter().collect::<Vec<_>>());
}
assert!(!selected.contains_key("cookie"));
assert!(!selected.contains_key("authorization"));
let full = encode(response(), &headers, &method, &Settings::default()).await;
let narrow = encode(response(), &selected, &method, &Settings::default()).await;
assert_eq!(full.status(), narrow.status());
assert_eq!(full.headers(), narrow.headers());
assert_eq!(full.into_body().collect().await.unwrap().to_bytes(), narrow.into_body().collect().await.unwrap().to_bytes());
}
}
}
assert!(request_headers(&HeaderMap::new()).is_empty());
}
#[test]
fn scoped_codec_overrides_keep_global_codecs_authoritative() {
let global = Settings::default();
src/config/snapshot.rs
+12 -7
View File
@@ -392,14 +392,19 @@ pub fn normalize_host_pattern(value: &str) -> Result<String> {
pub fn request_host(value: &str) -> Result<String> {
ensure!(!value.contains('@'), "invalid Host");
let authority: Authority = value.parse().context("invalid Host authority")?;
if let Some(port) = authority.port() {
let number: u16 = port.as_str().parse().context("invalid Host port")?;
ensure!(number > 0, "invalid Host port");
// Authority::port() returns None for both an absent and an invalid port.
// Inspect the explicit suffix so an out-of-range port cannot bypass validation.
let raw_host = authority.host();
let suffix = &authority.as_str()[raw_host.len()..];
if !suffix.is_empty() {
let port = suffix.strip_prefix(':').context("invalid Host port")?;
// HTTP permits an empty port suffix, which selects the default port.
if !port.is_empty() {
let number: u16 = port.parse().context("invalid Host port")?;
ensure!(number > 0, "invalid Host port");
}
}
let host = authority
.host()
.trim_start_matches('[')
.trim_end_matches(']');
let host = raw_host.trim_start_matches('[').trim_end_matches(']');
if let Ok(ip) = host.parse::<IpAddr>() {
return Ok(ip.to_string());
}
src/config/tests.rs
+17
View File
@@ -142,6 +142,11 @@ fn missing_source_fields_remain_reverse_proxy_for_old_configs() {
#[test]
fn host_normalization() {
assert_eq!(request_host("EXAMPLE.com.:8080").unwrap(), "example.com");
assert_eq!(request_host("example.com:65535").unwrap(), "example.com");
assert_eq!(request_host("[::1]").unwrap(), "::1");
assert_eq!(request_host("[::1]:8080").unwrap(), "::1");
assert_eq!(request_host("a:").unwrap(), "a");
assert_eq!(request_host("[::1]:").unwrap(), "::1");
for invalid in [
"",
"a..b",
@@ -150,6 +155,12 @@ fn host_normalization() {
"user@host",
"-bad.test",
"a:99999",
"a:0",
"a:abc",
"[::1]:99999",
"[::1]:0",
"[::1]suffix",
"[::1]suffix:80",
] {
assert!(request_host(invalid).is_err(), "{invalid}");
}
@@ -516,6 +527,7 @@ fn invalid_per_backend_health_policy_rejected_without_global_override() {
mode: crate::balance::HealthMode::Http,
interval_ms: 300,
timeout_ms: 400,
slow_response_ms: 0,
..Default::default()
});
assert!(compile(c.clone(), 1, &policy()).is_err());
@@ -690,6 +702,9 @@ fn local_response_hosts_reserve_frontends_and_do_not_require_backends() {
#[test]
fn conflicts_are_rejected() {
let mut c = demo_config("127.0.0.1:9000".parse().unwrap());
// Keep both entries as real Custom Locations; the untouched legacy root
// is intentionally normalized away before duplicate-matcher validation.
c.vhosts[0].routes[0].path_prefix = "/api".into();
let mut duplicate = c.vhosts[0].routes[0].clone();
duplicate.id = "other".into();
c.vhosts[0].routes.push(duplicate);
@@ -800,6 +815,8 @@ fn http_health_check_defaults_host_to_first_exact_vhost_dns_name() {
fn per_member_http_health_inherits_vhost_dns_even_when_global_health_is_tcp() {
let mut c = demo_config("127.0.0.1:9000".parse().unwrap());
c.vhosts[0].frontend.hosts = vec!["app.example.test".into()];
// demo_config explicitly disables probes; this case exercises TCP fallback.
c.vhosts[0].load_balancing.health_check.mode = crate::balance::HealthMode::Tcp;
assert_eq!(c.vhosts[0].load_balancing.health_check.mode, crate::balance::HealthMode::Tcp);
c.vhosts[0].backends[0].health_check = Some(crate::balance::HealthCheck {
mode: crate::balance::HealthMode::Http, path: "/healthz".into(), ..Default::default()
src/headers.rs
+1
View File
@@ -73,6 +73,7 @@ fn protected(name: &str) -> bool {
| "forwarded"
| "x-real-ip"
| "x-request-id"
| "x-rustproxy-backend"
| "x-rust-proxy-backend"
| "x-rust-proxy-backend-time"
| "x-rustproxy-backend-time"
src/headers/tests.rs
+3
View File
@@ -22,6 +22,7 @@ mod tests {
"x-forwarded-for",
"Upgrade",
"X-RustProxy-Backend",
"X-Rust-Proxy-Backend",
"X-RustProxy-Backend-Time",
"X-Rust-Proxy-Backend-Time",
] {
@@ -34,6 +35,8 @@ mod tests {
response: vec![],
};
assert!(rules.compile().is_err(), "{name}");
let response_rules = HeaderRules { request: vec![], response: rules.request };
assert!(response_rules.compile().is_err(), "response {name}");
}
let rules = HeaderRules {
request: vec![Rule {
src/logging/error_runtime.rs
+9 -1
View File
@@ -33,12 +33,14 @@ fn redact_regex(pattern: &'static str) -> &'static regex::Regex {
static AUTH: OnceLock<regex::Regex> = OnceLock::new();
static QUERY: OnceLock<regex::Regex> = OnceLock::new();
static JSON: OnceLock<regex::Regex> = OnceLock::new();
static ESCAPED_JSON: OnceLock<regex::Regex> = OnceLock::new();
static KV: OnceLock<regex::Regex> = OnceLock::new();
static PEM: OnceLock<regex::Regex> = OnceLock::new();
match pattern {
"auth" => AUTH.get_or_init(|| regex::Regex::new(r"(?i)(authorization\s*[:=]\s*)(?:bearer\s+)?[^\s,;]+" ).expect("redaction regex")),
"query" => QUERY.get_or_init(|| regex::Regex::new(r"(?i)([?&](?:token|access_token|api_token|password|backup_password|secret|credential|credentials|key_secret|application_secret|consumer_key)=)[^&\s]+" ).expect("redaction regex")),
"json" => JSON.get_or_init(|| regex::Regex::new(r#"(?i)(\"(?:authorization|proxy_authorization|api_token|access_token|application_secret|consumer_key|key_secret|backup_password|password|credential|credentials|secret|token|private_key|master_key)\"\s*:\s*\")[^\"]*(\")"#).expect("redaction regex")),
"json" => JSON.get_or_init(|| regex::Regex::new(r#"(?i)("(?:authorization|proxy_authorization|api_token|access_token|application_secret|consumer_key|key_secret|backup_password|password|credential|credentials|secret|token|private_key|master_key)"\s*:\s*")(?:\\.|[^"\\])*(")"#).expect("redaction regex")),
"escaped_json" => ESCAPED_JSON.get_or_init(|| regex::Regex::new(r#"(?i)(\\"(?:authorization|proxy_authorization|api_token|access_token|application_secret|consumer_key|key_secret|backup_password|password|credential|credentials|secret|token|private_key|master_key)\\"\s*:\s*\\")(?:[^"\\]|\\\\(?:\\\\|\\"|[^"\\]))*(\\")"#).expect("redaction regex")),
"kv" => KV.get_or_init(|| regex::Regex::new(r#"(?i)((?:authorization|proxy[_-]?authorization|api[_-]?token|access[_-]?token|application[_-]?secret|consumer[_-]?key|key[_-]?secret|backup[_-]?password|password|credential|credentials|secret|token|private[_-]?key|master[_-]?key)\s*[=:]\s*)[^\s,;"']+"#).expect("redaction regex")),
"pem" => PEM.get_or_init(|| regex::Regex::new(r"(?s)-----BEGIN [^-\r\n]*PRIVATE KEY-----.*?-----END [^-\r\n]*PRIVATE KEY-----" ).expect("redaction regex")),
_ => unreachable!(),
@@ -76,6 +78,12 @@ pub fn redact_text(input: &str) -> String {
value = redact_regex("query")
.replace_all(&value, "$1[REDACTED]")
.into_owned();
// Debug-formatted fields can contain a JSON string with escaped quote
// delimiters. Redact that form before plain JSON, retaining quote escapes
// and consuming escaped quotes inside a secret instead of leaking its tail.
value = redact_regex("escaped_json")
.replace_all(&value, "$1[REDACTED]$2")
.into_owned();
value = redact_regex("json")
.replace_all(&value, "$1[REDACTED]$2")
.into_owned();
src/logging/format.rs
+13 -6
View File
@@ -136,14 +136,21 @@ impl Logs {
self.settings.load().capture_country
}
pub fn push(&self, entry: LogEntry) {
if let Ok(mut entries) = self.entries.try_lock() {
if entries.len() == HISTORY_CAPACITY {
entries.pop_front();
}
entries.push_back(entry.clone());
// String cloning and eviction cleanup do not need the history mutex.
let history_entry = entry.clone();
let evicted = if let Ok(mut entries) = self.entries.try_lock() {
let evicted = if entries.len() == HISTORY_CAPACITY {
entries.pop_front()
} else {
None
};
entries.push_back(history_entry);
evicted
} else {
self.dropped.fetch_add(1, Ordering::Relaxed);
}
None
};
drop(evicted);
let _ = self.live.send(entry.clone());
self.push_sink(entry);
}
src/logging/tests.rs
+131
View File
@@ -209,6 +209,41 @@ mod tests {
assert!(!text.contains("pem-data"));
}
#[test]
fn error_redaction_preserves_json_and_debug_escaping_without_leaking_secret_suffixes() {
for secret in ["pem-data", "left\"right", "left\\right", "ending\\", "left\\\"right", "first\nsecond"] {
let plain = serde_json::to_string(&serde_json::json!({
"private_key": secret, "visible": "keep"
})).unwrap();
let redacted = redact_text(&plain);
let value: Value = serde_json::from_str(&redacted).unwrap();
assert_eq!(value["private_key"], "[REDACTED]");
assert_eq!(value["visible"], "keep");
// EventVisitor::record_debug uses this exact formatting operation.
let debug = format!("{plain:?}");
let debug_redacted = redact_text(&debug);
let decoded: String = serde_json::from_str(&debug_redacted).unwrap();
let value: Value = serde_json::from_str(&decoded).unwrap();
assert_eq!(value["private_key"], "[REDACTED]");
assert_eq!(value["visible"], "keep");
assert_eq!(
redact_text(&format!("before {debug} after")),
format!("before {debug_redacted} after")
);
}
}
#[test]
fn error_redaction_removes_private_pem_and_preserves_public_diagnostics() {
for label in ["PRIVATE KEY", "RSA PRIVATE KEY", "EC PRIVATE KEY", "ENCRYPTED PRIVATE KEY"] {
let input = format!("before -----BEGIN {label}-----\nsynthetic-secret\n-----END {label}----- after");
assert_eq!(redact_text(&input), "before [REDACTED PRIVATE KEY] after");
}
let public = "-----BEGIN CERTIFICATE-----\npublic-test-data\n-----END CERTIFICATE-----";
assert_eq!(redact_text(public), public);
}
#[test]
fn error_json_is_one_record_and_reserved_fields_cannot_be_overwritten() {
let event = ErrorEvent {
@@ -291,4 +326,100 @@ mod tests {
LIVE_WINDOW + 80
);
}
#[test]
fn full_history_evicts_oldest_keeps_exact_fields_and_does_not_count_eviction_as_drop() {
fn history_entry(id: usize) -> LogEntry {
let mut item = entry();
item.id = id as u64;
item.time_ms += id as u64;
item.path = format!("/history/{id}");
item.authority = Some("app.example:443".into());
item.matched_by = Some("host");
item.upstream_error = Some("diagnostic with \"quotes\" and UTF-8: €".into());
item
}
let logs = Logs::default();
for id in 0..HISTORY_CAPACITY + 3 {
logs.push(history_entry(id));
}
let retained = logs.recent_limit(HISTORY_CAPACITY + 1);
assert_eq!(retained.len(), HISTORY_CAPACITY);
assert_eq!(
retained.iter().map(|item| item.id).collect::<Vec<_>>(),
(3..HISTORY_CAPACITY + 3).rev().map(|id| id as u64).collect::<Vec<_>>()
);
for item in [&retained[0], &retained[HISTORY_CAPACITY / 2], retained.last().unwrap()] {
assert_eq!(
serde_json::to_value(item).unwrap(),
serde_json::to_value(history_entry(item.id as usize)).unwrap()
);
}
assert_eq!(logs.recent_limit(0).len(), 0);
assert_eq!(logs.recent_limit(2)[0].id, (HISTORY_CAPACITY + 2) as u64);
assert_eq!(logs.dropped.load(Ordering::Relaxed), 0);
let stats = logs.stats();
assert_eq!(stats.queue_capacity, SINK_QUEUE);
assert_eq!(stats.queue_dropped, 0, "disabled sinks must not enqueue");
assert_eq!(stats.sent, 0);
assert_eq!(stats.errors, 0);
}
#[tokio::test]
async fn history_contention_counts_one_drop_but_preserves_live_and_file_delivery() {
let directory = tempfile::tempdir().unwrap();
let path = directory.path().join("access.jsonl");
let mut settings = Settings::default();
settings.file.enabled = true;
settings.file.path = path.to_string_lossy().into_owned();
let shutdown = CancellationToken::new();
let logs = Logs::start(settings, shutdown.clone());
let mut live = logs.subscribe();
let mut sentinel = entry();
sentinel.id = 6;
let mut history = logs.entries.lock().unwrap();
history.push_back(sentinel.clone());
let mut item = entry();
item.path = "/contention/€\"quoted\"".into();
item.authority = Some("app.example:443".into());
item.matched_by = Some("host");
let expected = serde_json::to_value(&item).unwrap();
let producer = logs.clone();
let (finished, completion) = std::sync::mpsc::channel();
let worker = std::thread::spawn(move || {
producer.push(item);
finished.send(()).unwrap();
});
// Release the guard before joining even on failure, so an accidental
// blocking-lock regression fails this test instead of hanging the suite.
let completed = completion.recv_timeout(Duration::from_secs(2));
drop(history);
worker.join().unwrap();
completed.expect("history contention must not block live/sink delivery");
assert_eq!(logs.dropped.load(Ordering::Relaxed), 1);
assert_eq!(serde_json::to_value(logs.recent()).unwrap(), json!([sentinel]));
assert_eq!(serde_json::to_value(live.try_recv().unwrap()).unwrap(), expected);
assert!(matches!(live.try_recv(), Err(broadcast::error::TryRecvError::Empty)));
let persisted = tokio::time::timeout(Duration::from_secs(5), async {
loop {
if let Ok(content) = tokio::fs::read_to_string(&path).await {
if content.ends_with('\n') && logs.stats().sent == 1 {
break content;
}
}
tokio::time::sleep(Duration::from_millis(20)).await;
}
}).await.expect("contended history entry must still reach the configured file sink");
shutdown.cancel();
assert_eq!(persisted.lines().count(), 1);
assert_eq!(serde_json::from_str::<Value>(persisted.trim_end()).unwrap(), expected);
let stats = logs.stats();
assert_eq!(stats.queue_capacity, SINK_QUEUE);
assert_eq!(stats.queue_dropped, 0);
assert_eq!(stats.sent, 1);
assert_eq!(stats.errors, 0);
}
}
src/not_found/tests.rs
+4 -2
View File
@@ -34,7 +34,9 @@ mod tests {
#[test]
fn application_style_hosts_require_a_bundled_template() {
assert!(host(crate::pages::Kind::Builtin, 404).validate().is_ok());
assert!(host(crate::pages::Kind::Builtin, 500).validate().is_err());
for status in crate::pages::BUILTIN_ERROR_CODES {
assert!(host(crate::pages::Kind::Builtin, status).validate().is_ok(), "{status}");
}
assert!(host(crate::pages::Kind::Builtin, 418).validate().is_err());
}
}
src/proxy/body.rs
+17 -7
View File
@@ -18,7 +18,7 @@ use std::{
};
use tokio::{
sync::OwnedSemaphorePermit,
time::{sleep, Sleep},
time::{sleep_until, Sleep},
};
pub struct Exchange {
@@ -263,7 +263,8 @@ pub struct MeteredBody {
inner: Body,
exchange: Option<Arc<Exchange>>,
direction: Direction,
timer: Pin<Box<Sleep>>,
timer: Option<Pin<Box<Sleep>>>,
deadline: tokio::time::Instant,
idle: Duration,
finished: bool,
}
@@ -274,13 +275,17 @@ impl MeteredBody {
inner,
exchange: Some(exchange),
direction,
timer: Box::pin(sleep(idle)),
// Ready/empty bodies never need a timer allocation. Keep the
// original deadline so a delayed first poll still times out.
timer: None,
deadline: tokio::time::Instant::now() + idle,
idle,
finished,
}
}
fn finish(&mut self) {
self.finished = true;
self.timer.take();
self.exchange.take();
}
}
@@ -318,11 +323,13 @@ impl HttpBody for MeteredBody {
}
}
}
this.timer
.as_mut()
.reset(tokio::time::Instant::now() + this.idle);
if this.inner.is_end_stream() {
this.finish();
} else {
this.deadline = tokio::time::Instant::now() + this.idle;
if let Some(timer) = this.timer.as_mut() {
timer.as_mut().reset(this.deadline);
}
}
Poll::Ready(Some(Ok(frame)))
}
@@ -342,7 +349,10 @@ impl HttpBody for MeteredBody {
Poll::Ready(None)
}
Poll::Pending => {
if this.timer.as_mut().poll(cx).is_ready() {
let timer = this
.timer
.get_or_insert_with(|| Box::pin(sleep_until(this.deadline)));
if timer.as_mut().poll(cx).is_ready() {
if let Some(exchange) = &this.exchange {
exchange.failed.store(true, Ordering::Relaxed);
exchange.backend_error(match this.direction {
src/proxy/body/tests.rs
+130
View File
@@ -95,4 +95,134 @@ mod tests {
assert_eq!(slots.available_permits(), 1);
assert_eq!(metrics.errors.load(Ordering::Relaxed), 1);
}
#[tokio::test]
async fn ready_and_empty_bodies_keep_accounting_without_timer_allocations() {
for payload in ["", "ready response"] {
let (exchange, slots) = observation().await;
let metrics = exchange.metrics.clone();
let logs = exchange.logs.clone();
let mut body = MeteredBody::new(
Body::from(payload),
exchange,
Direction::Response,
Duration::from_secs(1),
);
assert!(body.timer.is_none());
assert_eq!(slots.available_permits(), 0);
let mut received = Vec::new();
while let Some(frame) = body.frame().await {
received.extend_from_slice(frame.unwrap().data_ref().unwrap());
assert!(body.timer.is_none());
}
assert_eq!(received, payload.as_bytes());
assert_eq!(slots.available_permits(), 1);
assert_eq!(metrics.completed.load(Ordering::Relaxed), 1);
assert_eq!(
metrics.bytes_out.load(Ordering::Relaxed),
payload.len() as u64
);
assert_eq!(metrics.errors.load(Ordering::Relaxed), 0);
assert_eq!(logs.recent().len(), 1);
}
}
#[tokio::test]
async fn delayed_first_poll_does_not_extend_the_original_idle_deadline() {
let (exchange, slots) = observation().await;
let metrics = exchange.metrics.clone();
let pending = futures_util::stream::pending::<Result<Bytes, std::io::Error>>();
let mut body = MeteredBody::new(
Body::from_stream(pending),
exchange,
Direction::Request,
Duration::from_secs(1),
);
// Move the saved initial deadline into the past without wall-clock
// sleeps: the first Pending must create an already-expired timer.
body.deadline = tokio::time::Instant::now() - Duration::from_secs(1);
assert!(body.timer.is_none());
let result = tokio::time::timeout(Duration::from_millis(200), body.frame()).await;
assert!(result.unwrap().unwrap().is_err());
assert!(body.timer.is_none());
assert_eq!(slots.available_permits(), 1);
assert_eq!(metrics.errors.load(Ordering::Relaxed), 1);
}
#[tokio::test]
async fn progress_resets_a_lazy_timer_and_trailers_pass_unchanged() {
let (exchange, slots) = observation().await;
let metrics = exchange.metrics.clone();
let (sender, receiver) =
tokio::sync::mpsc::channel::<Result<Frame<Bytes>, std::io::Error>>(4);
let stream = futures_util::stream::unfold(receiver, |mut receiver| async {
receiver.recv().await.map(|frame| (frame, receiver))
});
let mut body = MeteredBody::new(
Body::new(http_body_util::StreamBody::new(stream)),
exchange,
Direction::Response,
Duration::from_secs(5),
);
assert!(futures_util::poll!(body.frame()).is_pending());
assert!(body.timer.is_some());
let original_deadline = body.deadline;
sender
.send(Ok(Frame::data(Bytes::from_static(b"part"))))
.await
.unwrap();
assert_eq!(
body.frame().await.unwrap().unwrap().data_ref().unwrap(),
"part"
);
assert!(body.deadline >= original_deadline);
assert_eq!(body.timer.as_ref().unwrap().deadline(), body.deadline);
let mut trailers = http::HeaderMap::new();
trailers.insert("x-checksum", "complete".parse().unwrap());
sender
.send(Ok(Frame::trailers(trailers.clone())))
.await
.unwrap();
drop(sender);
assert_eq!(
body.frame().await.unwrap().unwrap().trailers_ref(),
Some(&trailers)
);
assert!(body.frame().await.is_none());
assert!(body.timer.is_none());
assert_eq!(metrics.bytes_out.load(Ordering::Relaxed), 4);
assert_eq!(metrics.errors.load(Ordering::Relaxed), 0);
assert_eq!(slots.available_permits(), 1);
}
#[tokio::test]
async fn cancellation_and_read_errors_keep_their_original_outcomes() {
for cancel in [true, false] {
let (exchange, slots) = observation().await;
let metrics = exchange.metrics.clone();
let logs = exchange.logs.clone();
let inner = if cancel {
Body::from_stream(futures_util::stream::pending::<Result<Bytes, std::io::Error>>())
} else {
Body::from_stream(futures_util::stream::once(async {
Err::<Bytes, _>(std::io::Error::other("test read error"))
}))
};
let mut body =
MeteredBody::new(inner, exchange, Direction::Response, Duration::from_secs(1));
if cancel {
assert!(futures_util::poll!(body.frame()).is_pending());
drop(body);
} else {
assert!(body.frame().await.unwrap().is_err());
assert!(body.timer.is_none());
}
assert_eq!(slots.available_permits(), 1);
assert_eq!(metrics.errors.load(Ordering::Relaxed), 1);
assert_eq!(
logs.recent()[0].outcome,
if cancel { "cancelled" } else { "body_error" }
);
}
}
}
src/proxy/cache_helpers.rs
+1
View File
@@ -301,6 +301,7 @@ fn apply_backend_diagnostic_values(
backend_header: bool,
backend_timing_header: bool,
) {
response.headers_mut().remove("x-rustproxy-backend");
response.headers_mut().remove("x-rust-proxy-backend");
response.headers_mut().remove("x-rust-proxy-backend-time");
response.headers_mut().remove("x-rustproxy-backend-time");
src/proxy/cache_plan.rs
+93
View File
@@ -0,0 +1,93 @@
//! Request-local cache decisions derived from the already validated route.
//! A disabled plan retains its diagnostic reason without preparing cache-only
//! policy, cookie matching or request metadata. Access and origin policy stay
//! in the common forwarding path and must never depend on this plan.
use crate::cache::{RequestPolicy, Settings};
use http::{HeaderMap, Method};
#[derive(Debug, Clone, Copy)]
pub(super) enum CachePlan {
Disabled(&'static str),
Enabled { force_cache: bool, ignore_cookie: bool },
}
impl CachePlan {
pub(super) fn new(
disabled_reason: Option<&'static str>,
settings: &Settings,
path: &str,
force_match: impl FnOnce() -> bool,
) -> Self {
if let Some(reason) = disabled_reason {
return Self::Disabled(reason);
}
let force_cache = force_match();
Self::Enabled {
force_cache,
ignore_cookie: !force_cache && settings.ignore_cookie_for_path(path),
}
}
pub(super) fn force_cache(self) -> bool {
matches!(self, Self::Enabled { force_cache: true, .. })
}
pub(super) fn ignore_cookie(self) -> bool {
matches!(self, Self::Enabled { ignore_cookie: true, .. })
}
pub(super) fn request_policy(
self,
method: &Method,
headers: &HeaderMap,
body_empty: bool,
path: &str,
has_query: bool,
settings: &Settings,
) -> RequestPolicy {
match self {
Self::Disabled(reason) => RequestPolicy::Bypass(reason),
Self::Enabled { force_cache: true, .. } =>
crate::cache::forced_request_policy(method, headers, body_empty),
Self::Enabled { .. } =>
crate::cache::request_policy(method, headers, body_empty, path, has_query, settings),
}
}
}
/// Header rules may only make a request less cacheable. An original bypass is
/// final, so there is no reason to parse the rewritten headers in that case.
pub(super) fn combine_policy(
original: RequestPolicy,
rewritten: impl FnOnce() -> RequestPolicy,
) -> RequestPolicy {
if matches!(original, RequestPolicy::Bypass(_)) {
return original;
}
match (original, rewritten()) {
(_, RequestPolicy::Bypass(reason)) => RequestPolicy::Bypass(reason),
(RequestPolicy::Revalidate(reason), _) => RequestPolicy::Revalidate(reason),
(_, RequestPolicy::Revalidate(reason)) => RequestPolicy::Revalidate(reason),
_ => RequestPolicy::Use,
}
}
/// Origin Range semantics also disable transformation/compression when the
/// cache is off. Only a cache-eligible request needs byte-range parsing.
pub(super) fn range_selection(
cache_eligible: bool,
method: &Method,
headers: &HeaderMap,
) -> crate::cache::RangeSelection {
if cache_eligible {
crate::cache::request_range(method, headers)
} else if headers.contains_key("range") {
crate::cache::RangeSelection::Unsupported
} else {
crate::cache::RangeSelection::None
}
}
#[cfg(test)]
mod tests;
src/proxy/cache_plan/tests.rs
+113
View File
@@ -0,0 +1,113 @@
use super::*;
use crate::cache::RangeSelection;
use http::HeaderValue;
fn settings() -> Settings {
Settings {
enabled: true,
static_ignore_cookies: true,
static_extensions: vec!["css".into()],
..Default::default()
}
}
#[test]
fn disabled_cache_retains_reason_without_matching_force_rules_or_reparsing_headers() {
let settings = settings();
let mut headers = HeaderMap::new();
headers.insert("authorization", HeaderValue::from_static("private"));
headers.insert("cache-control", HeaderValue::from_static("max-age=not-a-number"));
for reason in [
"cache_disabled", "route_disabled", "admin_control_plane", "basic_auth",
"tinyauth", "sticky_cookie", "request_upgrade",
] {
let plan = CachePlan::new(Some(reason), &settings, "/style.css", || {
panic!("disabled cache must not match force-cache rules")
});
assert!(!plan.force_cache());
assert!(!plan.ignore_cookie());
let policy = plan.request_policy(&Method::GET, &headers, true, "/style.css", false, &settings);
assert_eq!(policy, RequestPolicy::Bypass(reason));
assert_eq!(combine_policy(policy, || panic!("disabled cache must not parse post-rule policy")), policy);
}
}
#[test]
fn cache_on_keeps_all_existing_request_policy_outcomes() {
let settings = settings();
for forced in [false, true] {
let plan = CachePlan::new(None, &settings, "/style.css", || forced);
for (name, value) in [
("x-irrelevant", "yes"), ("cache-control", "no-store"),
("cache-control", "no-cache"), ("cache-control", "max-age=0"),
("cache-control", "min-fresh=broken"), ("authorization", "private"),
("cookie", "session=private"), ("range", "bytes=0-2"),
("range", "bytes=0-2,4-5"), ("if-range", "\"etag\""),
("if-match", "\"etag\""), ("upgrade", "websocket"),
] {
let mut headers = HeaderMap::new();
headers.insert(http::HeaderName::from_bytes(name.as_bytes()).unwrap(), value.parse().unwrap());
for method in [Method::GET, Method::HEAD, Method::POST] {
for body_empty in [false, true] {
let expected = if forced {
crate::cache::forced_request_policy(&method, &headers, body_empty)
} else {
crate::cache::request_policy(&method, &headers, body_empty, "/style.css", false, &settings)
};
assert_eq!(plan.request_policy(&method, &headers, body_empty, "/style.css", false, &settings), expected);
}
}
}
}
}
#[test]
fn request_header_rules_cannot_relax_bypass_or_revalidation() {
let cases = [
RequestPolicy::Use,
RequestPolicy::Revalidate("request_no_cache"),
RequestPolicy::Revalidate("request_max_age"),
RequestPolicy::Bypass("request_authorization"),
RequestPolicy::Bypass("request_no_store"),
];
for original in cases {
for rewritten in cases {
let expected = match (original, rewritten) {
(RequestPolicy::Bypass(reason), _) | (_, RequestPolicy::Bypass(reason)) => RequestPolicy::Bypass(reason),
(RequestPolicy::Revalidate(reason), _) | (_, RequestPolicy::Revalidate(reason)) => RequestPolicy::Revalidate(reason),
_ => RequestPolicy::Use,
};
assert_eq!(combine_policy(original, || rewritten), expected);
}
}
}
#[test]
fn cache_off_still_blocks_range_transformation_and_preserves_origin_headers() {
for method in [Method::GET, Method::HEAD, Method::POST] {
for range in [None, Some("bytes=0-2"), Some("bytes=0-2,4-5"), Some("broken"), Some("bytes=99-")] {
let mut headers = HeaderMap::new();
if let Some(range) = range { headers.insert("range", range.parse().unwrap()); }
headers.insert("if-range", HeaderValue::from_static("\"same-validator\""));
headers.insert("if-none-match", HeaderValue::from_static("\"client-validator\""));
let original = headers.clone();
let baseline = crate::cache::request_range(&method, &headers);
assert_eq!(range_selection(true, &method, &headers), baseline);
assert_eq!(matches!(range_selection(false, &method, &headers), RangeSelection::None), matches!(baseline, RangeSelection::None));
assert_eq!(headers, original, "origin must receive the original Range/conditional fields");
}
}
}
#[test]
fn force_cache_keeps_cookie_variants_and_disabled_cookie_matching_has_no_effect() {
let settings = settings();
let ordinary = CachePlan::new(None, &settings, "/style.css", || false);
assert!(ordinary.ignore_cookie());
let forced = CachePlan::new(None, &settings, "/style.css", || true);
assert!(forced.force_cache());
assert!(!forced.ignore_cookie());
let disabled = CachePlan::new(Some("route_disabled"), &settings, "/style.css", || true);
assert!(!disabled.force_cache());
assert!(!disabled.ignore_cookie());
}
src/proxy/forward/route_cache.rs
+27 -53
View File
@@ -167,7 +167,7 @@ let body_empty = if cache_enabled
}
}
} else {
request.body().is_end_stream()
cache_enabled && request.body().is_end_stream()
};
let public_suffix = request
.uri()
@@ -175,29 +175,18 @@ let public_suffix = request
.map(|v| v.as_str())
.unwrap_or("/")
.to_owned();
let force_cache = policy_reason.is_none()
&& cache_settings.active_for(route.cache)
&& route
let cache_plan = cache_plan::CachePlan::new(policy_reason, cache_settings, &path, || {
cache_settings.active_for(route.cache) && route
.frontend
.as_ref()
.is_some_and(|frontend| frontend.force_cache.matches(&path, &public_suffix));
let ignore_static_cookie = !force_cache && cache_settings.ignore_cookie_for_path(&path);
let original_policy = policy_reason
.map(crate::cache::RequestPolicy::Bypass)
.unwrap_or_else(|| {
if force_cache {
crate::cache::forced_request_policy(request.method(), request.headers(), body_empty)
} else {
crate::cache::request_policy(
request.method(),
request.headers(),
body_empty,
&path,
request.uri().query().is_some(),
cache_settings,
)
}
});
.is_some_and(|frontend| frontend.force_cache.matches(&path, &public_suffix))
});
let force_cache = cache_plan.force_cache();
let ignore_static_cookie = cache_plan.ignore_cookie();
let original_policy = cache_plan.request_policy(
request.method(), request.headers(), body_empty, &path,
request.uri().query().is_some(), cache_settings,
);
let only_cached =
cache_settings.active_for(route.cache) && crate::cache::only_if_cached(request.headers());
@@ -282,48 +271,31 @@ if websocket_legacy_support {
}
}
}
let range_selection = crate::cache::request_range(request.method(), request.headers());
let if_range = request.headers().get("if-range").cloned();
let range_requested = !matches!(range_selection, crate::cache::RangeSelection::None);
let post_policy = if force_cache {
crate::cache::forced_request_policy(request.method(), request.headers(), body_empty)
} else {
crate::cache::request_policy(
request.method(),
request.headers(),
body_empty,
&path,
request.uri().query().is_some(),
cache_settings,
let effective_policy = cache_plan::combine_policy(original_policy, || {
cache_plan.request_policy(
request.method(), request.headers(), body_empty, &path,
request.uri().query().is_some(), cache_settings,
)
};
});
let force_vary_cookie = force_cache && request.headers().contains_key("cookie");
let force_vary_authorization = force_cache && request.headers().contains_key("authorization");
let effective_policy = match (original_policy, post_policy) {
(crate::cache::RequestPolicy::Bypass(reason), _) => {
crate::cache::RequestPolicy::Bypass(reason)
}
(_, crate::cache::RequestPolicy::Bypass(reason)) => {
crate::cache::RequestPolicy::Bypass(reason)
}
(crate::cache::RequestPolicy::Revalidate(reason), _) => {
crate::cache::RequestPolicy::Revalidate(reason)
}
(_, crate::cache::RequestPolicy::Revalidate(reason)) => {
crate::cache::RequestPolicy::Revalidate(reason)
}
_ => crate::cache::RequestPolicy::Use,
};
let effective_reason = effective_policy.bypass_reason();
let revalidation_reason = effective_policy.revalidation_reason();
let effective_cacheable = effective_reason.is_none();
let range_selection = cache_plan::range_selection(
effective_cacheable, request.method(), request.headers(),
);
let if_range = if effective_cacheable {
request.headers().get("if-range").cloned()
} else { None };
let range_requested = !matches!(range_selection, crate::cache::RangeSelection::None);
// Match Varnish-style cache identity: a load-balanced backend member must not
// fragment the object cache. The vhost scope + canonical host + request target
// already identify the public resource; Vary handles representation variants.
let cache_route = route.id.as_str();
let mut cache_key_headers = None;
if ignore_static_cookie && request.headers().contains_key("cookie") {
if effective_cacheable && ignore_static_cookie && request.headers().contains_key("cookie") {
let mut headers = request.headers().clone();
headers.remove("cookie");
cache_key_headers = Some(headers);
@@ -344,7 +316,9 @@ let cache_ticket = if effective_cacheable {
} else {
None
};
let client_conditionals = request.headers().clone();
let client_conditionals = if cache_ticket.is_some() {
request.headers().clone()
} else { HeaderMap::new() };
let cached_revalidation = if let Some(ticket) = cache_ticket.as_ref() {
state.cache.revalidation_tiered(ticket).await
} else {
src/proxy/forward/success_finish.rs
+2 -1
View File
@@ -94,7 +94,8 @@ if response.status() == StatusCode::NOT_MODIFIED && revalidation_sent {
// Both origin and transformed policies must be cacheable; hiding a
// Set-Cookie/private header never authorizes storage of private content.
let transformed_plan =
if ignore_static_cookie && crate::cache::varies_on(response.headers(), "cookie") {
if origin_plan.is_none()
|| (ignore_static_cookie && crate::cache::varies_on(response.headers(), "cookie")) {
None
} else {
crate::cache::plan(
src/proxy/forward/success_prepare.rs
+6 -1
View File
@@ -1,5 +1,10 @@
{
let origin_validation_headers = response.headers().clone();
// Only a real cache revalidation needs origin headers after response rules.
let origin_validation_headers = if revalidation_sent
&& response.status() == StatusCode::NOT_MODIFIED
{
response.headers().clone()
} else { HeaderMap::new() };
if let Some(exchange) = &exchange {
exchange
.upstream_status
src/proxy/forward/upstream.rs
+3 -1
View File
@@ -119,7 +119,9 @@ if websocket_upstream_h2 {
// Do not classify a slow client upload as a slow origin response. Probes
// independently measure origin liveness/latency with body-free requests.
let request_body_empty = request.body().size_hint().exact() == Some(0);
let proactive_base_headers = request.headers().clone();
let proactive_base_headers = if cache_ticket.is_some() && cache_settings.proactive_refresh_enabled {
request.headers().clone()
} else { HeaderMap::new() };
let response_header_timeout = frontend
.and_then(|frontend| frontend.proxy_timeouts.response_header_timeout())
.unwrap_or(state.header_timeout);
src/proxy/forward/web_source.rs
+48 -38
View File
@@ -28,7 +28,9 @@
// makes PHP-FPM omit HTTP_HOST, breaking WordPress.
request.headers_mut().insert(HOST, original_host.clone());
let request_method = request.method().clone();
let request_headers = request.headers().clone();
// This branch owns local-origin delivery. Move its headers instead of
// cloning them; the request remains only for exchange/body extensions.
let mut request_headers = std::mem::take(request.headers_mut());
let request_uri = request.uri().clone();
let request_version = request.version();
let public_suffix = request_uri.path_and_query()
@@ -41,32 +43,33 @@
Err(_) => return error(StatusCode::BAD_REQUEST,
"RustProxy: invalid or incomplete request body\n"),
}
} else { request.body().is_end_stream() };
let force_cache = enabled && route.frontend.as_ref().is_some_and(|frontend|
frontend.force_cache.matches(&path, &public_suffix));
let policy = if enabled {
if force_cache {
crate::cache::forced_request_policy(&request_method, &request_headers, empty_body)
} else {
crate::cache::request_policy(
&request_method, &request_headers, empty_body, &path,
request_uri.query().is_some(), cache_settings)
}
} else {
crate::cache::RequestPolicy::Bypass("cache_disabled")
};
} else { enabled && request.body().is_end_stream() };
let cache_plan = cache_plan::CachePlan::new(
(!enabled).then_some("cache_disabled"), cache_settings, &path,
|| route.frontend.as_ref().is_some_and(|frontend|
frontend.force_cache.matches(&path, &public_suffix)),
);
let force_cache = cache_plan.force_cache();
let ignore_cookie = cache_plan.ignore_cookie();
let policy = cache_plan.request_policy(
&request_method, &request_headers, empty_body, &path,
request_uri.query().is_some(), cache_settings,
);
// HEAD can use an existing GET representation but may not fill a cache
// entry with its empty downstream body.
let eligible = enabled && policy.bypass_reason().is_none();
let mut key_headers = request_headers.clone();
let ignore_cookie = !force_cache && cache_settings.ignore_cookie_for_path(&path);
if ignore_cookie { key_headers.remove("cookie"); }
let key_headers = if eligible && ignore_cookie && request_headers.contains_key("cookie") {
let mut headers = request_headers.clone();
headers.remove("cookie");
Some(headers)
} else { None };
let cache_ticket = if eligible {
state.cache.ticket_scoped(state.cache.revision(), scope, context.scheme,
original_host.to_str().unwrap_or(""), &route.id, &public_suffix, &key_headers)
original_host.to_str().unwrap_or(""), &route.id, &public_suffix,
key_headers.as_ref().unwrap_or(&request_headers))
} else { None };
let range = crate::cache::request_range(&request_method, &request_headers);
let if_range = request_headers.get("if-range").cloned();
let range = cache_plan::range_selection(eligible, &request_method, &request_headers);
let if_range = if eligible { request_headers.get("if-range").cloned() } else { None };
let range_requested = !matches!(range, crate::cache::RangeSelection::None);
// Unlike the original Web Server path, retain stale and client-forced
// revalidation candidates. A local 304 can refresh them without replacing
@@ -122,9 +125,14 @@
// Neither the Request nor the streaming Body is held across a borrowed
// await: the filesystem/PHP source owns the body for its full lifetime.
let static_body = std::mem::replace(request.body_mut(), Body::empty());
let (origin_request_headers, revalidation_sent) =
let (origin_request_headers, revalidation_sent) = if cache_ticket.is_some() {
web_cache_revalidation::web_origin_request_headers(
&request_headers, cached_revalidation.as_ref());
&request_headers, cached_revalidation.as_ref())
} else {
// No cache result can need the original conditionals later. Preserve
// all origin HTTP semantics by handing over the untouched header map.
(std::mem::take(&mut request_headers), false)
};
let started = Instant::now();
let response = crate::web_server::serve(service, request_method.clone(),
origin_request_headers, request_uri, &service_path, static_body,
@@ -181,25 +189,27 @@
state.cache.discard(ticket);
}
}
let origin_headers = response.headers().clone();
let original_plan = crate::cache::plan(response.status(), &origin_headers,
cache_settings, started.elapsed());
// Do not allow a later response-header rewrite to override a private,
// explicitly uncacheable PHP/CGI origin response.
let explicit_origin_policy = origin_headers.contains_key("cache-control")
|| origin_headers.contains_key("expires")
|| origin_headers.contains_key("set-cookie")
|| origin_headers.contains_key("vary");
let origin_safe = !explicit_origin_policy || original_plan.is_some();
let no_transform = crate::transform::no_transform(&origin_headers);
let skip_compression = crate::compression::no_transform(&origin_headers);
let origin_headers = response.headers();
let origin_safe = cache_ticket.is_some() && {
let explicit_origin_policy = origin_headers.contains_key("cache-control")
|| origin_headers.contains_key("expires")
|| origin_headers.contains_key("set-cookie")
|| origin_headers.contains_key("vary");
!explicit_origin_policy || crate::cache::plan(response.status(), origin_headers,
cache_settings, started.elapsed()).is_some()
};
let no_transform = crate::transform::no_transform(origin_headers);
let skip_compression = crate::compression::no_transform(origin_headers);
let origin_validators = crate::transform::OriginValidators {
etag: origin_headers.get("etag").cloned(),
last_modified: origin_headers.get("last-modified").cloned(),
};
if skip_compression || range_requested {
response.extensions_mut().insert(crate::compression::Skip);
}
response.extensions_mut().insert(crate::transform::OriginValidators {
etag: origin_headers.get("etag").cloned(),
last_modified: origin_headers.get("last-modified").cloned(),
});
response.extensions_mut().insert(origin_validators);
apply_response_rules(response.headers_mut(), &snapshot, route.frontend.as_deref(),
Some(route), false);
if response.extensions().get::<crate::pages::PageServed>().is_some() {
@@ -271,4 +281,4 @@
let transformed = response.extensions().get::<crate::transform::Applied>().is_some();
cache_settings.annotate_transformed(response.headers_mut(), transformed);
response
}
}
src/proxy/mod.rs
+3
View File
@@ -1,6 +1,7 @@
include!("types.rs");
include!("error_fallback.rs");
include!("cache_helpers.rs");
mod cache_plan;
mod web_cache_revalidation;
include!("cache_http_invalidation.rs");
include!("upstream_helpers.rs");
@@ -14,3 +15,5 @@ include!("tests/h3_navigation_tests.rs");
#[cfg(test)]
include!("tests/http_invalidation_tests.rs");
#[cfg(test)]
include!("tests/cache_fast_path_tests.rs");
src/proxy/request_id.rs
+26
View File
@@ -0,0 +1,26 @@
fn request_id_hash(sequence: u64) -> String {
use rand::TryRng;
use sha2::{Digest, Sha256};
let mut nonce = [0u8; 32];
let mut rng = rand::rngs::SysRng;
rng.try_fill_bytes(&mut nonce)
.expect("secure operating system random source unavailable");
let mut hash = Sha256::new();
hash.update(sequence.to_be_bytes());
hash.update(nonce);
request_id_hex(&hash.finalize())
}
fn request_id_hex(digest: &[u8]) -> String {
const HEX: &[u8; 16] = b"0123456789abcdef";
let mut encoded = String::with_capacity(digest.len() * 2);
for &byte in digest {
encoded.push(HEX[(byte >> 4) as usize] as char);
encoded.push(HEX[(byte & 0x0f) as usize] as char);
}
encoded
}
#[cfg(test)]
include!("request_id/tests.rs");
src/proxy/request_id/tests.rs
+58
View File
@@ -0,0 +1,58 @@
mod request_id_tests {
use super::*;
use sha2::{Digest, Sha256};
#[test]
fn hexadecimal_encoding_matches_previous_sha256_display() {
for byte in 0u8..=255 {
let digest = Sha256::digest([byte; 32]);
assert_eq!(request_id_hex(&digest), format!("{digest:x}"));
}
assert_eq!(
request_id_hex(&[0, 1, 15, 16, 127, 128, 254, 255]),
"00010f107f80feff"
);
}
#[test]
fn request_ids_keep_their_64_hexadecimal_format_and_randomness() {
let mut seen = std::collections::HashSet::new();
for _ in 0..512 {
let id = request_id_hash(1);
assert_eq!(id.len(), 64);
assert!(id
.bytes()
.all(|b| b.is_ascii_digit() || (b'a'..=b'f').contains(&b)));
assert!(
seen.insert(id),
"fixed sequence must still use independent OS entropy"
);
}
}
#[test]
#[ignore = "explicit release-mode microbenchmark; not an acceptance threshold"]
fn benchmark_request_id_formatting() {
use std::hint::black_box;
const ITERATIONS: u32 = 200_000;
let digest = Sha256::digest(b"same SHA256 digest for both formatters");
for round in 0..7 {
for candidate in if round % 2 == 0 {
[false, true]
} else {
[true, false]
} {
let start = Instant::now();
for _ in 0..ITERATIONS {
let value = if candidate {
request_id_hex(black_box(&digest))
} else {
format!("{:x}", black_box(&digest))
};
black_box(value);
}
println!("{{\"round\":{round},\"candidate\":{candidate},\"iterations\":{ITERATIONS},\"ns_per_op\":{}}}", start.elapsed().as_nanos() as f64 / ITERATIONS as f64);
}
}
}
}
src/proxy/tests/authority_tests.rs
+5 -5
View File
@@ -218,6 +218,7 @@ mod authority_tests {
#[test]
fn generated_policy_and_gateway_errors_use_html_pages() {
for status in [
StatusCode::BAD_GATEWAY,
StatusCode::FORBIDDEN,
StatusCode::TOO_MANY_REQUESTS,
StatusCode::SERVICE_UNAVAILABLE,
@@ -231,11 +232,6 @@ mod authority_tests {
);
assert_eq!(response.headers().get("cache-control").unwrap(), "no-store");
}
let response = error(StatusCode::BAD_GATEWAY, "internal detail\n");
assert_ne!(
response.headers().get("content-type").unwrap(),
"text/html; charset=utf-8"
);
}
#[test]
fn locally_generated_pages_remove_cache_metadata() {
@@ -344,6 +340,7 @@ mod authority_tests {
#[test]
fn backend_diagnostics_are_opt_in_and_strip_spoofed_values() {
let mut response = Response::new(Body::empty());
response.headers_mut().insert("x-rustproxy-backend", HeaderValue::from_static("canonical-spoof"));
response
.headers_mut()
.insert("x-rust-proxy-backend", HeaderValue::from_static("spoofed"));
@@ -356,6 +353,7 @@ mod authority_tests {
HeaderValue::from_static("999ms"),
);
apply_backend_diagnostic_values(&mut response, false, false);
assert!(response.headers().get("x-rustproxy-backend").is_none());
assert!(response.headers().get("x-rust-proxy-backend").is_none());
assert!(response
.headers()
@@ -369,7 +367,9 @@ mod authority_tests {
response
.extensions_mut()
.insert(OriginTiming(Duration::from_millis(12)));
response.headers_mut().insert("x-rustproxy-backend", HeaderValue::from_static("canonical-spoof"));
apply_backend_diagnostic_values(&mut response, true, true);
assert!(response.headers().get("x-rustproxy-backend").is_none());
assert_eq!(
response.headers().get("x-rust-proxy-backend").unwrap(),
"origin-a"
src/proxy/tests/cache_fast_path_tests.rs
+319
View File
@@ -0,0 +1,319 @@
mod cache_fast_path_tests {
use super::*;
use crate::{auth::Auth, config::{self, Config, Update, ValidationPolicy}, db::Store};
use std::net::SocketAddr;
use tempfile::TempDir;
use tower::ServiceExt;
const CSS: &[u8] = b"/* origin comment */\nbody { color: red; margin: 0px; }\n";
struct Fixture { app: Arc<App>, _directory: TempDir }
struct Received { status: StatusCode, headers: HeaderMap, body: Bytes }
impl Fixture {
async fn new(config: Config) -> Self {
let directory = TempDir::new().unwrap();
let policy = ValidationPolicy {
tls_addr: None, cert_dir: None, data_dir: None,
proxy_addr: "127.0.0.1:8080".parse().unwrap(),
admin_addr: "127.0.0.1:9090".parse().unwrap(),
};
let store = Store::open(&directory.path().join("db"), &config).await.unwrap();
let auth = Auth::new(store.clone(), directory.path().to_owned()).await.unwrap();
let snapshot = config::compile(config, 0, &policy).unwrap();
let app = App::new(store, snapshot, policy, auth, 32,
Duration::from_secs(5), Duration::from_secs(5), Duration::from_secs(30));
Self { app, _directory: directory }
}
async fn send(&self, method: Method, path: &str, headers: &[(&str, &str)], body: Body) -> Received {
send(self.app.clone(), method, path, headers, body).await
}
async fn get(&self, path: &str, headers: &[(&str, &str)]) -> Received {
self.send(Method::GET, path, headers, Body::empty()).await
}
async fn change(&self, edit: impl FnOnce(&mut Config)) {
let current = self.app.config.load_full();
let mut config = current.envelope.config.clone();
edit(&mut config);
self.app.update(Update { expected_revision: current.envelope.revision, config }).await.unwrap();
}
async fn stop(self) {
self.app.shutdown.cancel();
self.app.tunnels.close();
self.app.tunnels.wait().await;
self.app.store.close().await;
}
}
async fn send(app: Arc<App>, method: Method, path: &str, headers: &[(&str, &str)], body: Body) -> Received {
let mut builder = Request::builder().method(method).uri(path)
.header(HOST, "localhost").header(USER_AGENT, "cache-plan-regression/1.0");
for (name, value) in headers { builder = builder.header(*name, *value); }
let mut request = builder.body(body).unwrap();
request.extensions_mut().insert(PeerAddr("192.0.2.50:45678".parse().unwrap()));
request.extensions_mut().insert(LocalAddr("127.0.0.1:8080".parse().unwrap()));
let response = router(app).oneshot(request).await.unwrap();
let (parts, body) = response.into_parts();
let body = body.collect().await.unwrap().to_bytes();
Received { status: parts.status, headers: parts.headers, body }
}
fn config_for(origin: SocketAddr, enabled: bool) -> Config {
let mut config = config::demo_config(origin);
config.cache.enabled = enabled;
config.cache.expose_status = true;
config.cache.expose_hits = true;
config.cache.expose_cache_status = true;
config.cache.expose_only_hits = false;
config.cache.static_ignore_cookies = false;
config
}
fn web_config(directory: &TempDir, enabled: bool) -> Config {
std::fs::write(directory.path().join("style.css"), CSS).unwrap();
let mut config = config_for("127.0.0.1:9000".parse().unwrap(), enabled);
config.web_services.push(crate::web_server::Service {
id: "site".into(), name: "Site".into(), document_root: directory.path().to_owned(),
cache_control: Some("public, max-age=600".into()), ..Default::default()
});
config.vhosts[0].source = crate::vhost::Source::WebServer { web_service_id: "site".into() };
config.vhosts[0].backends.clear();
config.vhosts[0].routes.clear();
config
}
#[tokio::test]
async fn web_cache_disabled_keeps_get_head_range_conditionals_and_observability() {
let directory = TempDir::new().unwrap();
let fixture = Fixture::new(web_config(&directory, false)).await;
let first = fixture.get("/style.css", &[]).await;
assert_eq!(first.status, StatusCode::OK);
assert_eq!(first.body, CSS);
assert!(!first.headers.contains_key("x-cache"));
let etag = first.headers["etag"].to_str().unwrap();
let head = fixture.send(Method::HEAD, "/style.css", &[], Body::empty()).await;
assert_eq!(head.status, StatusCode::OK);
assert_eq!(head.headers["content-length"], CSS.len().to_string());
assert!(head.body.is_empty());
let partial = fixture.get("/style.css", &[("range", "bytes=0-3"), ("if-range", etag)]).await;
assert_eq!(partial.status, StatusCode::PARTIAL_CONTENT);
assert_eq!(partial.body, &CSS[..4]);
let mismatch = fixture.get("/style.css", &[("range", "bytes=0-3"), ("if-range", "\"different\"")]).await;
assert_eq!(mismatch.status, StatusCode::OK);
assert_eq!(mismatch.body, CSS);
let conditional = fixture.get("/style.css", &[("if-none-match", etag)]).await;
assert_eq!(conditional.status, StatusCode::NOT_MODIFIED);
assert!(conditional.body.is_empty());
let invalid = fixture.get("/style.css", &[("range", "bytes=99999-")]).await;
assert_eq!(invalid.status, StatusCode::RANGE_NOT_SATISFIABLE);
assert_eq!(invalid.headers["content-range"], format!("bytes */{}", CSS.len()));
let only_cached = fixture.get("/style.css", &[("cache-control", "only-if-cached")]).await;
assert_eq!(only_cached.body, CSS);
let logs = fixture.app.logs.recent();
assert_eq!(logs.len(), 7);
assert!(logs.iter().all(|entry| entry.cache_reason == "cache_disabled"));
assert!(logs.iter().all(|entry| entry.request_id.len() == 64));
assert_eq!(fixture.app.metrics.completed.load(Ordering::Relaxed), 7);
assert_eq!(fixture.app.metrics.active.load(Ordering::Relaxed), 0);
assert_eq!(fixture.app.logs.stats().queue_dropped, 0);
fixture.stop().await;
}
#[tokio::test]
async fn web_cache_hot_toggles_keep_miss_hit_and_bypass_diagnostics() {
let directory = TempDir::new().unwrap();
let fixture = Fixture::new(web_config(&directory, false)).await;
assert!(!fixture.get("/style.css", &[]).await.headers.contains_key("x-cache"));
fixture.change(|config| config.cache.enabled = true).await;
assert_eq!(fixture.get("/style.css", &[]).await.headers["x-cache"], "MISS");
let hit = fixture.get("/style.css", &[]).await;
assert_eq!(hit.headers["x-cache"], "HIT");
assert_eq!(hit.body, CSS);
fixture.change(|config| config.vhosts[0].routes.push(crate::vhost::PathRoute {
id: "uncached".into(), path_prefix: "/style.css".into(), cache: Some(false),
source: crate::vhost::Source::WebServer { web_service_id: "site".into() },
..Default::default()
})).await;
let bypass = fixture.get("/style.css", &[]).await;
assert_eq!(bypass.headers["x-cache"], "BYPASS");
assert_eq!(bypass.headers["x-cache-reason"], "cache_disabled");
fixture.change(|config| {
config.vhosts[0].routes.clear();
config.vhosts[0].cache.enabled = Some(false);
}).await;
assert!(!fixture.get("/style.css", &[]).await.headers.contains_key("x-cache"));
fixture.change(|config| config.vhosts[0].cache.enabled = None).await;
assert_eq!(fixture.get("/style.css", &[]).await.headers["x-cache"], "MISS");
assert_eq!(fixture.get("/style.css", &[]).await.headers["x-cache"], "HIT");
fixture.stop().await;
}
#[tokio::test]
async fn web_transform_remains_independent_of_disabled_cache_and_range() {
let directory = TempDir::new().unwrap();
let mut config = web_config(&directory, false);
config.transform.enabled = true;
config.vhosts[0].transform.enabled = true;
let fixture = Fixture::new(config).await;
let normal = fixture.get("/style.css", &[]).await;
assert_eq!(normal.status, StatusCode::OK);
assert!(normal.body.len() < CSS.len(), "transform must remain active with cache off");
let range = fixture.get("/style.css", &[("range", "bytes=0-3"), ("accept-encoding", "gzip")]).await;
assert_eq!(range.status, StatusCode::PARTIAL_CONTENT);
assert_eq!(range.body, &CSS[..4]);
assert!(!range.headers.contains_key("content-encoding"));
fixture.change(|config| config.vhosts[0].transform.enabled = false).await;
assert_eq!(fixture.get("/style.css", &[]).await.body, CSS);
fixture.stop().await;
}
#[tokio::test]
async fn cache_fast_paths_keep_global_and_location_access_control() {
let directory = TempDir::new().unwrap();
for enabled in [false, true] {
let mut config = web_config(&directory, enabled);
let deny = crate::acl::Acl { default: crate::acl::Action::Deny, rules: vec![] };
config.host_policies.push(config::HostPolicy { hosts: vec!["localhost".into()], acl: deny.clone() });
let fixture = Fixture::new(config).await;
assert_eq!(fixture.get("/style.css", &[]).await.status, StatusCode::FORBIDDEN);
fixture.change(|config| {
config.host_policies.clear();
config.vhosts[0].routes.push(crate::vhost::PathRoute {
id: "private".into(), path_prefix: "/style.css".into(), acl: Some(deny),
source: crate::vhost::Source::WebServer { web_service_id: "site".into() },
..Default::default()
});
}).await;
assert_eq!(fixture.get("/style.css", &[]).await.status, StatusCode::FORBIDDEN);
fixture.stop().await;
}
}
struct Origin { addr: SocketAddr, requests: Arc<AtomicU64>, task: tokio::task::JoinHandle<()> }
impl Drop for Origin { fn drop(&mut self) { self.task.abort(); } }
async fn origin() -> Origin {
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let requests = Arc::new(AtomicU64::new(0));
let count = requests.clone();
let app = Router::new().fallback(move |request: Request| {
let count = count.clone();
async move {
count.fetch_add(1, Ordering::Relaxed);
let (parts, body) = request.into_parts();
if parts.uri.path() == "/slow" { tokio::time::sleep(Duration::from_millis(25)).await; }
let input = body.collect().await.unwrap().to_bytes();
let variant = parts.headers.get("x-variant").and_then(|v| v.to_str().ok()).unwrap_or("default");
let output = if parts.method == Method::POST { input } else { Bytes::from(format!("origin-{variant}")) };
let mut response = Response::new(Body::from(output.clone()));
response.headers_mut().insert("etag", HeaderValue::from_static("\"origin-v1\""));
response.headers_mut().insert("cache-control", HeaderValue::from_static(
if parts.uri.path() == "/private" { "private, max-age=600" } else { "public, max-age=600" }));
response.headers_mut().insert("vary", HeaderValue::from_static("X-Variant"));
response.headers_mut().insert("content-type", HeaderValue::from_static("text/plain"));
for name in ["range", "if-range", "if-none-match", "cookie", "authorization", "x-request-id"] {
if let Some(value) = parts.headers.get(name) {
let echoed: http::HeaderName = format!("x-seen-{name}").parse().unwrap();
response.headers_mut().insert(echoed, value.clone());
}
}
if parts.headers.get("if-none-match").is_some_and(|v| v == "\"origin-v1\"") {
*response.status_mut() = StatusCode::NOT_MODIFIED;
*response.body_mut() = Body::empty();
} else if parts.headers.get("range").is_some_and(|v| v == "bytes=0-2")
&& parts.headers.get("if-range").is_none_or(|v| v == "\"origin-v1\"") {
*response.status_mut() = StatusCode::PARTIAL_CONTENT;
response.headers_mut().insert("content-range", format!("bytes 0-2/{}", output.len()).parse().unwrap());
*response.body_mut() = Body::from(output.slice(..3));
}
response
}
});
let task = tokio::spawn(async move { axum::serve(listener, app).await.unwrap(); });
Origin { addr, requests, task }
}
#[tokio::test]
async fn proxy_disabled_cache_preserves_headers_post_and_conditionals() {
let origin = origin().await;
let fixture = Fixture::new(config_for(origin.addr, false)).await;
let range = fixture.get("/inspect", &[("range", "bytes=0-2"), ("if-range", "\"origin-v1\""),
("cookie", "session=one"), ("authorization", "Bearer example")]).await;
assert_eq!(range.status, StatusCode::PARTIAL_CONTENT);
assert_eq!(range.body, "ori");
assert_eq!(range.headers["x-seen-range"], "bytes=0-2");
assert_eq!(range.headers["x-seen-if-range"], "\"origin-v1\"");
assert_eq!(range.headers["x-seen-cookie"], "session=one");
assert_eq!(range.headers["x-seen-authorization"], "Bearer example");
assert_eq!(range.headers["x-seen-x-request-id"].as_bytes().len(), 64);
assert!(!range.headers.contains_key("x-cache"));
let conditional = fixture.get("/inspect", &[("if-none-match", "\"origin-v1\"")]).await;
assert_eq!(conditional.status, StatusCode::NOT_MODIFIED);
assert!(conditional.body.is_empty());
let input = futures_util::stream::iter([
Ok::<_, std::io::Error>(Bytes::from_static(b"streamed ")),
Ok(Bytes::from_static(b"content")),
]);
let post = fixture.send(Method::POST, "/echo", &[], Body::from_stream(input)).await;
assert_eq!(post.status, StatusCode::OK);
assert_eq!(post.body, "streamed content");
assert_eq!(origin.requests.load(Ordering::Relaxed), 3);
assert!(fixture.app.logs.recent().iter().all(|entry| entry.cache_reason == "cache_disabled"));
fixture.stop().await;
}
#[tokio::test]
async fn proxy_cache_on_preserves_vary_private_revalidation_and_ranges() {
let origin = origin().await;
let fixture = Fixture::new(config_for(origin.addr, true)).await;
for variant in ["a", "b"] {
assert_eq!(fixture.get("/vary", &[("x-variant", variant)]).await.headers["x-cache"], "MISS");
}
let hit = fixture.get("/vary", &[("x-variant", "a")]).await;
assert_eq!(hit.headers["x-cache"], "HIT");
assert_eq!(hit.body, "origin-a");
let range = fixture.get("/vary", &[("x-variant", "a"), ("range", "bytes=0-2"), ("if-range", "\"origin-v1\"")]).await;
assert_eq!(range.status, StatusCode::PARTIAL_CONTENT);
assert_eq!(range.headers["x-cache"], "HIT");
assert_eq!(range.body, "ori");
let validated = fixture.get("/vary", &[("x-variant", "a"), ("cache-control", "no-cache")]).await;
assert_eq!(validated.status, StatusCode::OK);
assert_eq!(validated.headers["x-cache"], "HIT");
assert_eq!(validated.headers["x-cache-reason"], "cache_revalidated");
assert_eq!(validated.body, "origin-a");
let conditional = fixture.get("/vary", &[("x-variant", "a"), ("if-none-match", "\"origin-v1\"")]).await;
assert_eq!(conditional.status, StatusCode::NOT_MODIFIED);
let bypass = fixture.get("/vary", &[("authorization", "Bearer example")]).await;
assert_eq!(bypass.headers["x-cache-reason"], "request_authorization");
for _ in 0..2 {
assert_eq!(fixture.get("/private", &[]).await.headers["x-cache"], "MISS");
}
assert_eq!(origin.requests.load(Ordering::Relaxed), 6);
fixture.stop().await;
}
#[tokio::test]
async fn proxy_cache_enabled_concurrent_fill_stays_single_flight() {
let origin = origin().await;
let fixture = Fixture::new(config_for(origin.addr, true)).await;
let mut requests = tokio::task::JoinSet::new();
for _ in 0..12 {
let app = fixture.app.clone();
requests.spawn(async move { send(app, Method::GET, "/slow", &[], Body::empty()).await });
}
while let Some(result) = requests.join_next().await {
let response = result.unwrap();
assert_eq!(response.status, StatusCode::OK);
assert_eq!(response.body, "origin-default");
}
assert_eq!(origin.requests.load(Ordering::Relaxed), 1);
assert_eq!(fixture.app.metrics.completed.load(Ordering::Relaxed), 12);
assert_eq!(fixture.app.metrics.active.load(Ordering::Relaxed), 0);
fixture.stop().await;
}
}
src/proxy/upstream_helpers.rs
+9 -14
View File
@@ -1,16 +1,4 @@
fn request_id_hash(sequence: u64) -> String {
use rand::TryRng;
use sha2::{Digest, Sha256};
let mut nonce = [0u8; 32];
let mut rng = rand::rngs::SysRng;
rng.try_fill_bytes(&mut nonce)
.expect("secure operating system random source unavailable");
let mut hash = Sha256::new();
hash.update(sequence.to_be_bytes());
hash.update(nonce);
format!("{:x}", hash.finalize())
}
include!("request_id.rs");
fn should_replace_error_page(response: &Response, minimal_welcome: bool) -> bool {
!minimal_welcome
@@ -43,7 +31,13 @@ async fn observe(State(state): State<Arc<App>>, mut request: Request, next: Next
.unwrap_or_default()
.tls;
request.extensions_mut().insert(snapshot.clone());
let original_headers = request.headers().clone();
let original_headers = if snapshot.envelope.config.compression.enabled()
&& request.method() != Method::HEAD
{
crate::compression::request_headers(request.headers())
} else {
HeaderMap::new()
};
let method = request.method().clone();
let is_head = method == Method::HEAD;
let ws_attempt = request
@@ -242,6 +236,7 @@ async fn observe(State(state): State<Arc<App>>, mut request: Request, next: Next
"x-rustproxy-request-id",
"x-rustproxy-revision",
"x-rust-proxy-backend",
"x-rustproxy-backend",
"x-rust-proxy-backend-time",
"x-rustproxy-backend-time",
"x-cache",
src/transform/tests.rs
+2 -2
View File
@@ -426,8 +426,8 @@ mod tests {
let input = include_str!("../admin/web/style.css");
let result = css::minify(input.as_bytes()).expect("minify panel stylesheet");
let output = std::str::from_utf8(&result).expect("UTF-8 CSS");
assert!(output.contains("html:is([data-corner-style=\"square\"],[data-corner-style=\"subtle\"],[data-corner-style=\"rounded\"],[data-corner-style=\"extra\"]) :is("), "lost descendant pseudo-class combinator");
assert!(output.contains("html:is([data-corner-style=square],[data-corner-style=subtle],[data-corner-style=rounded],[data-corner-style=extra]) :is("), "lost descendant pseudo-class combinator");
assert!(output.contains(".card,.stat,dialog,.login-card,"), "lost card selector list");
assert!(output.contains("border-radius:var(--rp-radius-surface)!important"), "lost radius declaration");
assert!(output.contains("border-radius:var(--rp-radius-surface) !important"), "lost radius declaration");
}
}
src/web_server/filesystem.rs
+53 -23
View File
@@ -1,7 +1,32 @@
use super::{CompiledService, SymlinkPolicy};
use std::{fs::File, io, path::Path};
use std::{
fs::File,
io,
path::{Path, PathBuf},
};
#[cfg(unix)]
use std::ffi::CString;
use std::{ffi::CString, sync::Arc};
/// Only filesystem state crosses into the blocking pool. Cloning an entire
/// CompiledService here would also copy MIME, rewrite, PHP and security rules
/// for every file or candidate opened during a request.
pub(super) struct FilesystemRoot {
root: PathBuf,
symlink_policy: SymlinkPolicy,
#[cfg(unix)]
root_handle: Arc<File>,
}
impl FilesystemRoot {
pub(super) fn for_service(service: &CompiledService) -> Self {
Self {
root: service.root.clone(),
symlink_policy: service.symlink_policy,
#[cfg(unix)]
root_handle: service.root_handle.clone(),
}
}
}
#[derive(Debug)]
pub(super) struct Opened {
@@ -54,7 +79,7 @@ pub(super) fn path_components(path: &str) -> io::Result<Vec<String>> {
}
#[cfg(unix)]
pub(super) fn secure_open_sync(service: &CompiledService, components: &[String]) -> io::Result<Opened> {
pub(super) fn secure_open_sync(root: &FilesystemRoot, components: &[String]) -> io::Result<Opened> {
use std::os::unix::io::{AsRawFd, FromRawFd};
if components.is_empty() {
@@ -66,7 +91,7 @@ pub(super) fn secure_open_sync(service: &CompiledService, components: &[String])
let dot = cstring_component(".")?;
let fd = unsafe {
libc::openat(
service.root_handle.as_raw_fd(),
root.root_handle.as_raw_fd(),
dot.as_ptr(),
libc::O_RDONLY | libc::O_CLOEXEC | libc::O_DIRECTORY | libc::O_NOFOLLOW,
)
@@ -78,7 +103,10 @@ pub(super) fn secure_open_sync(service: &CompiledService, components: &[String])
let metadata = current.metadata()?;
return Ok(Opened { file: current, metadata });
}
let mut current = service.root_handle.try_clone()?;
// Borrow the already-owned root descriptor for the first openat. Opening a
// child does not change its directory offset, so a dup/close pair buys no
// isolation. Empty paths still open "." above for independent readdir state.
let mut current: Option<File> = None;
for (index, component) in components.iter().enumerate() {
let name = cstring_component(component)?;
let intermediate = index + 1 != components.len();
@@ -86,7 +114,7 @@ pub(super) fn secure_open_sync(service: &CompiledService, components: &[String])
// explicit Linux-only opt-in: the opened descriptor is checked through
// /proc/self/fd after each resolution step before it can be used further.
let mut flags = libc::O_RDONLY | libc::O_CLOEXEC;
if service.symlink_policy == SymlinkPolicy::Deny {
if root.symlink_policy == SymlinkPolicy::Deny {
flags |= libc::O_NOFOLLOW;
}
if intermediate {
@@ -95,13 +123,14 @@ pub(super) fn secure_open_sync(service: &CompiledService, components: &[String])
// Avoid blocking on special files. Only regular files/directories are served.
flags |= libc::O_NONBLOCK;
}
let fd = unsafe { libc::openat(current.as_raw_fd(), name.as_ptr(), flags) };
let parent = current.as_ref().unwrap_or(root.root_handle.as_ref());
let fd = unsafe { libc::openat(parent.as_raw_fd(), name.as_ptr(), flags) };
if fd < 0 {
return Err(io::Error::last_os_error());
}
if service.symlink_policy == SymlinkPolicy::InsideRoot {
if root.symlink_policy == SymlinkPolicy::InsideRoot {
#[cfg(target_os = "linux")]
verify_linux_fd_beneath_root(service, fd)?;
verify_linux_fd_beneath_root(root, fd)?;
#[cfg(not(target_os = "linux"))]
{
unsafe { libc::close(fd) };
@@ -111,8 +140,9 @@ pub(super) fn secure_open_sync(service: &CompiledService, components: &[String])
));
}
}
current = unsafe { File::from_raw_fd(fd) };
current = Some(unsafe { File::from_raw_fd(fd) });
}
let current = current.expect("nonempty path opened its final component");
let metadata = current.metadata()?;
if !metadata.is_file() && !metadata.is_dir() {
return Err(io::Error::new(
@@ -124,7 +154,7 @@ pub(super) fn secure_open_sync(service: &CompiledService, components: &[String])
}
#[cfg(target_os = "linux")]
fn verify_linux_fd_beneath_root(service: &CompiledService, fd: libc::c_int) -> io::Result<()> {
fn verify_linux_fd_beneath_root(root: &FilesystemRoot, fd: libc::c_int) -> io::Result<()> {
let target = match std::fs::read_link(format!("/proc/self/fd/{fd}")) {
Ok(target) => target,
Err(error) => {
@@ -132,7 +162,7 @@ fn verify_linux_fd_beneath_root(service: &CompiledService, fd: libc::c_int) -> i
return Err(error);
}
};
if target.starts_with(&service.root) {
if target.starts_with(&root.root) {
Ok(())
} else {
unsafe { libc::close(fd) };
@@ -144,13 +174,13 @@ fn verify_linux_fd_beneath_root(service: &CompiledService, fd: libc::c_int) -> i
}
#[cfg(not(unix))]
pub(super) fn secure_open_sync(service: &CompiledService, components: &[String]) -> io::Result<Opened> {
let mut candidate = service.root.clone();
pub(super) fn secure_open_sync(root: &FilesystemRoot, components: &[String]) -> io::Result<Opened> {
let mut candidate = root.root.clone();
for component in components {
candidate.push(component);
}
let canonical = std::fs::canonicalize(&candidate)?;
if !canonical.starts_with(&service.root) {
if !canonical.starts_with(&root.root) {
return Err(io::Error::new(
io::ErrorKind::PermissionDenied,
"path escaped document root",
@@ -169,13 +199,13 @@ pub(super) fn secure_open_sync(service: &CompiledService, components: &[String])
#[cfg(unix)]
pub(super) fn list_directory_sync(
service: &CompiledService,
root: &FilesystemRoot,
components: &[String],
) -> io::Result<Vec<DirectoryEntry>> {
use std::ffi::CStr;
use std::os::unix::io::IntoRawFd;
let opened = secure_open_sync(service, components)?;
let opened = secure_open_sync(root, components)?;
if !opened.metadata.is_dir() {
return Err(io::Error::new(io::ErrorKind::NotADirectory, "not a directory"));
}
@@ -206,7 +236,7 @@ pub(super) fn list_directory_sync(
// Re-open through the root fd using the configured symlink policy. The
// default omits every symlink; Linux inside_root may include a target only
// after the opened descriptor has been proven to remain below the root.
let Ok(child_opened) = secure_open_sync(service, &child) else {
let Ok(child_opened) = secure_open_sync(root, &child) else {
continue;
};
entries.push(DirectoryEntry {
@@ -226,19 +256,19 @@ pub(super) fn list_directory_sync(
#[cfg(not(unix))]
pub(super) fn list_directory_sync(
service: &CompiledService,
root: &FilesystemRoot,
components: &[String],
) -> io::Result<Vec<DirectoryEntry>> {
let opened = secure_open_sync(service, components)?;
let opened = secure_open_sync(root, components)?;
if !opened.metadata.is_dir() {
return Err(io::Error::new(io::ErrorKind::NotADirectory, "not a directory"));
}
let mut path = service.root.clone();
let mut path = root.root.clone();
for component in components {
path.push(component);
}
let canonical = std::fs::canonicalize(path)?;
if !canonical.starts_with(&service.root) {
if !canonical.starts_with(&root.root) {
return Err(io::Error::new(io::ErrorKind::PermissionDenied, "path escaped document root"));
}
let mut entries = Vec::new();
@@ -249,7 +279,7 @@ pub(super) fn list_directory_sync(
};
let mut child = components.to_vec();
child.push(name.clone());
let Ok(child_opened) = secure_open_sync(service, &child) else {
let Ok(child_opened) = secure_open_sync(root, &child) else {
continue;
};
entries.push(DirectoryEntry {
src/web_server/htaccess.rs
+3
View File
@@ -1,3 +1,6 @@
include!("htaccess/model.rs");
include!("htaccess/parse.rs");
include!("htaccess/rewrite.rs");
#[cfg(test)]
mod tests;
src/web_server/htaccess/model.rs
+4 -1
View File
@@ -228,7 +228,10 @@ fn supported_response_header(name: &HeaderName) -> bool {
fn parse_duration(value: &str) -> Result<Duration> {
let value = value.trim().trim_matches('"').trim();
if let Some(seconds) = value.strip_prefix('A').or_else(|| value.strip_prefix('a')) {
// The long form also starts with 'a'; only numeric suffixes use A<seconds>.
if let Some(seconds) = value.strip_prefix('A').or_else(|| value.strip_prefix('a'))
.filter(|seconds| !seconds.is_empty() && seconds.bytes().all(|byte| byte.is_ascii_digit()))
{
let seconds: u64 = seconds.parse().context("invalid Expires duration")?;
return Ok(Duration::from_secs(seconds));
}
src/web_server/htaccess/tests.rs
+48
View File
@@ -0,0 +1,48 @@
use super::parse_duration;
use std::time::Duration;
#[test]
fn expires_duration_accepts_compact_and_access_based_forms() {
for (value, seconds) in [
("A3600", 3600),
("a60", 60),
("A0", 0),
("\"A3600\"", 3600),
("access plus 1 hour", 3600),
("\"access plus 1 hour\"", 3600),
(" ACCESS PLUS 2 HOURS ", 7200),
("access plus 3 seconds", 3),
("access plus 2 minutes", 120),
("access plus 1 day", 86_400),
("access plus 1 week", 604_800),
("access plus 1 month", 2_592_000),
("access plus 1 year", 31_536_000),
] {
assert_eq!(
parse_duration(value).unwrap(),
Duration::from_secs(seconds),
"{value}"
);
}
}
#[test]
fn expires_duration_keeps_invalid_and_unsupported_forms_rejected() {
for value in [
"",
"A",
"A1x",
"A-1",
"A18446744073709551616",
"M3600",
"access",
"access plus",
"access plus two hours",
"access plus -1 hour",
"access minus 1 hour",
"modification plus 1 hour",
"access plus 1 fortnight",
] {
assert!(parse_duration(value).is_err(), "accepted {value:?}");
}
}
src/web_server/response.rs
+1
View File
@@ -1,2 +1,3 @@
include!("response/headers.rs");
mod file_body;
include!("response/file.rs");
src/web_server/response/file.rs
+19 -5
View File
@@ -1,3 +1,12 @@
// Tokio's file reads dispatch blocking work. Larger, bounded chunks avoid one
// dispatch/copy/body-accounting pass per 4 KiB for ordinary static responses.
// Size small files and ranges to their payload so they do not reserve 64 KiB.
const FILE_STREAM_CAPACITY: usize = 64 * 1024;
pub(super) fn file_stream_capacity(body_length: u64) -> usize {
body_length.clamp(1, FILE_STREAM_CAPACITY as u64) as usize
}
pub(super) async fn file_response(
service: &CompiledService,
opened: Opened,
@@ -96,7 +105,7 @@ async fn file_response_inner(
let mut response = Response::new(Body::empty());
*response.status_mut() = status;
let boundary = multipart.then(|| format!("rustproxy-{:016x}", rand::random::<u64>()));
if let Some(boundary) = &boundary {
let response_length = if let Some(boundary) = &boundary {
response.headers_mut().insert(
CONTENT_TYPE,
HeaderValue::from_str(&format!("multipart/byteranges; boundary={boundary}")).unwrap(),
@@ -118,13 +127,16 @@ async fn file_response_inner(
CONTENT_LENGTH,
HeaderValue::from_str(&total.to_string()).unwrap(),
);
total
} else {
response.headers_mut().insert(CONTENT_TYPE, mime.clone());
response.headers_mut().insert(
CONTENT_LENGTH,
HeaderValue::from_str(&body_length.to_string()).unwrap(),
);
}
body_length
};
let stream_capacity = file_stream_capacity(response_length);
response
.headers_mut()
.insert(ACCEPT_RANGES, HeaderValue::from_static("bytes"));
@@ -184,7 +196,9 @@ async fn file_response_inner(
.write_all(format!("--{boundary}--\r\n").as_bytes())
.await;
});
*response.body_mut() = Body::from_stream(ReaderStream::new(reader));
*response.body_mut() = Body::new(file_body::FileBody::new(
ReaderStream::with_capacity(reader, stream_capacity), response_length,
));
return response;
}
if start != 0 {
@@ -195,8 +209,8 @@ async fn file_response_inner(
);
}
}
let stream = ReaderStream::new(file.take(body_length));
*response.body_mut() = Body::from_stream(stream);
let stream = ReaderStream::with_capacity(file.take(body_length), stream_capacity);
*response.body_mut() = Body::new(file_body::FileBody::new(stream, body_length));
response
}
src/web_server/response/file_body.rs
+77
View File
@@ -0,0 +1,77 @@
//! Exact-length, data-only local file delivery.
//!
//! Hyper may stop polling after Content-Length bytes. ReaderStream alone only
//! learns EOF on the next read, leaving cache fills and body accounting open.
//! These streams come from a bounded local file read (or our multipart writer),
//! so their validated byte count is also a trustworthy end-of-stream boundary.
//! Never use this wrapper for arbitrary upstream HTTP bodies with trailers.
use bytes::Bytes;
use futures_util::Stream;
use http_body::{Body, Frame, SizeHint};
use std::{io, pin::Pin, task::{Context, Poll}};
pub(super) struct FileBody<S> {
stream: Option<S>,
remaining: u64,
}
impl<S> FileBody<S> {
pub(super) fn new(stream: S, length: u64) -> Self {
Self { stream: (length != 0).then_some(stream), remaining: length }
}
}
impl<S> Body for FileBody<S>
where
S: Stream<Item = Result<Bytes, io::Error>> + Unpin,
{
type Data = Bytes;
type Error = io::Error;
fn poll_frame(
self: Pin<&mut Self>,
cx: &mut Context<'_>,
) -> Poll<Option<Result<Frame<Bytes>, io::Error>>> {
let this = self.get_mut();
let Some(stream) = this.stream.as_mut() else { return Poll::Ready(None) };
match Pin::new(stream).poll_next(cx) {
Poll::Ready(Some(Ok(data))) => {
if data.len() as u64 > this.remaining {
this.stream = None;
this.remaining = 0;
return Poll::Ready(Some(Err(io::Error::new(
io::ErrorKind::InvalidData, "local file body exceeded its declared length",
))));
}
this.remaining -= data.len() as u64;
if this.remaining == 0 {
// Drop the file/duplex reader now. No additional EOF read
// or caller poll is needed to release it or finish a fill.
this.stream = None;
}
Poll::Ready(Some(Ok(Frame::data(data))))
}
Poll::Ready(None) => {
this.stream = None;
this.remaining = 0;
Poll::Ready(Some(Err(io::Error::new(
io::ErrorKind::UnexpectedEof, "local file body ended before its declared length",
))))
}
Poll::Ready(Some(Err(error))) => {
this.stream = None;
this.remaining = 0;
Poll::Ready(Some(Err(error)))
}
Poll::Pending => Poll::Pending,
}
}
fn is_end_stream(&self) -> bool { self.stream.is_none() }
fn size_hint(&self) -> SizeHint { SizeHint::with_exact(self.remaining) }
}
#[cfg(test)]
mod tests;
src/web_server/response/file_body/tests.rs
+120
View File
@@ -0,0 +1,120 @@
use super::*;
use crate::cache::{self, Cache, Settings};
use axum::{body::Body as AxumBody, response::Response};
use futures_util::stream;
use http::{HeaderMap, HeaderValue, StatusCode};
use http_body_util::BodyExt;
use std::time::Duration;
use tokio::io::AsyncReadExt;
use tokio_util::io::ReaderStream;
#[tokio::test]
async fn final_file_frame_is_terminal_without_an_extra_eof_poll() {
let directory = tempfile::tempdir().unwrap();
let path = directory.path().join("payload");
tokio::fs::write(&path, b"abcdef").await.unwrap();
let file = tokio::fs::File::open(path).await.unwrap();
let mut body = FileBody::new(ReaderStream::with_capacity(file.take(6), 3), 6);
assert_eq!(body.size_hint().exact(), Some(6));
assert!(!body.is_end_stream());
assert_eq!(body.frame().await.unwrap().unwrap().into_data().unwrap(), "abc");
assert_eq!(body.size_hint().exact(), Some(3));
assert!(!body.is_end_stream());
assert_eq!(body.frame().await.unwrap().unwrap().into_data().unwrap(), "def");
assert!(body.is_end_stream(), "the final data frame must end the body before another poll");
assert_eq!(body.size_hint().exact(), Some(0));
assert!(body.stream.is_none(), "the reader/file is released at the length boundary");
assert!(body.frame().await.is_none());
}
#[tokio::test]
async fn short_file_and_local_stream_errors_never_become_successful_eof() {
let mut short = FileBody::new(stream::iter([Ok(Bytes::from_static(b"abc"))]), 6);
assert_eq!(short.frame().await.unwrap().unwrap().into_data().unwrap(), "abc");
assert_eq!(short.frame().await.unwrap().unwrap_err().kind(), io::ErrorKind::UnexpectedEof);
assert!(short.is_end_stream());
assert!(short.frame().await.is_none());
let mut failed = FileBody::new(stream::iter([Err(io::Error::other("read failed"))]), 6);
assert_eq!(failed.frame().await.unwrap().unwrap_err().to_string(), "read failed");
assert!(failed.is_end_stream());
let mut oversized = FileBody::new(stream::iter([Ok(Bytes::from_static(b"abcdef"))]), 3);
assert_eq!(oversized.frame().await.unwrap().unwrap_err().kind(), io::ErrorKind::InvalidData);
assert!(oversized.is_end_stream());
}
#[tokio::test]
async fn file_shrunk_after_open_returns_unexpected_eof() {
let directory = tempfile::tempdir().unwrap();
let path = directory.path().join("payload");
tokio::fs::write(&path, b"abcdef").await.unwrap();
let file = tokio::fs::File::open(&path).await.unwrap();
let expected = file.metadata().await.unwrap().len();
std::fs::OpenOptions::new().write(true).open(path).unwrap().set_len(3).unwrap();
let mut body = FileBody::new(ReaderStream::with_capacity(file.take(expected), 3), expected);
assert_eq!(body.frame().await.unwrap().unwrap().into_data().unwrap(), "abc");
assert_eq!(body.frame().await.unwrap().unwrap_err().kind(), io::ErrorKind::UnexpectedEof);
assert!(body.is_end_stream());
assert!(body.stream.is_none());
}
#[tokio::test]
async fn zero_length_file_is_terminal_without_polling_the_reader() {
let mut body = FileBody::new(stream::pending::<Result<Bytes, io::Error>>(), 0);
assert!(body.is_end_stream());
assert_eq!(body.size_hint().exact(), Some(0));
assert!(body.frame().await.is_none());
}
fn settings() -> Settings { Settings { enabled: true, ..Default::default() } }
fn cacheable(body: AxumBody, length: u64) -> Response {
let mut response = Response::new(body);
response.headers_mut().insert("cache-control", HeaderValue::from_static("public, max-age=600"));
response.headers_mut().insert("content-length", length.to_string().parse().unwrap());
response
}
#[tokio::test]
async fn fully_delivered_file_fills_cache_before_consumer_drops_without_polling_none() {
let directory = tempfile::tempdir().unwrap();
let path = directory.path().join("payload");
tokio::fs::write(&path, b"abcdef").await.unwrap();
let file = tokio::fs::File::open(path).await.unwrap();
let body = FileBody::new(ReaderStream::with_capacity(file.take(6), 6), 6);
let cache = Cache::new(1);
let settings = settings();
let ticket = cache.ticket(1, "http", "localhost", "site", "/file", &HeaderMap::new()).unwrap();
let response = cacheable(AxumBody::new(body), 6);
let plan = cache::plan(StatusCode::OK, response.headers(), &settings, Duration::ZERO).unwrap();
let mut body = cache.wrap(response, ticket.clone(), plan, &settings).into_body();
assert_eq!(body.frame().await.unwrap().unwrap().into_data().unwrap(), "abcdef");
// Reproduce HTTP/1 Content-Length completion: do not ask for another frame.
drop(body);
assert_eq!(cache.view().entries, 1);
assert_eq!(cache.view().rejected_fills, 0);
assert_eq!(cache.view().fills_active, 0);
let hit = cache.get(&ticket).unwrap();
assert_eq!(hit.response.into_body().collect().await.unwrap().to_bytes(), "abcdef");
}
#[tokio::test]
async fn incomplete_or_cancelled_file_cannot_fill_cache() {
for short in [false, true] {
let cache = Cache::new(1);
let settings = settings();
let ticket = cache.ticket(1, "http", "localhost", "site", "/file", &HeaderMap::new()).unwrap();
let chunks = if short { vec![Ok(Bytes::from_static(b"abc"))] }
else { vec![Ok(Bytes::from_static(b"abc")), Ok(Bytes::from_static(b"def"))] };
let response = cacheable(AxumBody::new(FileBody::new(stream::iter(chunks), 6)), 6);
let plan = cache::plan(StatusCode::OK, response.headers(), &settings, Duration::ZERO).unwrap();
let mut body = cache.wrap(response, ticket.clone(), plan, &settings).into_body();
assert_eq!(body.frame().await.unwrap().unwrap().into_data().unwrap(), "abc");
if short { assert!(body.frame().await.unwrap().is_err()); }
drop(body);
assert!(cache.get(&ticket).is_none());
assert_eq!(cache.view().rejected_fills, 1);
assert_eq!(cache.view().fills_active, 0);
}
}
src/web_server/routing.rs
+3 -3
View File
@@ -1,5 +1,5 @@
use super::{
filesystem::{path_components, secure_open_sync, Opened},
filesystem::{path_components, secure_open_sync, FilesystemRoot, Opened},
response::accepts_content_encoding,
CompiledService,
};
@@ -9,8 +9,8 @@ use http::{header::LOCATION, HeaderMap, HeaderValue, StatusCode};
use std::io;
pub(super) async fn open_components(service: &CompiledService, components: Vec<String>) -> io::Result<Opened> {
let service = service.clone();
tokio::task::spawn_blocking(move || secure_open_sync(&service, &components))
let root = FilesystemRoot::for_service(service);
tokio::task::spawn_blocking(move || secure_open_sync(&root, &components))
.await
.map_err(|_| io::Error::other("Web Server filesystem task failed"))?
}
src/web_server/serve.rs
+3 -3
View File
@@ -2,7 +2,7 @@ use super::{
fastcgi::{
method_not_allowed_response, php_for_path, php_response, protected_script_without_handler,
},
filesystem::{list_directory_sync, path_components},
filesystem::{list_directory_sync, path_components, FilesystemRoot},
htaccess,
response::{
add_service_headers, builtin_error_response, directory_listing, directory_redirect,
@@ -345,10 +345,10 @@ async fn serve_inner(
) {
return builtin_error_response(StatusCode::FORBIDDEN, &method);
}
let service_for_list = service.clone();
let root = FilesystemRoot::for_service(service);
let components_for_list = components.clone();
let mut entries = match tokio::task::spawn_blocking(move || {
list_directory_sync(&service_for_list, &components_for_list)
list_directory_sync(&root, &components_for_list)
})
.await
{
src/web_server/tests.rs
+456
View File
@@ -1769,3 +1769,459 @@ async fn php_metrics_are_separate_from_static_and_record_capacity() {
assert_eq!(php.max_concurrent, 2);
assert_eq!(php.permits.available_permits(), 2);
}
#[tokio::test]
async fn static_streaming_keeps_large_payloads_intact_in_bounded_frames() {
let tmp = TempDir::new().unwrap();
let service = service(tmp.path(), false);
let mut accounted = 0u64;
for length in [
0usize,
1,
1040,
16 * 1024 + 3,
64 * 1024,
64 * 1024 + 1,
1024 * 1024,
] {
let expected: Vec<u8> = (0..length).map(|index| (index % 251) as u8).collect();
std::fs::write(tmp.path().join("asset.bin"), &expected).unwrap();
let response =
serve_request(&service, &request(Method::GET, "/asset.bin"), "/asset.bin").await;
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(response.headers()[CONTENT_LENGTH], length.to_string());
let mut body = response.into_body();
let mut received = Vec::new();
let mut frames = 0usize;
while let Some(frame) = body.frame().await {
let data = frame.unwrap().into_data().unwrap();
assert!(
data.len() <= 64 * 1024,
"static frame exceeded the memory bound"
);
received.extend_from_slice(&data);
frames += 1;
}
assert_eq!(received, expected);
if length >= 64 * 1024 {
// A normal regular file must not regress to one dispatched read and
// body-accounting pass for every 4 KiB of its representation.
assert!(frames < length.div_ceil(4 * 1024));
}
accounted += length as u64;
assert_eq!(service.web_metrics.view()["response_bytes"], accounted);
}
}
#[tokio::test]
async fn large_static_ranges_keep_offsets_validators_and_head_semantics() {
let tmp = TempDir::new().unwrap();
let expected: Vec<u8> = (0..256 * 1024 + 31)
.map(|index| (index % 251) as u8)
.collect();
std::fs::write(tmp.path().join("asset.bin"), &expected).unwrap();
let service = service(tmp.path(), false);
let initial = serve_request(&service, &request(Method::HEAD, "/asset.bin"), "/asset.bin").await;
let etag = initial.headers()[ETAG].clone();
assert!(initial
.into_body()
.collect()
.await
.unwrap()
.to_bytes()
.is_empty());
for (range, start, end) in [
("bytes=0-0", 0, 1),
("bytes=65530-131089", 65530, 131090),
("bytes=-65543", expected.len() - 65543, expected.len()),
("bytes=131065-", 131065, expected.len()),
] {
for method in [Method::GET, Method::HEAD] {
let mut requested = request(method.clone(), "/asset.bin");
requested
.headers_mut()
.insert(RANGE, range.parse().unwrap());
requested.headers_mut().insert(IF_RANGE, etag.clone());
let response = serve_request(&service, &requested, "/asset.bin").await;
assert_eq!(response.status(), StatusCode::PARTIAL_CONTENT);
assert_eq!(
response.headers()[CONTENT_LENGTH],
(end - start).to_string()
);
assert_eq!(
response.headers()[CONTENT_RANGE],
format!("bytes {start}-{}/{}", end - 1, expected.len())
);
assert_eq!(response.headers()[ETAG], etag);
let data = response.into_body().collect().await.unwrap().to_bytes();
if method == Method::HEAD {
assert!(data.is_empty());
} else {
assert_eq!(&data[..], &expected[start..end]);
}
}
}
let mut changed = request(Method::GET, "/asset.bin");
changed
.headers_mut()
.insert(RANGE, "bytes=65530-131089".parse().unwrap());
changed
.headers_mut()
.insert(IF_RANGE, "\"changed\"".parse().unwrap());
let response = serve_request(&service, &changed, "/asset.bin").await;
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(
response.into_body().collect().await.unwrap().to_bytes(),
expected
);
let mut conditional = request(Method::GET, "/asset.bin");
conditional
.headers_mut()
.insert(RANGE, "bytes=65530-131089".parse().unwrap());
conditional.headers_mut().insert(IF_NONE_MATCH, etag);
let response = serve_request(&service, &conditional, "/asset.bin").await;
assert_eq!(response.status(), StatusCode::NOT_MODIFIED);
assert!(response
.into_body()
.collect()
.await
.unwrap()
.to_bytes()
.is_empty());
let mut invalid = request(Method::GET, "/asset.bin");
invalid
.headers_mut()
.insert(RANGE, "bytes=999999-".parse().unwrap());
let response = serve_request(&service, &invalid, "/asset.bin").await;
assert_eq!(response.status(), StatusCode::RANGE_NOT_SATISFIABLE);
assert_eq!(
response.headers()[CONTENT_RANGE],
format!("bytes */{}", expected.len())
);
assert!(response
.into_body()
.collect()
.await
.unwrap()
.to_bytes()
.is_empty());
}
#[tokio::test]
async fn large_precompressed_and_multipart_representations_remain_exact() {
use std::io::Write;
let tmp = TempDir::new().unwrap();
let plain: Vec<u8> = (0..512 * 1024)
.map(|index| b'a' + (index % 26) as u8)
.collect();
let mut encoder = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::none());
encoder.write_all(&plain).unwrap();
let compressed = encoder.finish().unwrap();
std::fs::write(tmp.path().join("asset.css"), &plain).unwrap();
std::fs::write(tmp.path().join("asset.css.gz"), &compressed).unwrap();
let service = Service {
id: "large-compressed".into(),
name: "Large compressed".into(),
document_root: tmp.path().to_path_buf(),
precompressed: Precompressed {
enabled: true,
brotli: false,
gzip: true,
},
..Default::default()
}
.compile()
.unwrap();
let mut requested = request(Method::GET, "/asset.css");
requested
.headers_mut()
.insert(http::header::ACCEPT_ENCODING, "gzip".parse().unwrap());
let response = serve_request(&service, &requested, "/asset.css").await;
assert_eq!(response.headers()[CONTENT_ENCODING], "gzip");
assert_eq!(response.headers()[CONTENT_TYPE], "text/css; charset=utf-8");
assert_eq!(response.headers()[VARY], "Accept-Encoding");
assert_eq!(
response.headers()[CONTENT_LENGTH],
compressed.len().to_string()
);
let etag = response.headers()[ETAG].clone();
assert_eq!(
response.into_body().collect().await.unwrap().to_bytes(),
compressed
);
*requested.method_mut() = Method::HEAD;
let response = serve_request(&service, &requested, "/asset.css").await;
assert_eq!(response.headers()[CONTENT_ENCODING], "gzip");
assert_eq!(
response.headers()[CONTENT_LENGTH],
compressed.len().to_string()
);
assert!(response
.into_body()
.collect()
.await
.unwrap()
.to_bytes()
.is_empty());
*requested.method_mut() = Method::GET;
requested.headers_mut().insert(IF_NONE_MATCH, etag);
let response = serve_request(&service, &requested, "/asset.css").await;
assert_eq!(response.status(), StatusCode::NOT_MODIFIED);
assert_eq!(response.headers()[CONTENT_ENCODING], "gzip");
assert!(response
.into_body()
.collect()
.await
.unwrap()
.to_bytes()
.is_empty());
requested.headers_mut().remove(IF_NONE_MATCH);
requested
.headers_mut()
.insert(RANGE, "bytes=65530-131089".parse().unwrap());
let response = serve_request(&service, &requested, "/asset.css").await;
assert_eq!(response.status(), StatusCode::PARTIAL_CONTENT);
assert_eq!(response.headers()[CONTENT_ENCODING], "gzip");
assert_eq!(
response.into_body().collect().await.unwrap().to_bytes(),
compressed[65530..131090]
);
requested
.headers_mut()
.insert(RANGE, "bytes=65520-131124,262100-458800".parse().unwrap());
let response = serve_request(&service, &requested, "/asset.css").await;
assert_eq!(response.status(), StatusCode::PARTIAL_CONTENT);
assert!(!response.headers().contains_key(CONTENT_ENCODING));
let content_type = response.headers()[CONTENT_TYPE].to_str().unwrap();
let boundary = content_type
.strip_prefix("multipart/byteranges; boundary=")
.unwrap();
let mut expected = Vec::new();
for (start, end) in [(65520, 131124), (262100, 458800)] {
write!(&mut expected, "--{boundary}\r\nContent-Type: text/css; charset=utf-8\r\nContent-Range: bytes {start}-{end}/{}\r\n\r\n", plain.len()).unwrap();
expected.extend_from_slice(&plain[start..=end]);
expected.extend_from_slice(b"\r\n");
}
write!(&mut expected, "--{boundary}--\r\n").unwrap();
assert_eq!(
response.headers()[CONTENT_LENGTH],
expected.len().to_string()
);
let multipart_length = expected.len();
assert_eq!(
response.into_body().collect().await.unwrap().to_bytes(),
expected
);
*requested.method_mut() = Method::HEAD;
let response = serve_request(&service, &requested, "/asset.css").await;
assert_eq!(response.status(), StatusCode::PARTIAL_CONTENT);
assert!(!response.headers().contains_key(CONTENT_ENCODING));
assert_eq!(
response.headers()[CONTENT_LENGTH],
multipart_length.to_string()
);
assert!(response
.into_body()
.collect()
.await
.unwrap()
.to_bytes()
.is_empty());
}
#[cfg(unix)]
#[test]
fn filesystem_context_owns_root_and_preserves_independent_directory_offsets() {
use std::io::Read;
let tmp = TempDir::new().unwrap();
std::fs::create_dir(tmp.path().join("nested")).unwrap();
std::fs::write(tmp.path().join("top.txt"), b"top").unwrap();
std::fs::write(tmp.path().join("nested/file.txt"), b"nested").unwrap();
let service = service(tmp.path(), true);
let root = filesystem::FilesystemRoot::for_service(&service);
drop(service);
for _ in 0..3 {
let components = filesystem::path_components("/nested/file.txt").unwrap();
let mut opened = filesystem::secure_open_sync(&root, &components).unwrap();
let mut contents = String::new();
opened.file.read_to_string(&mut contents).unwrap();
assert_eq!(contents, "nested");
let entries = filesystem::list_directory_sync(&root, &[]).unwrap();
assert_eq!(
entries
.iter()
.map(|entry| entry.name.as_str())
.collect::<Vec<_>>(),
["nested", "top.txt"]
);
let entries = filesystem::list_directory_sync(&root, &["nested".into()]).unwrap();
assert_eq!(entries.len(), 1);
assert_eq!(entries[0].name, "file.txt");
}
}
#[cfg(target_os = "linux")]
#[tokio::test]
async fn filesystem_context_keeps_symlink_policy_on_intermediate_components() {
use std::os::unix::fs::symlink;
let tmp = TempDir::new().unwrap();
let outside = TempDir::new().unwrap();
std::fs::create_dir(tmp.path().join("safe")).unwrap();
std::fs::write(tmp.path().join("safe/asset.txt"), b"safe").unwrap();
std::fs::write(outside.path().join("asset.txt"), b"secret").unwrap();
symlink(tmp.path().join("safe"), tmp.path().join("inside-link")).unwrap();
symlink(outside.path(), tmp.path().join("outside-link")).unwrap();
for policy in [SymlinkPolicy::Deny, SymlinkPolicy::InsideRoot] {
let service = Service {
id: "links".into(),
name: "Links".into(),
document_root: tmp.path().to_path_buf(),
directory_listing: true,
symlink_policy: policy,
..Default::default()
}
.compile()
.unwrap();
let response = serve_request(
&service,
&request(Method::GET, "/inside-link/asset.txt"),
"/inside-link/asset.txt",
)
.await;
if policy == SymlinkPolicy::InsideRoot {
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(
response.into_body().collect().await.unwrap().to_bytes(),
"safe"
);
} else {
assert_ne!(response.status(), StatusCode::OK);
}
let response = serve_request(
&service,
&request(Method::GET, "/outside-link/asset.txt"),
"/outside-link/asset.txt",
)
.await;
assert_ne!(response.status(), StatusCode::OK);
assert!(!response
.into_body()
.collect()
.await
.unwrap()
.to_bytes()
.windows(6)
.any(|part| part == b"secret"));
for _ in 0..3 {
let response = serve_request(&service, &request(Method::GET, "/"), "/").await;
assert_eq!(response.status(), StatusCode::OK);
let body = response.into_body().collect().await.unwrap().to_bytes();
let listing = std::str::from_utf8(&body).unwrap();
assert!(listing.contains("safe/"));
assert!(!listing.contains("outside-link"));
assert_eq!(
listing.contains("inside-link"),
policy == SymlinkPolicy::InsideRoot
);
}
}
}
#[cfg(unix)]
#[tokio::test]
async fn borrowed_root_descriptor_stays_anchored_after_root_path_replacement() {
let tmp = TempDir::new().unwrap();
let configured_root = tmp.path().join("site");
std::fs::create_dir_all(configured_root.join("nested")).unwrap();
std::fs::write(configured_root.join("nested/asset.txt"), b"original").unwrap();
let service = service(&configured_root, false);
std::fs::rename(&configured_root, tmp.path().join("moved-site")).unwrap();
std::fs::create_dir_all(configured_root.join("nested")).unwrap();
std::fs::write(
configured_root.join("nested/asset.txt"),
b"replacement-secret",
)
.unwrap();
for _ in 0..3 {
let response = serve_request(
&service,
&request(Method::GET, "/nested/asset.txt"),
"/nested/asset.txt",
)
.await;
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(
response.into_body().collect().await.unwrap().to_bytes(),
"original"
);
}
}
#[cfg(target_os = "linux")]
fn open_descriptors_for(path: &Path) -> usize {
std::fs::read_dir("/proc/self/fd")
.unwrap()
.filter_map(|entry| std::fs::read_link(entry.ok()?.path()).ok())
.filter(|target| target == path)
.count()
}
#[cfg(target_os = "linux")]
async fn wait_for_file_descriptors_to_close(path: &Path) {
tokio::time::timeout(Duration::from_secs(5), async {
while open_descriptors_for(path) != 0 {
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await
.expect("cancelled static response retained its file descriptor");
}
#[cfg(target_os = "linux")]
#[tokio::test]
async fn cancelled_static_and_multipart_bodies_release_files_and_count_only_emitted_bytes() {
let tmp = TempDir::new().unwrap();
let path = tmp.path().join("asset.bin");
std::fs::write(&path, vec![b'x'; 1024 * 1024]).unwrap();
let service = service(tmp.path(), false);
let unpolled = serve_request(&service, &request(Method::GET, "/asset.bin"), "/asset.bin").await;
drop(unpolled);
wait_for_file_descriptors_to_close(&path).await;
assert_eq!(service.web_metrics.view()["response_bytes"], 0);
let mut accounted = 0u64;
for range in [None, Some("bytes=0-400000,600000-1000000")] {
let mut requested = request(Method::GET, "/asset.bin");
if let Some(range) = range {
requested
.headers_mut()
.insert(RANGE, range.parse().unwrap());
}
let mut body = serve_request(&service, &requested, "/asset.bin")
.await
.into_body();
let first = body.frame().await.unwrap().unwrap().into_data().unwrap();
assert!(!first.is_empty());
assert!(first.len() <= 64 * 1024);
accounted += first.len() as u64;
assert_eq!(service.web_metrics.view()["response_bytes"], accounted);
// The multipart writer may fill its bounded duplex buffer in the
// background. Neither it nor a pending file read counts unsent bytes.
tokio::task::yield_now().await;
assert_eq!(service.web_metrics.view()["response_bytes"], accounted);
drop(body);
wait_for_file_descriptors_to_close(&path).await;
assert_eq!(service.web_metrics.view()["response_bytes"], accounted);
}
}
tests/transport_config.rs
+4 -1
View File
@@ -20,7 +20,10 @@ fn local_listener_backend_requires_explicit_risk_acceptance() {
let error = compile(config.clone(), 1, &policy())
.expect_err("local administration listener must be rejected by default");
assert!(error.to_string().contains("accept_local_listener_risk=true"));
// Backend compilation adds vhost/member context; inspect the full cause
// chain so the assertion still verifies the actual risk gate.
let detail = format!("{error:#}");
assert!(detail.contains("accept_local_listener_risk=true"), "{detail}");
config.vhosts[0].backends[0].accept_local_listener_risk = true;
compile(config, 2, &policy())