0.34 #2

Merged
gru merged 16 commits from 0.34 into master 2026-10-18 12:01:51 +02:00
18 changed files with 360 additions and 65 deletions
Showing only changes of commit 47ce1f18c2 - Show all commits

No files matched your search

+17 -2
View File
@@ -5,6 +5,20 @@
## 0.34.0
- DEV 1 KiB follow-up (OPT4): keep ordinary unmodified public request targets
borrowed through URI rewriting, avoid creating a metering body adapter for
request bodies with no frames/trailers, and consume already-validated upstream
URI ownership rather than cloning on the ordinary HTTP/1.1 path. Reuse the
formatted IPv4 peer address in RFC 7239 forwarding metadata, fast-path
short ASCII access-log fields, and skip nominated-hop parsing for absent,
plain `close` or single `keep-alive` Connection headers. Preserve cache
refresh owned targets, streamed upload metering, full hop-by-hop filtering
for complex values, TLS/H2/H3 authority handling and request limits.
- OPT3 compiled binary passed 151 live HTTP checks, 0 failed, 1 skipped
(optional Zstandard). OPT4 Rust sources must still be compiled and benchmarked;
see `docs/PROXY_1KB_OPT4_DEV_0.34.0.md`.
- DEV 1 KiB throughput follow-up (OPT3): keep normalized safe routing
paths borrowed from an independently retained URI instead of allocating
during observation and forwarding; use the original strict percent/decode
@@ -16,8 +30,9 @@
body streaming, analytics, metrics and defaults remain unchanged.
- Add regressions for path validator equivalence, multi-valued hop-by-hop
fields and untrusted IPv6 forwarding. User-supplied OPT2-FIX1 binary passed
89 targeted live checks. **This newest source has not been compiled or
RPS-tested**; see `docs/PROXY_1KB_OPT3_DEV_0.34.0.md`.
89 targeted live checks. A compiled OPT3 executable was subsequently tested
(151 passing HTTP cases; one skipped), but short shared-host A/B did not
establish a stable RPS gain; see `docs/PROXY_1KB_OPT3_DEV_0.34.0.md`.
- DEV 1 KiB accounting follow-up: move per-domain HIT/MISS/BYPASS, reason,
stored-entry and saved-body statistics outside the HOT cache's shared mutex.
+16 -2
View File
@@ -1,13 +1,27 @@
# RustProxy 0.34.0
## DEV: OPT4 small-response proxy hot path (uncompiled sources)
The compiled OPT3 binary passed 151 real HTTP acceptance checks (1 optional
Zstandard case skipped). OPT4 removes avoidable allocations and string work
for ordinary uncached HTTP/1.1 requests, with regressions for empty request
bodies, URI/Host/Connection semantics and Unicode log truncation. OPT4 has
**not been compiled or RPS-measured**. No settings or functionality disabled;
version remains 0.34.0. Details:
[`docs/PROXY_1KB_OPT4_DEV_0.34.0.md`](docs/PROXY_1KB_OPT4_DEV_0.34.0.md).
## DEV: small-response proxy hot path, OPT3 source candidate
Seven compatible optimizations reduce path copies and work constructing
forwarded headers, log IPs, hop-by-hop headers, request IDs and cache-off
reason totals. HTTP/1.1 proxy with a 1,040-byte response remains the first
benchmark priority; local Web Server is secondary. No version or policy changes.
The **new code is not compiled or RPS-verified** here. See
[`docs/PROXY_1KB_OPT3_DEV_0.34.0.md`](docs/PROXY_1KB_OPT3_DEV_0.34.0.md).
A subsequently supplied compiled OPT3 executable passed 151 real HTTP
assertions (one optional codec skipped). Two short same-host benchmark runs
did **not** establish a stable RPS gain over OPT2. See the historical
[`OPT3 source report`](docs/PROXY_1KB_OPT3_DEV_0.34.0.md)
and the newer [`OPT4 analysis`](docs/PROXY_1KB_OPT4_DEV_0.34.0.md).
## DEV: cache counter contention and saturated analytics (source candidate)
+10
View File
@@ -2,6 +2,16 @@
## 0.34.0 - Scoped cache correctness and proxy body hot path
- [x] OPT4 source candidate: borrow original no-rewrite public target,
bypass the request body adapter only for bodies reporting end-of-stream,
avoid redundant upstream URI refcount operations, and simplify common
forwarding/header/log branches without weakening validation.
- [x] Add OPT4 regression tests for body accounting and permits, Unicode,
URI identity, and safe Connection header filtering.
- [ ] Compile and run OPT4 Rust tests, re-test ACL/TLS/H2/H3/WebSocket/cache;
collect A/B-nginx RPS and profiling evidence before accepting a gain.
- [x] DEV 1 KiB: remove shared HOT store mutex from cache outcome/reason
accounting; preserve domain cap, exact quiescent totals and reset rules.
- [x] Reserve bounded analytics queue capacity before allocating dropped samples;
+17 -15
View File
@@ -2,12 +2,12 @@
4140a2956264b44f259e6a37fc00ba02097bbe68b09296e04d3d243c1287129b .env.example
9e6356475bf587091bab72246ec9e325b498f7e48a6220e9a18baa50d0a385a6 .gitignore
46bab1d2c2d6e0a80d57a6b232b1ba4c220884b95cd45ddccbafce2aa160cddb AGENTS.md
10f70748d2c070e2b9923b5aa6572347c7977a88f8bb5d4abe532827914bade7 CHANGELOG.md
16b657a20824fd310c1eab6498b0b0db6f250a56b09c556d9c149ce1f61f0c0a CHANGELOG.md
ca201e3cd3f9842eae0006d0b40a0a72bcd3d4da6e4e5ecaf200d1b9a1390db2 Cargo.lock
fec5f1bae9a59ef1fabd27c5e45d31448dd8f811736eebfa8826b3806a7e3349 Cargo.toml
25c7f3c8162cbf5cc156439774118a81dae9fdc52b611ec7e40c230fc99a772c Dockerfile
09234b68d69eb10047396baab4bf5811b37e954ad3f5eaacf90247265d4fbd52 README.md
aee9bbb367e061aeb79916e38d121e7767f6f9d0c16a2dcc4b28f091586a33b1 ROADMAP.md
73194a4738e42a3c36623e750022bd2ef6dc9483c287c583a0a172d815962454 README.md
2c387b854dc5433c3dd4a8cd1e70272f2fd8c5d5a483cd9f7fe9b4ceeed95c36 ROADMAP.md
da8311e3db945290666324a2f24bf4521b365272d7ac3f862ded1be993eea646 UPDATE_CHECK.toml
f645f218dd062bf6cb54a8ee634290562cad364a4cccf09f16e3c8303582af8c build.rs
ea8d091843f86f4c66a5c76b20268b2feb397070851b6e05c5bc7d7a8d9157eb deploy/README.md
@@ -59,6 +59,7 @@ ad6e3f003b820a4f45e30e39577d120a69d167c70910bda4891971759458c3dd docs/PROMETHEU
401ab1bb2c7f3961742195d81f5daf5c9cf81609a4b175bea1d10d2a4b727082 docs/PROXY_1KB_COUNTERS_DEV_0.34.0.md
f13ee0005d57d2576ccc970745f17069096df4afa23d712ea0f748da85433f7d docs/PROXY_1KB_HOTPATH_DEV_0.34.0.md
c9bbdd1cd8df02b2a19016b7bbeee1cf55917614ac672f1f94e0a05db7d309ef docs/PROXY_1KB_OPT3_DEV_0.34.0.md
27002c4d4e19bb7d58521b8ba4cd5faf180d2917b6ac200dd7fc3664c2060145 docs/PROXY_1KB_OPT4_DEV_0.34.0.md
deea89aed410c0619bdbdb12437edc7aff9dca2fec7fdc059e918c649fc36b51 docs/PROXY_HOTPATH_0.34.0.md
a7d6189b0e8cb854918456e0063101ee77bfb2419a9ba0430eb0e3f8678c0f09 docs/PROXY_RPS_0.34.0.md
887d1fdcc3b546b024afcd18b1716b8e3a74a3800012bf4bda7a122d2eabe233 docs/README.md
@@ -184,7 +185,8 @@ fa42ec0475e4398d647b3c094309b4332cd2b5649db223cae0b8e821d6b44a28 scripts/tests/
72185f59f31bf04a4e88d53fe2fca52509f3663cb5f40b1d3bc319a5f945ef23 scripts/tests/panel_html.mjs
b0136544af717b75df2cb107900ea65c3f945fdd0040369dbb93919008fb954c scripts/tests/panel_html.py
8d1903e2b17b3d59cd0a5bec3e298e468b3d88ed2dda8b724239cc45d8353c59 scripts/tests/progressbar_0332_browser_test.py
e07652d8a9d170325779009e6d4a826a4d3cfc57515d73ada5f8717dd7a2aca6 scripts/tests/proxy_1kb_opt3_static_test.py
7fc86a34a47c6e1f0f9fe2748d0ea7f59e455f8bf4e5ad081e185356e43721b0 scripts/tests/proxy_1kb_opt3_static_test.py
ec05272ad8c14c7a0265a84cc7b6ebc0dca0651413106d655681681873b88fc8 scripts/tests/proxy_1kb_opt4_static_test.py
2a6e75c05f3860acf65ea8132f0083904478b690145115ec017c120d70f26cca scripts/tests/proxy_hotpath_034_static_test.py
6fa2f22816996e989dad9dc7884adbf24f02f0ca6a6634b06e219eb08526c7fe scripts/tests/rps_hotpath_034_static_test.py
5be5cdc60324f4f0f6638640000b888a5babb0a9bbf29511c824a0ba38878c2a scripts/tests/sendfile_034_panel_browser_test.py
@@ -393,7 +395,7 @@ b45d6040810c93ae2acb04ef6b0a59c189751edf9008cd81a4521c9e04e7f07d src/docker.rs
4c2c988191a97f81568a8df64af772463d010844f1abf327871901cdb32d4336 src/docker/model.rs
5f4610c59b2d4ff6a205a2d9ae11bc84be689b2a68302459f546b3f43e53593c src/docker/reconcile.rs
a12c47df4b90b6fc36c978a42477e6ddffbbbf69aed7252101c2fd829709bd6e src/docker/tests.rs
ed17a905bc249b4c36dfd07711a87bea9f385a01d16264f798e1bae932dc51a0 src/forwarding.rs
2cb42ac0a5444fdcfea34c616dd6fe5f5ea1d06c84bcd64714ddf096830ec99f src/forwarding.rs
b25fbb00de0e37ad5fa1a049839265e5f7e9f83b6214c40a95f57ea4807685bc src/forwarding/tests.rs
af7a4e91c32ffbe4701c8c06e48ba3f8bbbe55e8b9e877aecb30ccdf858e9e8f src/geoip.rs
80850f7acd1ed4c0048b065e9e81740869a8efbb0d9b5d2e83d8ec7fbc494703 src/geoip/mmdb.rs
@@ -441,33 +443,33 @@ d3c6b08088df0003144544936a55e2a3661aac85292821b594f00adfd19a2951 src/prometheus
e03f0d6b3f35dae6b98c394837aa0d17e4d73a8306767b4c607868bc00c30286 src/prometheus/vhosts_backends.rs
fb2d8d45cd2e947f783eb981b37fa418c39fcb327a3af17a648fadea7f37f888 src/prometheus/web_php.rs
2c97367686b54989573ce7fb81b2c6dde72adb254df9522bc05ded74f1fbffb6 src/proxy/body.rs
c9b76abd91f0dfe81e07073e89d35faa033c19e11fa88b223f075428542284f1 src/proxy/body/tests.rs
66b7f2cb47f2e70dc26b7b19eb8650872e34d3e44cf250d2effdb758295f8b8a src/proxy/body/tests.rs
818dcfefa1c40d6a722dacf53c7f45c53141240f1c1913cfa651e19dd2718526 src/proxy/cache_helpers.rs
aa49c9928bb91e3d5bae0d340d88396a0417b9edec6192d41e1403533319a897 src/proxy/cache_http_invalidation.rs
676c1b325fd8e55bcdd86451c7d1d8b8d347ca00e8da6b3e5b148a58c376a9b4 src/proxy/cache_plan.rs
3404b03096862de1050088f62cdd811ea1ab61d191f902501173107f81f23b9b src/proxy/cache_plan/tests.rs
a8848cbdf85368548b003af2b43823c453866c472af4c7d2ff0ec7da14c44c8d src/proxy/error_fallback.rs
5670cd3557d862660596d8f9163a510aab8651206b5e61ae0f735a1a8a3c4313 src/proxy/forward/cache_lookup.rs
f2f9cae1c74eaf66fc365c0709a6eb64e0b27457d4a00728e02cff8a641a1295 src/proxy/forward/cache_lookup.rs
cfa941b60b5d167ec9d8e2566ea3342b588d1a9078a6ca711ad6062ddf00fec9 src/proxy/forward/response.rs
7a5f43fe65e5a746eb84881da0bf784215081ead66d18faa7780533ad5df9c8c src/proxy/forward/route_cache.rs
8b7d9802616fde19a5be3ae2d2e4703e066f203e6a60a6bd781d3a2c15388348 src/proxy/forward/success_finish.rs
44e1bae0a35ddae6a084c1d50b936fd10dd51059f331525b758447b526a2bddf src/proxy/forward/route_cache.rs
691b54a0cc33089e67d61ffc715b5c015976bae3fbd453b26cb06764be2482e5 src/proxy/forward/success_finish.rs
0a921474ef2b93eabdaf3888651cf29130eace5ccb1e0f5d6bc25e6aa69280be src/proxy/forward/success_prepare.rs
bb24da9c9ab986a9c4d4144d4d1be0f661561d4afc755f298b58bc679b182850 src/proxy/forward/upstream.rs
3f21439bbe4966fff43b3ebce8b59fa76ecce376197b4c56a2dce3a6d98f5556 src/proxy/forward/upstream.rs
bb8c9117e05b0cb4bbfe029ef4d1a031fc755cdce255b15dad072afd1c845737 src/proxy/forward/web_source.rs
2b6ae70b84fd1eba0dba6d8d7e2db62227193dc5b77e4a6111ceab1b219800c2 src/proxy/forward_entry.rs
9b25f5e3ea4443a63ee6f45d12e1957dd316fe8e85325fc00ea416faaab1025a src/proxy/h2_graceful.rs
31e925a408ba6273deb0793df5f5f82eeabbcaffb365cc0457ed9c67d9200f34 src/proxy/headers.rs
013f0495127e351cf535a928526b56bd3a63202d8abd0fa59da5dfb8dc783f27 src/proxy/headers/tests.rs
cc7fe83f42acb19d79abe83107427cdc6b0ed14083702f4fe66af22f7cdfc4cc src/proxy/headers.rs
0f9314d1380febe436def36b5f381731e97cbac90625637575f38f3d72eaaace src/proxy/headers/tests.rs
3f25429418941eb72c3847125340db74684ad316eb278ecf658953026fd41f4f src/proxy/mod.rs
55133ceace2e5262d9c54fa41bea099a27b817d1e09c1cd0e2493fd9cee4da38 src/proxy/redirect_helpers.rs
01b3027cfe92512f71be056a1ae26749b3ad3417d43388f1c05dcf5884d83966 src/proxy/request_id.rs
d2df55b6ccfc7ceb8a3d2c49f664acd183448e3d932d5399d6692e87f79a094b src/proxy/request_id/tests.rs
e56c65c85e42ee4fe6adf3655f3b2d176798e37c648db6bfb8e1602300b723eb src/proxy/tests/authority_tests.rs
e8d1210a6b8e8c6a388d016d22e7c18255cf737c5583124b35cd50bd2592f23c src/proxy/tests/authority_tests.rs
1aa9bbbfb2659facdaeaf185e11221b1c8cd5fe6e23c7c1dacd3aa55e3db4fcb src/proxy/tests/cache_fast_path_tests.rs
0f2eacadf1447db8aa1c1a78005aa262e0f747096bfe7e0a0e41a2cd31a0d17f src/proxy/tests/h3_navigation_tests.rs
d134b0eb617f82f9971b3f3461379ea37c57bd0079bf54748d0c434b2ef1b475 src/proxy/tests/http_invalidation_tests.rs
7789a80202d606dcea3776345464a5487bf6124e4878bf8c94db91b67f0d4eb8 src/proxy/types.rs
1cbca5fbe91f40447e9eadc349e2d56f220119d14504f07ce90ed70f945a767e src/proxy/upstream_helpers.rs
a74815a0b46d1c94ffbf8cc3a8bbbc967daa4aec0981139d2d92624c10553026 src/proxy/types.rs
9a8e38e88bff389e448f54dc07e63fef9652acc72e05833363668c28949daccf src/proxy/upstream_helpers.rs
2d9142e3ae95895b5271b4009ab48236a3a2074ce51179c021193b35777560e4 src/proxy/web_cache_revalidation.rs
d7b4b809b12ee89293df45e9a64b1c9981fb3f89deaab3762f4a3ad3992dd7b1 src/proxy/web_cache_revalidation/tests.rs
69f775f6746b4c6d729dc10961dbef54168846f4ce9cf7fe453ab5a2b879c9b8 src/proxy/websocket.rs
+97
View File
@@ -0,0 +1,97 @@
# RustProxy 0.34.0 DEV / OPT4: HTTP/1.1 small-body follow-up
## Objective and validity
Primary goal: 1,040-byte reverse-proxy response, HTTP/1.1 keep-alive,
cache disabled; target about 100k RPS and at least 80% of nginx on the **same
isolated host**. Secondary goal: local static Web Server, benchmarked separately.
No version bump or feature, metric, logging, ACL, authentication, TLS,
HTTP/2/3, stream, retry or cache-policy bypass is introduced.
The **provided and compiled OPT3 binary** (SHA-256
`2efe273340f9cf92c5900c85c0bd1b775c87a9d1eaff1a10ec7635d268b080b1`)
passed 151 black-box HTTP assertions: 0 failures and 1 optional Zstandard
case skipped. This is evidence for the **input** binary, not for the new
OPT4 sources. OPT4 still requires a native Rust build, cargo tests and a
post-build benchmark. The diagnostic same-host CPU shares resources and is
not an acceptance benchmark.
## Code changes relative to OPT3
1. `src/proxy/forward/route_cache.rs`: store normal public path/query as
`Cow<'_, str>` referencing the independently retained, immutable original
URI. Without a path rewrite both public/origin suffixes stay borrowed;
strict encoded/rewrite handling remains an owned validated string.
Cache refresh tasks explicitly call `.to_string()` only when they actually
must retain an owned target across spawned tasks.
2. `src/proxy/upstream_helpers.rs`: skip allocating a request-body metering
adapter **only** when `Body::is_end_stream()` establishes that it contains
no DATA frames or trailers. Every nonempty/streaming request retains the
original metering, timeout, backend lease accounting and cancellation.
The response adapter still retains the exchange/inflight permit and handles
completion/response bytes for empty requests.
3. `src/proxy/types.rs`, `src/proxy/forward/upstream.rs`: consume owned
`http::Uri` when preparing the upstream target. HTTP/1.1 returns it directly;
H2 takes the same URI into parts for strict authority rewriting. Remove an
unnecessary intermediate URI clone; replay still builds independent URIs.
4. `src/forwarding.rs`: reuse the formatted IPv4 peer address when creating
RFC 7239 `Forwarded` as well as X-Real-IP/X-Forwarded-For. Preserve quoted
IPv6 bracket and proxy-chain logic for trusted and untrusted peers.
5. `src/proxy/upstream_helpers.rs`: fast-path byte-safe truncation of short
or ASCII host/path log values, keeping the original max-*characters*
semantics for non-ASCII UTF-8.
6. `src/proxy/headers.rs`: avoid nominated-field token parsing for absent,
lone `keep-alive`, or lone `close` Connection values. A `close` field on
the wire still forces the general parser, so a nominated `Close` header
is removed exactly as before. Multi-value, comma lists and malformed tokens
retain the previous defensive parser.
7. `src/proxy/headers.rs`: similarly avoid the generic token parser when
checking HTTP upgrade on plain keep-alive, close, or absent Connection.
Complex multi-valued requests and explicit Upgrade remain unchanged.
## Tests and limits
- New Rust cases validate empty vs nonempty request-body permits and bytes,
H1 URI/authority/query, ASCII/Unicode truncation, and Connection variants
including a literal nominated `Close` field. Existing regression coverage
still includes sticky backends, cache, ACL and trusted forwarded identity.
- Source, SQLite migration and upgrade tests are necessary but **not** Rust
compilation or performance evidence. The compile check must run before use.
- Run `./dev.sh check && CARGO_INCREMENTAL=0 ./dev.sh test`, followed by
the native black-box acceptance suite (proxy, policy, cache, transform,
compression, static serving), plus TLS/H2/H3 and websocket tests.
## Measurements
An initial diagnostic same-host `ab` sample at concurrency 16 gave
OPT2 33,042.55 RPS and compiled OPT3 36,962.11 RPS, without HTTP errors;
this single sample is not proof of a reliable uplift. A separate complete
12-sample run (two 3-second repetitions each at 16/64 clients) produced:
| Concurrency | OPT2 median RPS | OPT3 median RPS | nginx median RPS | OPT3/nginx |
| --- | ---: | ---: | ---: | ---: |
| 16 | 21,608 | 29,906 | 92,586 | 32.3% |
| 64 | 38,060 | 34,781 | 123,809 | 28.1% |
All 12 samples had valid 200 responses and correct 1040-byte body checks.
OPT3 was slower than OPT2 at concurrency 64; the variability is too large to
claim a sustained uplift or regression based on two short same-host samples.
Raw evidence includes run order, response checksum and nginx comparator.
No result here measures OPT4, which was not built in this environment.
## Next evaluation commands
```bash
./dev.sh check
CARGO_INCREMENTAL=0 ./dev.sh test
bash dev.sh
python3 scripts/performance/regression.py --binary opt4=/absolute/path/target/debug/rust-proxy --out /tmp/opt4-http
python3 scripts/performance/ab_1kb_dev.py --binary opt3=/absolute/path/opt3/rust-proxy --binary opt4=/absolute/path/target/debug/rust-proxy --nginx /usr/sbin/nginx --ab /usr/bin/ab --out /tmp/opt4-ab --duration 5 --repeats 3 --concurrency 16,64 --workers 4 --max-idle 64
```
For authoritative acceptance: pin client/server/origin to nonoverlapping CPU
sets, align build profile and all flags, compare 30s x7 runs, record median,
p95/p99, correct 1040 B response checksum and status, CPU, RSS, FDs, cache
stats, analytic/sink queue drops, errors, and both HTTP/1.1 and optional
HTTP/2 endpoints. Reject changes on new 4xx/5xx, body mismatches, regressions
of access controls or material memory/fd increases.
+1 -1
View File
@@ -22,7 +22,7 @@ for name in ('src/proxy/forward/cache_lookup.rs', 'src/proxy/forward/success_fin
assert 'if !trusted_peer' in read('src/forwarding.rs')
assert 'let mut ips = chain(incoming).unwrap_or_default();' in read('src/forwarding.rs')
headers = read('src/proxy/headers.rs')
assert 'let only_keep_alive' in headers
assert 'let no_nominations' in headers
assert 'let nominated: Vec<HeaderName>' in headers
assert 'headers.remove(name)' in headers
assert 'String::from_utf8(encoded)' in read('src/proxy/request_id.rs')
+52
View File
@@ -0,0 +1,52 @@
#!/usr/bin/env python3
"""OPT4 source invariants: not compilation or performance proof."""
from pathlib import Path
r = Path(__file__).resolve().parents[2]
def source(name):
return (r / name).read_text(encoding="utf-8")
assert 'version = "0.34.0"' in source('Cargo.toml')
route = source('src/proxy/forward/route_cache.rs')
assert "Cow<'_, str>" in route
assert 'routing_uri.path_and_query()' in route
assert 'Ok(None) => public_suffix.clone()' in route
assert 'std::borrow::Cow::Owned(encoded)' in route
for name in ('src/proxy/forward/cache_lookup.rs','src/proxy/forward/success_finish.rs'):
assert 'public_suffix: public_suffix.clone()' not in source(name)
assert 'public_suffix: public_suffix.to_string()' in source(name)
helpers = source('src/proxy/upstream_helpers.rs')
assert 'fn wrap_observed_request' in helpers
assert 'if body.is_end_stream() { body }' in helpers
assert 'let request = request.map(|body| wrap_observed_request' in helpers
assert 'fn log_prefix' in helpers
assert 'input.is_ascii()' in helpers and 'input.chars().take(max_chars).collect()' in helpers
assert 'path: log_prefix(request.uri().path(), 256)' in helpers
assert 'pub request_origin_uri: http::Uri' in source('src/config/model.rs')
target = source('src/proxy/types.rs')
assert 'fn upstream_request_target(\n uri: Uri' in target
assert 'return Some(uri);' in target
assert 'let mut parts = uri.into_parts();' in target
assert 'upstream_request_target(uri, request.headers_mut()' in target
assert 'upstream_request_target(\n uri, request.headers_mut()' in source('src/proxy/forward/upstream.rs')
forward = source('src/forwarding.rs')
assert 'peer_node.as_deref().unwrap_or(&peer_text)' in forward
assert 'let mut ips = chain(incoming).unwrap_or_default();' in forward
headers = source('src/proxy/headers.rs')
assert 'let no_nominations' in headers
assert 'eq_ignore_ascii_case(b"close")' in headers
assert 'let nominated: Vec<HeaderName>' in headers
for name, test in [
('src/proxy/body/tests.rs','optimized_empty_request_keeps_exchange_until_response'),
('src/proxy/body/tests.rs','optimized_nonempty_request_still_meters_data_and_holds_capacity'),
('src/proxy/tests/authority_tests.rs','optimized_log_prefix_preserves_unicode_char_count_and_ascii_bytes'),
('src/proxy/tests/authority_tests.rs','moved_uri_keeps_h1_origin_and_query'),
('src/proxy/headers/tests.rs','absent_and_close_connection_preserve_non_hop_headers'),
]:
assert test in source(name), (name, test)
for p in (r / 'src').rglob('*.rs'):
if '/tests/' in p.as_posix() or p.name == 'tests.rs':
continue
assert len(p.read_text(encoding='utf-8').splitlines()) <= 500, p
assert not source('src/cache/stats.rs').startswith('//!')
print('PASS OPT4 source guard (STATIC ONLY; no cargo / no OPT4 benchmark)')
+2 -1
View File
@@ -225,9 +225,10 @@ impl Compiled {
let original_url = format!("{scheme}://{authority_text}{target}");
let peer_text = peer_ip.to_string();
let peer_header = HeaderValue::from_str(&peer_text)?;
let peer_node = if peer_ip.is_ipv4() { None } else { Some(node(peer_ip)) };
let forwarded = format!(
"for={};by={};proto={scheme};host=\"{authority_text}\"",
node(peer_ip), node(canonical(local.ip()))
peer_node.as_deref().unwrap_or(&peer_text), node(canonical(local.ip()))
);
let mut values = HeaderMap::with_capacity(7);
values.insert("forwarded", forwarded.parse()?);
+30
View File
@@ -46,6 +46,36 @@ mod tests {
});
(exchange, slots)
}
#[tokio::test]
async fn optimized_empty_request_keeps_exchange_until_response() {
let (exchange, slots) = observation().await;
let metrics = exchange.metrics.clone();
let body = super::super::wrap_observed_request(
Body::empty(), &exchange, Duration::from_secs(1),
);
assert!(body.is_end_stream());
drop(body);
assert_eq!(slots.available_permits(), 0);
drop(exchange);
assert_eq!(slots.available_permits(), 1);
assert_eq!(metrics.completed.load(Ordering::Relaxed), 1);
}
#[tokio::test]
async fn optimized_nonempty_request_still_meters_data_and_holds_capacity() {
let (exchange, slots) = observation().await;
let metrics = exchange.metrics.clone();
let body = super::super::wrap_observed_request(
Body::from("payload"), &exchange, Duration::from_secs(1),
);
assert!(!body.is_end_stream());
drop(exchange);
assert_eq!(slots.available_permits(), 0);
assert_eq!(body.collect().await.unwrap().to_bytes().len(), 7);
assert_eq!(slots.available_permits(), 1);
assert_eq!(metrics.bytes_in.load(Ordering::Relaxed), 7);
}
#[tokio::test]
async fn permit_and_metrics_cover_both_bodies() {
let (exchange, slots) = observation().await;
+3 -3
View File
@@ -40,7 +40,7 @@ if let Some(ticket) = &cache_ticket {
cache_settings: cache_settings.clone(),
transform: transform_settings.cloned(),
uri: uri.clone(),
public_suffix: public_suffix.clone(),
public_suffix: public_suffix.to_string(),
host: host.clone(),
path: path.as_ref().to_owned(),
scope: scope.to_owned(),
@@ -77,7 +77,7 @@ if let Some(ticket) = &cache_ticket {
cache_settings: cache_settings.clone(),
transform: transform_settings.cloned(),
uri: uri.clone(),
public_suffix: public_suffix.clone(),
public_suffix: public_suffix.to_string(),
host: host.clone(),
path: path.as_ref().to_owned(),
scope: scope.to_owned(),
@@ -238,7 +238,7 @@ if handshake.is_none()
cache_settings: cache_settings.clone(),
transform: transform_settings.cloned(),
uri: uri.clone(),
public_suffix: public_suffix.clone(),
public_suffix: public_suffix.to_string(),
host: host.clone(),
path: path.as_ref().to_owned(),
scope: scope.to_owned(),
+6 -7
View File
@@ -169,12 +169,11 @@ let body_empty = if cache_enabled
} else {
cache_enabled && request.body().is_end_stream()
};
let public_suffix = request
.uri()
.path_and_query()
.map(|v| v.as_str())
.unwrap_or("/")
.to_owned();
// Keep the original URI alive through forwarding: ordinary targets can be
// borrowed while route rewrites still allocate and are strictly validated.
let public_suffix: std::borrow::Cow<'_, str> = std::borrow::Cow::Borrowed(
routing_uri.path_and_query().map(|v| v.as_str()).unwrap_or("/")
);
let cache_plan = cache_plan::CachePlan::new(policy_reason, cache_settings, &path, || {
cache_settings.active_for(route.cache) && route
.frontend
@@ -200,7 +199,7 @@ let origin_suffix = match route.rewritten_path(&path) {
encoded.push('?');
encoded.push_str(query);
}
encoded
std::borrow::Cow::Owned(encoded)
}
Err(_) => return error(StatusCode::BAD_REQUEST, "RustProxy: invalid rewritten request target\n"),
};
+1 -1
View File
@@ -168,7 +168,7 @@ if let (Some(ticket), Some(plan), Some(transformed)) =
cache_settings: cache_settings.clone(),
transform: transform_settings.cloned(),
uri: active_uri.clone(),
public_suffix: public_suffix.clone(),
public_suffix: public_suffix.to_string(),
host: host.clone(),
path: path.as_ref().to_owned(),
scope: scope.to_owned(),
+1 -1
View File
@@ -92,7 +92,7 @@ let retry_host_follows_backend = replay_headers.as_ref()
.and_then(|headers| headers.get(HOST))
.is_some_and(|value| value == expected_backend_host);
let Some(upstream_target) = upstream_request_target(
&uri, request.headers_mut(), &backend,
uri, request.headers_mut(), &backend,
websocket_legacy_support && !websocket_upstream_h2, websocket_upstream_h2,
) else {
return error(StatusCode::BAD_GATEWAY, "RustProxy: invalid upstream authority\n");
+18 -10
View File
@@ -7,14 +7,17 @@ pub fn strip_hop_by_hop(headers: &mut HeaderMap) {
// The common HTTP/1.1 "Connection: keep-alive" case has no dynamic
// nominated headers to collect. Keep the general multi-value/token
// parser for all other requests (including maliciously repeated fields).
let only_keep_alive = {
let no_nominations = {
let mut values = headers.get_all(CONNECTION).iter();
values.next().is_some_and(|value| {
value.as_bytes().eq_ignore_ascii_case(b"keep-alive")
&& values.next().is_none()
})
match (values.next(), values.next()) {
(None, _) => true,
(Some(value), None) => value.as_bytes().eq_ignore_ascii_case(b"keep-alive")
|| (value.as_bytes().eq_ignore_ascii_case(b"close")
&& !headers.contains_key("close")),
_ => false,
}
};
if !only_keep_alive {
if !no_nominations {
let nominated: Vec<HeaderName> = headers
.get_all(CONNECTION)
.iter()
@@ -41,10 +44,15 @@ pub fn strip_hop_by_hop(headers: &mut HeaderMap) {
}
}
pub fn requests_upgrade(headers: &HeaderMap) -> bool {
headers.contains_key("upgrade")
|| headers
.get_all(CONNECTION)
.iter()
if headers.contains_key("upgrade") { return true; }
let mut connection = headers.get_all(CONNECTION).iter();
match (connection.next(), connection.next()) {
(None, _) => return false,
(Some(value), None) if value.as_bytes().eq_ignore_ascii_case(b"keep-alive")
|| value.as_bytes().eq_ignore_ascii_case(b"close") => return false,
_ => {}
}
headers.get_all(CONNECTION).iter()
.filter_map(|v| v.to_str().ok())
.flat_map(|v| v.split(','))
.any(|v| v.trim().eq_ignore_ascii_case("upgrade"))
+40
View File
@@ -19,6 +19,21 @@ mod tests {
h.insert("connection", "keep-alive, Upgrade".parse().unwrap());
assert!(requests_upgrade(&h));
}
#[test]
fn common_connection_tokens_do_not_request_websocket_upgrade() {
for connection in [None, Some("keep-alive"), Some("KEEP-ALIVE"), Some("close")] {
let mut headers = HeaderMap::new();
if let Some(value) = connection { headers.insert("connection", value.parse().unwrap()); }
assert!(!requests_upgrade(&headers));
}
let mut headers = HeaderMap::new();
headers.insert("connection", "close, Upgrade".parse().unwrap());
assert!(requests_upgrade(&headers));
headers.insert("connection", "keep-alive".parse().unwrap());
headers.insert("upgrade", "websocket".parse().unwrap());
assert!(requests_upgrade(&headers));
}
#[test]
fn absent_cookies_leave_headers_unchanged() {
let mut headers = HeaderMap::new();
@@ -68,6 +83,31 @@ mod tests {
assert_eq!(headers["authorization"], "Bearer allowed");
}
}
#[test]
fn absent_and_close_connection_preserve_non_hop_headers() {
for value in [None, Some("close"), Some("CLOSE")] {
let mut headers = HeaderMap::new();
if let Some(value) = value { headers.insert("connection", value.parse().unwrap()); }
headers.insert("x-request-policy", "allowed".parse().unwrap());
headers.insert("transfer-encoding", "chunked".parse().unwrap());
strip_hop_by_hop(&mut headers);
assert!(!headers.contains_key("connection"));
assert!(!headers.contains_key("transfer-encoding"));
assert_eq!(headers["x-request-policy"], "allowed");
}
}
#[test]
fn connection_close_still_strips_a_close_field_if_present() {
let mut headers = HeaderMap::new();
headers.insert("connection", "close".parse().unwrap());
headers.insert("close", "private-hop-value".parse().unwrap());
headers.insert("x-public", "allowed".parse().unwrap());
strip_hop_by_hop(&mut headers);
assert!(!headers.contains_key("close"));
assert_eq!(headers["x-public"], "allowed");
}
#[test]
fn multiple_connection_fields_still_strip_all_nominated_headers() {
let mut headers = HeaderMap::new();
+23 -2
View File
@@ -58,7 +58,7 @@ mod authority_tests {
let uri = upstream_uri(backend, "/block_refresh/block/10").unwrap();
let mut headers = HeaderMap::new();
headers.insert(HOST, HeaderValue::from_static("unitraklub.pl"));
let target = upstream_request_target(&uri, &mut headers, backend, false, false).unwrap();
let target = upstream_request_target(uri, &mut headers, backend, false, false).unwrap();
assert_eq!(target.authority().unwrap().as_str(), expected_authority, "{protocol:?}");
assert_eq!(target.path(), "/block_refresh/block/10");
assert_eq!(headers.contains_key(HOST), !remove_host);
@@ -66,6 +66,27 @@ mod authority_tests {
}
}
#[test]
fn optimized_log_prefix_preserves_unicode_char_count_and_ascii_bytes() {
assert_eq!(log_prefix("/small.html", 256), "/small.html");
assert_eq!(log_prefix("abcdefghijklmnopqrstuvwxyz", 5), "abcde");
assert_eq!(log_prefix("za\u{17c}\u{f3}\u{142}\u{107}\u{1f525}", 4),
"za\u{17c}\u{f3}");
assert_eq!(log_prefix("", 0), "");
}
#[test]
fn moved_uri_keeps_h1_origin_and_query() {
let config = crate::config::demo_config("127.0.0.1:9000".parse().unwrap());
let snapshot = crate::config::compile(config, 1, &validation_policy()).unwrap();
let backend = &snapshot.pools[0].members[0].backend;
let uri = upstream_uri(backend, "/keep?key=hello%2Fworld").unwrap();
let mut headers = HeaderMap::new();
let target = upstream_request_target(uri, &mut headers, backend, false, false).unwrap();
assert_eq!(target.path_and_query().unwrap().as_str(), "/keep?key=hello%2Fworld");
assert_eq!(target.authority().unwrap(), backend.request_origin_uri.authority().unwrap());
}
#[test]
fn invalid_outgoing_host_is_rejected_instead_of_used_for_dialing() {
let mut config = crate::config::demo_config("127.0.0.1:9000".parse().unwrap());
@@ -76,7 +97,7 @@ mod authority_tests {
let uri = upstream_uri(backend, "/").unwrap();
let mut headers = HeaderMap::new();
headers.insert(HOST, HeaderValue::from_static("evil@example.invalid"));
assert!(upstream_request_target(&uri, &mut headers, backend, false, false).is_none());
assert!(upstream_request_target(uri, &mut headers, backend, false, false).is_none());
}
#[test]
+4 -4
View File
@@ -180,7 +180,7 @@ fn legacy_websocket_origin(
/// mode must retain the Host field: Hyper's legacy H1-compatible request may
/// be sent over an H2-negotiated connection, and H1 fallback needs Host.
fn upstream_request_target(
uri: &Uri,
uri: Uri,
headers: &mut HeaderMap,
backend: &crate::config::CompiledBackend,
websocket_h1: bool,
@@ -193,7 +193,7 @@ fn upstream_request_target(
crate::config::UpstreamProtocol::Http2 | crate::config::UpstreamProtocol::H2 => true,
};
if !h2_capable {
return Some(uri.clone());
return Some(uri);
}
let selected = headers.get(HOST).unwrap_or(&backend.host_header);
let authority: http::uri::Authority = selected.to_str().ok()?.parse().ok()?;
@@ -201,7 +201,7 @@ fn upstream_request_target(
if authority.host().is_empty() || authority.as_str().contains('@') {
return None;
}
let mut parts = uri.clone().into_parts();
let mut parts = uri.into_parts();
parts.authority = Some(authority);
let target = Uri::from_parts(parts).ok()?;
let strict_h2_request = websocket_h2
@@ -259,7 +259,7 @@ fn retry_request(
},
);
}
let target = upstream_request_target(&uri, request.headers_mut(), backend, false, false)?;
let target = upstream_request_target(uri, request.headers_mut(), backend, false, false)?;
*request.uri_mut() = target;
Some(request)
}
+22 -16
View File
@@ -1,5 +1,20 @@
include!("request_id.rs");
// Retain the exchange through the response, but do not allocate an additional
// body adapter for a request whose body has no frames or trailers.
fn wrap_observed_request(body: Body, exchange: &Arc<Exchange>, idle: Duration) -> Body {
if body.is_end_stream() { body }
else { Body::new(MeteredBody::new(body, exchange.clone(), Direction::Request, idle)) }
}
fn log_prefix(input: &str, max_chars: usize) -> String {
if input.len() <= max_chars || input.is_ascii() {
input[..input.len().min(max_chars)].to_owned()
} else {
input.chars().take(max_chars).collect()
}
}
fn should_replace_error_page(response: &Response, minimal_welcome: bool) -> bool {
!minimal_welcome
&& response
@@ -117,14 +132,12 @@ async fn observe(State(state): State<Arc<App>>, mut request: Request, next: Next
if let Some(frontend) = &frontend_context {
frontend.metrics.active.fetch_add(1, Ordering::Relaxed);
}
let host = request
.headers()
.get(HOST)
let host = log_prefix(
request.headers().get(HOST)
.and_then(|h| h.to_str().ok())
.unwrap_or_else(|| request.uri().authority().map(|v| v.as_str()).unwrap_or(""))
.chars()
.take(253)
.collect();
.unwrap_or_else(|| request.uri().authority().map(|v| v.as_str()).unwrap_or("")),
253,
);
let initial_client_ip = request
.extensions()
.get::<PeerAddr>()
@@ -164,7 +177,7 @@ async fn observe(State(state): State<Arc<App>>, mut request: Request, next: Next
request_id: request_id.clone(),
method: request.method().to_string(),
host,
path: request.uri().path().chars().take(256).collect(),
path: log_prefix(request.uri().path(), 256),
http_version,
transport,
websocket_bootstrap,
@@ -200,14 +213,7 @@ async fn observe(State(state): State<Arc<App>>, mut request: Request, next: Next
.as_ref()
.and_then(|frontend| frontend.proxy_timeouts.send_timeout())
.unwrap_or(state.body_idle_timeout);
let request = request.map(|body| {
Body::new(MeteredBody::new(
body,
exchange.clone(),
Direction::Request,
request_send_timeout,
))
});
let request = request.map(|body| wrap_observed_request(body, &exchange, request_send_timeout));
let mut response = next.run(request).await;
let final_route_context = response
.extensions()