gru/RustProxy
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v0.24.7

Browse Source
This commit is contained in:
Mateusz Gruszczyński committed 2026-10-03 10:05:46 +02:00
1 parent 5e23e9af02
commit 3883a21bc4
56 files changed
+1640 -403
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/API.md
+6
View File
@@ -309,6 +309,12 @@ existing audit log.
The `transform` configuration section is replaceable through the normal revision-CAS
`PUT /api/config/transform` endpoint. Vhost objects carry a `transform` override and
Docker assignments carry `transform_enabled`; both default to disabled per vhost.
Global transform settings also expose resource controls: `memory_budget_bytes` (1 MiB..2 GiB,
default 128 MiB), `max_body_bytes` (1 KiB..32 MiB, default 2 MiB), and `max_inflight`
(1..1024, default 32). `preset` is optional UI intent metadata with values
`conservative`, `recommended`, `performance`, or `uber`; numeric limits remain authoritative.
Resource pressure is fail-open: work that cannot reserve the configured memory/concurrency
capacity is left unchanged rather than queued behind full-body transformation work.
`GET /api/transform` returns in-memory transformation observability for the administration
Transform workspace: totals and per-vhost counters for examined/applied/skipped/error
docs/API_ACCESS.md
+1 -1
View File
@@ -1,4 +1,4 @@
# Administration API access - 0.24.5
# Administration API access - 0.24.7
## Sign-in boundary
docs/ARCHITECTURE.md
+1 -1
View File
@@ -33,7 +33,7 @@ committed document at startup; this is not one cross-memory/SQLite transaction.
| `auth`, `admin` | Administrator lifecycle, local API, browser sessions, CSRF, SSE and static UI |
| `db` | SQLite schema, migrations, desired-state persistence and revision checks |
| `metrics`, `logging` | Aggregate instrumentation, bounded access ring and trace output |
| `transform` | Bounded decode/transform stage for eligible static identity representations before cache/compression |
| `transform` | Bounded decode/transform stage for eligible static identity representations before cache/compression, with live process-wide memory and concurrency ceilings |
| `docker` | Desired Docker-agent settings, in-memory observations and ephemeral vhost reconciliation |
| `runtime` | Shared process state, snapshot updates, admission limits and tunnel tracking |
| `demo` | A separate opt-in demonstration backend with its own listener |
docs/CACHE.md
+4 -2
View File
@@ -240,8 +240,10 @@ existing deduplication guard.
## Diagnostics and API
`GET /api/cache` reports settings, HIT/MISS/BYPASS, hot/cold counts and bytes,
promotions/demotions, errors, fill pressure, object views and per-domain reason counters.
Hit ratio is `hits / (hits + misses)` and intentionally excludes BYPASS traffic.
configured COLD byte/object limits, promotions/demotions, errors, fill pressure, object
views and per-domain reason counters. The overview dashboard shows COLD disk occupancy
separately from RAM accounting. Hit ratio is `hits / (hits + misses)` and intentionally
excludes BYPASS traffic.
`POST /api/cache/purge` requires the existing administrator session, same-origin marker
and CSRF token. `?vhost=<id>` purges only that vhost scope; omitting it purges all cache
docs/INSTALL.md
+1 -1
View File
@@ -1,4 +1,4 @@
# Installing and updating RustProxy 0.24.5
# Installing and updating RustProxy 0.24.7
The scripts prepare a Linux/systemd installation. This source archive is **not a
verified production build**: Rust/Cargo, real CA operations, live systemd startup
docs/README.md
+2
View File
@@ -13,6 +13,8 @@ This index separates current operational documentation from historical/version-l
- [PROMETHEUS.md](PROMETHEUS.md), [HTTP3.md](HTTP3.md), [STREAMS.md](STREAMS.md) - monitoring and transport features.
- [SECURITY.md](SECURITY.md) - current security model and limitations.
- [TEST_REPORT.md](TEST_REPORT.md) - release verification status and explicit environment limitations.
- [0.24.7 transform resource-control audit](V0.24.7_AUDIT.md) - configurable transformation memory/concurrency budgets and resource presets.
- [0.24.6 memory/preset audit](V0.24.6_AUDIT.md) - persisted cache presets, transformation memory bounds and background-worker memory/task review.
- [0.24.5 specification audit](V0.24.5_AUDIT.md) - HTTP/3 empty-request-body normalization for HTTP/1.1 upstreams and DELETE regression matrix.
- [0.24.4 specification audit](V0.24.4_AUDIT.md) - process-local Linux THP mitigation and memory-retention verification matrix.
- [0.24.3 specification audit](V0.24.3_AUDIT.md) - memory-retention, H3 settings placement and dynamic diagnostics source-vs-spec matrix.
docs/TEST_REPORT.md
+17 -30
View File
@@ -1,23 +1,21 @@
# Verification report - RustProxy 0.24.5
# Verification report - RustProxy 0.24.7
## Scope
RustProxy 0.24.5 is a protocol-framing and cache-memory maintenance release over 0.24.4. It keeps the 0.24.4 process-local THP mitigation, fixes the H3 -> H1 boundary for actually empty request bodies whose length is initially unknown, and hardens the existing HOT RAM -> COLD disk cache lifecycle without changing the COLD storage format.
RustProxy 0.24.7 makes transformation memory/concurrency limits configurable and adds persisted RAM saver, Recommended, Performance and Uber performance resource presets. The same version now exposes live transform resource usage, overview process/cache RAM accounting and COLD disk occupancy. It retains the 0.24.6 cache/background-worker memory hardening.
## Source contracts reviewed
- `ReceiveBody` in `src/http3.rs` reports end-of-stream only after the H3 DATA side reaches EOF and trailers are resolved; absence of `Content-Length` alone is not treated as proof of an empty body.
- `normalize_h3_h1_unknown_body()` runs only for HTTP/3 requests selected for an HTTP/1.1 upstream and only when no declared/exact body length is already available.
- EOF replaces the request body with `Body::empty()`. Non-empty first frames are replayed through `ReplayBody` and the remaining request body stays streaming.
- HTTP/2 upstreams, non-H3 downstreams and declared `Content-Length` requests are not pre-read by this normalization.
- The change is generic to body-capable methods; there is no Proxmox-specific or DELETE-to-H2 fallback.
- Published HOT objects are bounded by `max_bytes` using body backing capacity plus directly owned metadata and a conservative HOT-index charge; logical payload bytes are reported separately.
- Fill buffers reserve a bounded transient budget before growth, enforce `max_object_bytes` per chunk, and make HOT LRU room before further cache-buffer allocation. Failure leaves the response streaming uncached.
- HOT -> COLD demotion moves the `Entry` rather than cloning the payload. The `Bytes` allocation remains charged to transient RAM while queued/writing and disappears from the persisted `ColdEntry` after the write completes.
- The COLD writer queue is bounded by both object count and bytes; full COLD reads/promotions/revalidation reads reserve transient RAM before loading a body.
- Global `cache.enabled=false` is authoritative over vhost overrides. HOT/object values no longer have the old artificial 1 GiB configuration ceiling.
- Settings exposes conservative/recommended/performance presets derived from effective host/cgroup memory, plus non-blocking high-budget/headroom and inconsistent transient/object/queue warnings. A 2 GiB effective-memory VM gets approximately 400 MiB HOT / 96 MiB transient / 16 MiB max object / 48 MiB COLD queue under the recommended profile.
- Release identity is 0.24.5 across Cargo/package/UI/OpenAPI/release helpers. The existing dependency graph is unchanged; only the root `rust-proxy` package version changes.
- Transform `preset`, `memory_budget_bytes` and `max_inflight` are part of the normal revisioned Settings document and therefore persist in SQLite; the reopen test covers them.
- Existing configurations default to 128 MiB transform temporary memory and 32 concurrent attempts.
- New transform attempts use atomic runtime byte/concurrency reservations and fail open with fixed `memory_pressure` / `concurrency_pressure` reasons rather than queueing full-body work.
- Presets modify only resource limits; transform type switches and extension allow-lists remain independent.
- Global transform representation/eligibility settings participate in cache compatibility identity. Resource-only preset/budget/concurrency fields do not.
- OpenAPI and both language packs expose the new controls and preset states.
- `GET /api/status` exposes `transform_resources` and `memory_accounting`; the latter keeps process RSS separate from HOT/transient cache accounting and treats transform bytes as an overlapping conservative reservation.
- Cache runtime telemetry exposes COLD byte/object limits so the overview can show disk occupancy against configured capacity without mixing disk and RAM.
- `GET /api/transform` exposes current process-wide reserved transform bytes and active slots alongside the existing counters.
- 0.24.6 aggregate/decode bounds, allocation reductions and bounded cache/background workers remain in place.
## Verification status in this workspace
@@ -25,28 +23,17 @@ RustProxy 0.24.5 is a protocol-framing and cache-memory maintenance release over
|---|---|
| `python3 scripts/preflight.py` | PASS |
| `python3 scripts/langs_check.py` | PASS |
| Browser JavaScript `node --check` | PASS |
| `python3 scripts/memory_contracts.py` | PASS |
| Browser JavaScript `node --check` | PASS for all administration JS bundles |
| OpenAPI/language JSON parse | PASS |
| Shell `bash -n` | PASS |
| Python syntax / bytecode compile | PASS |
| Docker agent `go test ./...` / `go vet ./...` / `go build ./...` | PASS |
| Existing `Cargo.lock` dependency graph | PRESERVED; root package version updated to 0.24.5 |
| `python3 scripts/check_source.py` | PASS |
| `python3 scripts/release_smoke.py` | PASS |
| `./dev.sh fmt` / `cargo fmt --all` | BLOCKED: Rust/Cargo toolchain unavailable in this workspace |
| `./dev.sh check` / `cargo check --locked --all-targets` | BLOCKED: Rust/Cargo toolchain unavailable in this workspace |
| `./dev.sh test` Rust phase | BLOCKED: Rust/Cargo toolchain unavailable in this workspace |
| H1/H2/H3 live method matrix | test-host acceptance gate |
| Proxmox DELETE H3 access-log verification | test-host acceptance gate |
| HOT 8 MiB sustained-load/RSS acceptance | test-host acceptance gate |
| HOT 2 GiB bounded-admission acceptance | test-host acceptance gate |
| 5 s HOT -> COLD demotion/RSS/transient drain | test-host acceptance gate |
| Oversized-object streaming without full cache buffering | source regression added; runtime acceptance gate remains |
| Concurrent COLD read/write under small transient budget | test-host acceptance gate |
| Sustained per-preset transform RSS/heap profiling | release-host acceptance gate |
## Protocol acceptance target
Run GET/POST/PUT/PATCH/DELETE over H1, H2 and H3. For DELETE, test both an empty body and a body-bearing request (use a mock upstream if the chosen Proxmox endpoint does not accept a DELETE body). The original H3 empty DELETE must no longer produce `501`; H1/H2 behavior must remain unchanged; Proxmox must receive `DELETE` and the same URI; and a raw HTTP/1.1 upstream must show non-chunked zero-length framing for the empty H3 case.
For cache acceptance, test a small 8 MiB HOT tier under sustained traffic, a deliberately large 2 GiB HOT tier, `cold_after_seconds=5`, a response above `max_object_bytes`, and concurrent COLD promotion/demotion under a small transient limit. Published `hot_allocated_bytes` must remain within the HOT limit; transient/queue usage must drain after COLD persistence; large responses must keep streaming after their cache candidate is rejected; and RSS must plateau rather than track total traffic volume.
See [V0.24.5_AUDIT.md](V0.24.5_AUDIT.md) for the source/spec matrix and remaining runtime gates.
See [V0.24.7_AUDIT.md](V0.24.7_AUDIT.md) for the resource-control design and remaining runtime acceptance gate.
docs/TRANSFORM.md
+5 -1
View File
@@ -16,10 +16,14 @@ statistics globally or per vhost:
- average transform time;
- applied transforms by JavaScript/CSS/HTML/SVG/JSON;
- bounded skip/fail-open reason counters;
- current process-wide temporary-memory reservation versus the configured budget;
- current active transform slots versus the configured concurrency limit;
- a per-vhost effectiveness table.
Counters reset on process restart. They are diagnostic telemetry, not persisted cache
metadata. The Cache workspace separately marks stored transformed objects.
metadata. Resource usage is process-wide even when a vhost counter scope is selected. The
reservation is the transform limiter's conservative buffer estimate, not a second RSS
measurement. The Cache workspace separately marks stored transformed objects.
## Cache diagnostics
docs/V0.24.6_AUDIT.md
+57
View File
@@ -0,0 +1,57 @@
# RustProxy 0.24.6 memory and cache-preset audit
## Scope
0.24.6 persists cache-preset intent and hardens temporary/background memory use. The cache numeric limits remain the runtime source of truth; `memory_preset` and `cold_preset` are metadata stored in the same revisioned JSON configuration row in SQLite so the administration UI can restore the selected state after reload.
This audit is source/static in the packaging workspace because Rust/Cargo and heap profilers are unavailable here. It adds compiler/runtime tests to the source tree and an offline memory-contract gate, but sustained RSS/heap acceptance must still run on a compiler-enabled release host.
## Cache preset persistence
| Area | Contract |
|---|---|
| Cache model | `Settings.memory_preset` accepts conservative/recommended/performance; `Settings.cold_preset` accepts recommended; old configurations deserialize both as `None`. |
| SQLite | `Store::replace` serializes the complete cache settings into `configuration.document`; the database reopen test now verifies both preset values survive close/reopen. |
| Cache identity | Preset metadata is excluded from `cache_revision()`, so changing only the selected label cannot purge HOT/COLD data. Effective numeric/policy changes still rotate cache identity normally. |
| UI | Settings restores the selected button/badge from configuration. Manual edits clear the affected preset metadata. |
| Recommended COLD | Enables COLD, idle/LRU/shutdown demotion, disables age demotion, sets 300 s idle, 3600 s max-hot, 60 s GC and 256 max-delete, and chooses a queue around half transient RAM. Disk path, disk bytes and entry/object limits are preserved. |
## Transformation memory review
| Path | 0.24.6 behavior |
|---|---|
| Concurrent transforms | A process-wide 128 MiB semaphore budget accounts for collected input plus expected decoded/output buffers. Transform is fail-open with `memory_pressure` when the budget is busy. |
| Response aggregation | `http_body_util::Limited` enforces `max_body_bytes` even when upstream `Content-Length` is false; over-limit aggregation becomes a body error instead of allocating without bound. |
| Decompression | Fixed 16 KiB reads and exact bounded reserve checks reject expansion beyond `max_body_bytes`. |
| JSON | `IgnoredAny` validates syntax without materializing `serde_json::Value`; whitespace compaction is one pass and preserves number lexemes/key order. |
| HTML | Raw element detection compares tag slices case-insensitively instead of allocating/lowercasing a tag name for every element. |
| CSS | Tracks the last output character instead of repeatedly scanning the full output string. |
| Large-buffer release | Existing glibc `malloc_trim(0)` throttling remains after successful large transforms. |
## Background-worker memory/task review
| Facility | Bound/ownership reviewed |
|---|---|
| Cache COLD writer | Existing object-count and byte budgets remain; delete fallback is limited to 2 detached tasks and otherwise applies bounded synchronous backpressure. |
| COLD cleanup | Manual/automatic cleanup share a single-flight semaphore. Candidate selection uses a bounded heap sized to the requested deletion limit; delete path metadata is flushed per 32-entry batch. |
| Stale-while-revalidate | Distinct background revalidations are capped at 8, limiting retained cached `Bytes` when objects are concurrently evicted/replaced. |
| Proactive refresh | Sleeping scheduled-key tasks are capped at 64..256 depending on configured refresh concurrency; actual refreshes keep the existing `proactive_refresh_max_inflight` bound. |
| Logging | History and broadcast buffers are fixed at 4096; sink delivery uses a bounded channel and `try_send`. |
| Diagnostics | One cancellation-aware sampler loop; each blocking sample is awaited before the next iteration. |
| DNS | Shared semaphore of 16 blocking resolver workers plus timeout; a timed-out OS resolver still owns its permit until the worker exits. |
| Backend health | One supervisor with a `JoinSet`, maximum 16 probes, at most one claimed probe per member, abort/drain on shutdown. |
| TCP/UDP streams | Per-stream connection/session semaphore, bounded UDP channel, `JoinSet` ownership and cancellation. |
| HTTP listeners | Accepted connections are gated by `max_connections`; connection tasks are owned by the listener `JoinSet`. App-owned tunnel/background tasks use `TaskTracker`. |
| Configuration writes | Validation/publication work is bounded by 8 update slots and serialized publication lock. |
| ACME | Issuance/renewal is single-flight through a semaphore; the scheduler is one cancellation-aware loop. |
| Docker discovery / update checker | One cancellation-aware periodic loop each; no per-tick detached task accumulation. |
| Admin live streams | Viewer permits and bounded broadcast/snapshot semantics prevent per-client unbounded queues. |
| Tunnel registry | Stores weak references and removes dead sessions on drop/list cleanup. |
## Added regression coverage
Rust tests cover preset serialization/database reopen, preset-only cache identity stability, transform memory-budget math, bounded decompression, false Content-Length aggregation, proactive pending-task limits, stale revalidation limits and single-flight COLD cleanup. `scripts/memory_contracts.py` independently verifies the source-level bounds above when Cargo is unavailable and is part of `./dev.sh test`.
## Release-host acceptance gate
Run `./dev.sh fmt`, `./dev.sh check` and `./dev.sh test`, then exercise representative transform/cache traffic while recording RSS/heap. Acceptance target: RSS reaches a plateau after warm-up; transform concurrency never exceeds the 128 MiB temporary budget estimate; COLD cleanup and revalidation counts remain bounded; no monotonically growing task/map/queue is observed after traffic stops.
docs/V0.24.7_AUDIT.md
+76
View File
@@ -0,0 +1,76 @@
# RustProxy 0.24.7 transform resource-control audit
## Scope
0.24.7 makes the transformation resource envelope configurable without changing transformation semantics. The global transform configuration now persists a UI resource preset, a process-wide temporary-memory ceiling, and a process-wide concurrent-transform ceiling. Existing configurations inherit the former 128 MiB memory behavior and a 32-attempt concurrency ceiling through serde defaults.
This workspace still has no Rust/Cargo toolchain, so compiler/type/runtime acceptance and sustained RSS profiling remain release-host gates.
## Resource controls
| Control | Default | Range | Behavior |
|---|---:|---:|---|
| `memory_budget_bytes` | 128 MiB | 1 MiB..2 GiB | Process-wide ceiling for estimated origin/decode/output buffers. It is not pre-allocated. If a new transform would exceed the current configured ceiling, the response fails open unchanged with `memory_pressure`. |
| `max_body_bytes` | 2 MiB | 1 KiB..32 MiB | Hard aggregation/decompression ceiling for one transformable response. |
| `max_inflight` | 32 | 1..1024 | Process-wide active transform-attempt ceiling. New work fails open unchanged with `concurrency_pressure` instead of waiting. |
The memory limiter uses atomic byte reservations so a live configuration change takes effect for new attempts without resizing or replacing a semaphore. Existing reservations release their exact charge on drop.
## Presets
Presets persist only administrator intent; numeric values are authoritative. They do not change enabled transform kinds or the extension allow-list.
| Preset | Memory | Max body | Max concurrent |
|---|---:|---:|---:|
| RAM saver | 32 MiB | 1 MiB | 8 |
| Recommended | 128 MiB | 2 MiB | 32 |
| Performance | 512 MiB | 8 MiB | 64 |
| Uber performance | 1 GiB | 32 MiB | 128 |
Manual edits of any of the three resource fields clear the preset label to Custom. Settings stores the selected preset in the same revisioned SQLite configuration document and restores it after reopen.
## Cache compatibility
Global transform representation/eligibility settings are now included in cache compatibility identity. Changing enabled transform kinds, extensions or `max_body_bytes` rotates the identity so cached representations cannot outlive a semantic policy change.
Resource-only fields (`preset`, `memory_budget_bytes`, `max_inflight`) are normalized out of the compatibility hash. Capacity tuning therefore does not purge HOT/COLD objects by itself.
## UI safeguards
The Transform settings page exposes all three resource controls and four presets. It warns when the configured temporary-memory budget is smaller than the worst-case estimate for one maximum-size compressed response, and when the budget exceeds 50% of detected effective host/cgroup RAM. Warnings do not block an intentional expert configuration.
## Runtime dashboard accounting
The same 0.24.7 release now exposes live resource accounting without changing any
resource limit. `GET /api/status` reports a `transform_resources` view and a
`memory_accounting` view. The Overview Process block shows process RSS, HOT cache RAM,
cache transient RAM, the transform reservation/budget, active transform slots and the
remaining non-cache RSS. The transform reservation is an intentionally conservative
limiter charge, so it is shown as an overlapping diagnostic rather than added to RSS.
`non_cache_rss_bytes` is `process RSS - accounted HOT/transient cache RAM` with saturating
subtraction. It therefore still includes transform buffers, connections, TLS, logging,
Tokio/runtime state, allocator overhead and any other untracked process allocations. This
is preferable to inventing per-subsystem heap precision that the allocator does not
provide.
The cache runtime view now also exposes `cold_limit_bytes` and `cold_limit_entries`; the
overview cache block shows current COLD disk bytes and objects against those limits. COLD
is explicitly disk occupancy and is never added to process RAM.
The dedicated Transform dashboard also shows current process-wide reserved bytes versus
`memory_budget_bytes` and active transform attempts versus `max_inflight`.
## Regression coverage
- transform settings validation and serde preset round-trip;
- local atomic memory reservation obeys a runtime byte ceiling;
- local atomic inflight reservation obeys a runtime concurrency ceiling;
- SQLite close/reopen preserves transform preset, memory budget and concurrency limit;
- cache identity ignores resource-only changes but rotates for transform representation/eligibility changes;
- `scripts/memory_contracts.py` verifies the new dynamic memory/concurrency bounds without Rust;
- source/release smoke checks verify the Transform UI, OpenAPI fields and release identity.
## Release-host acceptance gate
Run `./dev.sh fmt`, `./dev.sh check` and `./dev.sh test`, then stress each preset with identity and compressed JS/CSS/HTML/SVG/JSON bodies. Acceptance target: RSS plateaus after warm-up, active transform work never exceeds the configured byte/concurrency ceilings, `memory_pressure`/`concurrency_pressure` appear only when expected, and cache contents remain valid after changing resource-only settings.