`/convert` shared cache — RustProxy 0.29.3 and Varnish

Only fully downloaded, decoded and converted lists return a cacheable HTTP 200. All errors (400, 413, 415, 422, 429, 502, etc.) return Cache-Control: no-store. Authorized debug responses (?debug=1) are never shared or cached.

Example successful response

HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
Content-Length: 12345
ETag: "<sha256 of converted response bytes>"
Cache-Control: public, max-age=3600, s-maxage=604800, stale-while-revalidate=86400, stale-if-error=604800
X-Content-Type-Options: nosniff

max-age: cache freshness for browsers/clients (default 1 hour).
s-maxage: freshness for shared caches such as RustProxy/Varnish (default 7 days).
stale-while-revalidate: permits serving a stored stale object while the proxy refreshes it (1 additional day by default; cache support required).
stale-if-error: permits serving an already-stored stale object if refresh fails (7 additional days by default; cache support required).
ETag: SHA-256 of the converted bytes, not the upstream file ETag. Identical converted bodies have matching ETags. If-None-Match can produce an HTTP 304, but only after the application checks the complete source on a backend request. On a normal proxy HIT, no application work is performed.
The upstream source is always fetched as an unconditional GET. Client validators must not be sent to the source list because they refer to the converted resource.
GET and HEAD use the same validated body, ETag and length; HEAD does not send response bytes.

Environment variables in /var/www/adlist_mikrotik/.env:

CACHE_ENABLED=true
CACHE_MAX_AGE=3600
CACHE_S_MAXAGE=604800
CACHE_STALE_WHILE_REVALIDATE=86400
CACHE_STALE_IF_ERROR=604800

Existing .env files override config.py defaults, so update the actual deployed .env after upgrading. Restart the listapp service to apply changes.

RustProxy (important: origin headers alone cannot override BYPASS)

Enable the global cache and caching on the listapp.linuxiarz.pl vhost.
Set max_ttl_seconds=604800 globally and/or on that vhost. RustProxy 0.29.3 has a default cap of 86400 seconds, and an allowed maximum of 604800 seconds. Longer origin s-maxage values cannot override the cap.
Set bypass_query=false: /convert requires a query string (url=...&ip=...). Check bypass_prefixes to make sure /convert is not excluded.
Allow sufficient max_object_bytes for the converted result, not the source. The RustProxy default is 16 MiB; objects larger than the limit will not be stored. If necessary, increase the total HOT/transient budgets too.
Optionally enable cold_enabled=true and size COLD appropriately, so cached objects can survive RAM eviction and regular restarts. COLD is still bounded and not permanent storage.
Requests with Cookie, Authorization, Cache-Control: no-store, or various forced revalidation directives may BYPASS / revalidate. Normal unauthenticated periodic scans without these request headers are best for high cache hit rate. If you configure an exception, scope it only to public /convert requests and consider the security implications.
Enable cache diagnostics expose_status, expose_cache_status, and optionally expose_resident_age to inspect MISS/HIT/BYPASS and reason codes.

A long TTL does not guarantee physical retention: eviction, explicit purge, size limits and settings may remove entries. Source updates are not visible to normal cache users until freshness expires or the cache is purged; choose TTL accordingly.

Varnish

Varnish can use Cache-Control: public, s-maxage=... (subject to VCL policy, cookies, Authorization, cache size and eviction). It must be configured to pass/cache the /convert route appropriately. The application does not use an internal Redis response-body cache; the shared proxy stores successful responses.

Verification

Use the same full URL repeatedly, with GET (not HEAD), without cookies or request Cache-Control: no-cache. For example:

BASE='https://listapp.linuxiarz.pl/convert'
for n in 1 2 3; do
  echo "Request $n"
  curl -sS --get \
    --data-urlencode 'url=https://v.firebog.net/hosts/RPiList-Malware.txt' \
    --data-urlencode 'ip=0.0.0.0' \
    -D - -o /dev/null "$BASE" | \
    grep -Ei '^(HTTP/|cache-control:|etag:|age:|x-cache:|cache-status:|content-length:|content-type:)'
  sleep 2
done

Expect HTTP 200 and the public, s-maxage header from the application. If RustProxy caching is eligible, a typical sequence is MISS, HIT, HIT, while Age increases. BYPASS means the proxy did not even attempt a cache fill (check request and proxy configuration); MISS followed by repeated MISS can mean oversize objects, excluded responses or aborted body transfer. These labels require corresponding diagnostics to be enabled on the proxy.

Run local code checks:

python3 -m unittest discover -s tests -p 'test_*.py' -v
python3 tests/smoke_cache_routes.py

Duplicate Date and RustProxy 0.29.3

/convert deliberately does not set its own Date header after Response.make_conditional(). Werkzeug can create a Date header there; Uvicorn also supplies one, and RustProxy 0.29.3 rejects multiple Date headers with response_ttl_size_or_metadata.

Verify that exactly one Date header is returned directly by the ASGI backend (no reverse proxy in the path). All success responses should also include a valid Content-Length, ETag and public Cache-Control. For a full seven days, set RustProxy global max_ttl_seconds to 604800 AND configure the app's CACHE_S_MAXAGE=604800 in .env. The unsafe_force cache switch is not required for /convert.

5.7 KiB Raw Blame History

/convert shared cache — RustProxy 0.29.3 and Varnish