redesign

CACHE_SETUP.md

@@ -0,0 +1,78 @@
+# `/convert` shared cache — RustProxy 0.29.3 and Varnish
+
+Only fully downloaded, decoded and converted lists return a cacheable HTTP 200.
+All errors (400, 413, 415, 422, 429, 502, etc.) return `Cache-Control: no-store`.
+Authorized debug responses (`?debug=1`) are never shared or cached.
+
+## Example successful response
+
+```http
+HTTP/1.1 200 OK
+Content-Type: text/plain; charset=utf-8
+Content-Length: 12345
+ETag: "<sha256 of converted response bytes>"
+Cache-Control: public, max-age=3600, s-maxage=604800, stale-while-revalidate=86400, stale-if-error=604800
+X-Content-Type-Options: nosniff
+```
+
+- `max-age`: cache freshness for browsers/clients (default 1 hour).
+- `s-maxage`: freshness for shared caches such as RustProxy/Varnish (default 7 days).
+- `stale-while-revalidate`: permits serving a stored stale object while the proxy refreshes it (1 additional day by default; cache support required).
+- `stale-if-error`: permits serving an already-stored stale object if refresh fails (7 additional days by default; cache support required).
+- `ETag`: SHA-256 of the **converted bytes**, not the upstream file ETag. Identical converted bodies have matching ETags. `If-None-Match` can produce an HTTP 304, but only after the application checks the complete source on a backend request. On a normal proxy HIT, no application work is performed.
+- The upstream source is always fetched as an unconditional GET. Client validators must not be sent to the **source list** because they refer to the **converted** resource.
+- GET and HEAD use the same validated body, ETag and length; HEAD does not send response bytes.
+
+Environment variables in `/var/www/adlist_mikrotik/.env`:
+
+```dotenv
+CACHE_ENABLED=true
+CACHE_MAX_AGE=3600
+CACHE_S_MAXAGE=604800
+CACHE_STALE_WHILE_REVALIDATE=86400
+CACHE_STALE_IF_ERROR=604800
+```
+
+Existing `.env` files override `config.py` defaults, so update the actual deployed `.env` after upgrading. Restart the `listapp` service to apply changes.
+
+## RustProxy (important: origin headers alone cannot override BYPASS)
+
+1. Enable the global cache and caching on the `listapp.linuxiarz.pl` vhost.
+2. Set `max_ttl_seconds=604800` globally and/or on that vhost. RustProxy 0.29.3 has a **default cap of 86400 seconds**, and an allowed maximum of **604800 seconds**. Longer origin `s-maxage` values cannot override the cap.
+3. Set `bypass_query=false`: `/convert` **requires** a query string (`url=...&ip=...`). Check `bypass_prefixes` to make sure `/convert` is not excluded.
+4. Allow sufficient `max_object_bytes` for the **converted result**, not the source. The RustProxy default is 16 MiB; objects larger than the limit will not be stored. If necessary, increase the total HOT/transient budgets too.
+5. Optionally enable `cold_enabled=true` and size COLD appropriately, so cached objects can survive RAM eviction and regular restarts. COLD is still bounded and not permanent storage.
+6. Requests with `Cookie`, `Authorization`, `Cache-Control: no-store`, or various forced revalidation directives may BYPASS / revalidate. Normal unauthenticated periodic scans without these request headers are best for high cache hit rate. If you configure an exception, scope it only to public `/convert` requests and consider the security implications.
+7. Enable cache diagnostics `expose_status`, `expose_cache_status`, and optionally `expose_resident_age` to inspect MISS/HIT/BYPASS and reason codes.
+
+A long TTL does not guarantee physical retention: eviction, explicit purge, size limits and settings may remove entries. Source updates are not visible to normal cache users until freshness expires or the cache is purged; choose TTL accordingly.
+
+## Varnish
+
+Varnish can use `Cache-Control: public, s-maxage=...` (subject to VCL policy, cookies, Authorization, cache size and eviction). It must be configured to pass/cache the `/convert` route appropriately. The application does not use an internal Redis response-body cache; the shared proxy stores successful responses.
+
+## Verification
+
+Use the *same full URL* repeatedly, with GET (not HEAD), without cookies or request `Cache-Control: no-cache`. For example:
+
+```bash
+BASE='https://listapp.linuxiarz.pl/convert'
+for n in 1 2 3; do
+  echo "Request $n"
+  curl -sS --get \
+    --data-urlencode 'url=https://v.firebog.net/hosts/RPiList-Malware.txt' \
+    --data-urlencode 'ip=0.0.0.0' \
+    -D - -o /dev/null "$BASE" | \
+    grep -Ei '^(HTTP/|cache-control:|etag:|age:|x-cache:|cache-status:|content-length:|content-type:)'
+  sleep 2
+done
+```
+
+Expect `HTTP 200` and the `public, s-maxage` header from the application. If RustProxy caching is eligible, a typical sequence is `MISS`, `HIT`, `HIT`, while `Age` increases. `BYPASS` means the proxy did not even attempt a cache fill (check request and proxy configuration); `MISS` followed by repeated `MISS` can mean oversize objects, excluded responses or aborted body transfer. These labels require corresponding diagnostics to be enabled on the proxy.
+
+Run local code checks:
+
+```bash
+python3 -m unittest discover -s tests -p 'test_*.py' -v
+python3 tests/smoke_cache_routes.py
+```