RustProxy/docs/MIGRATION.md

# 0.7.3 -> 0.8.0 deployment changes

## 0.15.1 -> 0.16.0

There is no SQLite schema migration. Existing vhosts, backends, cache files, administrator accounts and sessions are retained.

Custom locations are now interpreted only as overrides. An untouched legacy synthetic `root` prefix `/` location with no custom backend/cache/ACL/header/rewrite/local-response behavior is normalized away and replaced by the internal main-vhost fallback. A deliberately customized `/` location remains a real override. No manual edit is required for the normal legacy `root` entry.

Publishing the administration listener through a vhost remains explicit-risk functionality. For reserved session-cookie passthrough the public request must be HTTPS, the external hostname must be listed in `administration.allowed_hosts`, the backend must directly address the configured administration listener by IP, and that backend must have `accept_local_listener_risk=true`. Ordinary upstreams continue to have `rustproxy_session` removed in both directions.

Access-log JSON may contain the additional fields `http_version`, `transport`, `websocket_bootstrap`, `websocket_phase`, `websocket_error`, `h3_reset_code` and `upstream_status`. Cache/system metrics add memory-breakdown gauges; existing consumers should ignore unknown fields as usual.

## 0.15.0 -> 0.15.1

There is no schema or configuration-format migration. This patch changes only administration UI language-pack placement, Custom locations editing/help/preview and dashboard visualizations. Existing vhosts, locations, cache settings and SQLite data remain unchanged.

## 0.14.0 -> 0.15.0

There is no new SQLite schema version in 0.15.0. Renewal observations remain in the existing
schema-v3 `certificate_renewals` table. The application no longer contains a runtime JSON importer.

When `upgrade.sh` / `scripts/upgrade.py` is used while RustProxy is stopped, an obsolete
`<data-dir>/certs/renewals.json` observation sidecar is validated as a regular file and removed as
the final source-upgrade step. It is not imported and no second legacy copy is retained. If the
source-upgrade transaction fails before that final removal, the old sidecar remains untouched.

If a site replaces source manually instead of using the stopped upgrader, the old sidecar is simply
ignored. Back it up and remove it. An empty renewal-observation table is safe: the ordinary scheduler
reconstructs non-secret status/timestamps on its first 5–10 minute startup check, while certificates,
keys, ACME account material and renewal profiles remain unchanged.

## 0.13.1 -> 0.13.2

Back up `config.sqlite3` and the configured certificate files before updating. On first
0.13.2 open, RustProxy applies the additive SQLite schema-v3 migration and creates
`certificate_renewals`; existing configuration, administrator accounts, sessions, audit
records and revisions are retained. The table stores only non-secret renewal observation
state (timestamps/status/message). If it is empty and the older private
`certs/renewals.json` exists, validated observations are imported once and the sidecar is
removed only after the database write succeeds. That one-time runtime compatibility path existed in
0.13.2/0.14.0 and is removed in 0.15.0; use the 0.15.0 stopped upgrader described above instead.

The managed `update.sh` path for 0.13.2+ also marks the stop as a software update so an
enabled cold tier receives remaining hot cache objects even when ordinary
`cold_on_shutdown=false`. This is a best-effort persistent handoff bounded by the managed
stop timeout; no cache persistence is possible when cold storage is disabled.

The raw JSON editor moved to **Settings -> Manual config**. `/configuration` remains a
browser compatibility alias, so old bookmarks do not require migration.

## 0.8.0 -> 0.8.1 cache behavior

The source default for `cache.enabled` changes to `true`. Existing saved configurations
that explicitly contain `enabled: false` remain disabled; the new default applies to new
or missing cache settings and is not a silent rewrite of an administrator choice.
`cache.max_object_bytes` may now be configured up to 1 GiB when `cache.max_bytes` is at
least as large. Browser conditional requests/no-cache refreshes now use conditional HIT
or backend revalidation instead of blanket BYPASS; `no-store` remains a hard exclusion.


- Existing vhosts deserialize new traffic-policy fields with conservative/default-off
  behavior except `http2_enabled` and `http3_enabled`, which default true to preserve prior protocol behavior.
- The pre-production inline/global listener format is intentionally no longer readable.
  Recreate development listener configuration under `listeners.endpoints` and bind each
  vhost with `frontend.listener_ids`. No automatic vhost migration is performed.
- Existing backend TLS settings are preserved. Only newly created panel backend rows use
  the 0.13.0 default `tls_verify:false`; enable verification explicitly for origins that require certificate identity checks.
- First installs fetch verified GeoIP country files. Existing installations upgraded with
  `update.sh` keep their data directory; install the two verified files under `data/geoip`
  before enabling country rules if the installation predates 0.8.0.


No database schema migration is introduced. The release changes deployment defaults
and control-plane startup behavior:

- `admin setup-token` can create the one-time setup token before first server start.
- New managed systemd installations default to `0.0.0.0:80` and `0.0.0.0:443`.
  Existing installations keep the saved service unit during `update.sh`; update does
  not silently add port 443.
- Wildcard administration is opt-in only. Direct server startup requires
  `--admin-listen 0.0.0.0:9090 serve --allow-remote-admin`.
  The installer uses the equivalent `--admin` and `--allow-remote-admin` options.
  This is plain HTTP; keep it behind a firewall/private network or use a tunnel.

Back up private state before updating and run the real Rust check/test suite on the
target build host.

---

# 0.7.1 -> 0.7.2 build fix

This is a distinct source release. It removes the absent-SWAGGER-environment panic;
it does not migrate the database, alter vhosts or reset any account/certificate.
Back up private data as usual and update the complete source tree, not just build.rs.
The patch changes Cargo dependencies and adds a required build helper, so copying
only the Rust file would leave an inconsistent package.

The existing `dev.sh` Python 3.10+ requirement now also applies to direct Cargo
builds. No `.cargo/config.toml` or manual `SWAGGER_UI_DOWNLOAD_URL` export is required.
The default pinned tag is fetched on the first build; a reviewed local tag archive
remains supported for restricted build hosts. See [API_ACCESS.md](API_ACCESS.md).
No default to an older Swagger distribution is introduced.

Use the existing stopped-source `update.sh --source-tree` workflow or build a new
binary and use the managed-service updater as documented in [INSTALL.md](INSTALL.md).
`dev.sh` backs up an older Cargo.lock before regenerating it; do not fake the new
lockfile by replacing its root version text. A valid new lockfile is not included.
Rebuild/restart after successful tests; the old process cannot load embedded changes.
There is no need to clear application data or certificates, or to run `cargo clean`
just to remove this panic. Test the real admin session and `/docs` after building.

---

# 0.7.0 -> 0.7.1 access changes

No database/configuration schema migration is introduced by this patch. Back up
existing data/certificates and use the existing install/update instructions.
The reported release is source-only; compile and test before installing a binary.

Operational JS and full translation packs are now session-protected. Old tabs
should be reloaded after upgrading. Automation reading `/api/auth/status.version`
or `/healthz.version` must use authenticated `/api/status` instead. Dedicated
Prometheus scraping retains its own CIDR/token policy and remains independent of
browser sessions. No reverse-proxy vhost auth policy is relaxed by this patch.

In 0.7.1 the Swagger pin depended on `.cargo/config.toml`. That requirement is
removed in 0.7.2 (see the build notes above). Source-tree updates still back up and
replace that managed directory; retain any unrelated local Cargo settings from
your private backup. Read [API_ACCESS.md](API_ACCESS.md) for current build inputs.

Run `python3 scripts/admin_audit.py --url http://127.0.0.1:9090` after deployment.
It prints only status checks, no response bodies or credentials.

---

# Updating, backup and recovery

## 0.7.0: renewal and service deployment

See [INSTALL.md](INSTALL.md) for separate installer/updater commands and complete
code-plus-data rollback. Older source deployments retain `update.sh --source-tree`.
Back up the entire private state, including Certbot profiles, before any update.
Do not overlay a new binary onto a live, untested database without a recovery plan.

Managed 0.6 certificate issuer metadata defaults to automatic renewal in 0.7.
The scheduler starts after a 5-10 minute delay and reuses existing consented
profiles. Imported PEM without issuer metadata is not scheduled. Review auto-renew
flags, service PATH, DNS permissions and HTTP-01 reachability before starting.
The new process does not implicitly switch staging certificates to production.

Missing WebSocket flags preserve existing enabled behavior. New UI-authored vhosts
default to disabled. Existing backend members remain in the normal pool; new
dedicated location origins default to `pool_eligible:false`. Basic Auth is opt-in,
requires actual TLS by default and combines with (rather than bypasses) IP ACLs.

Normal dev/server startup no longer starts the demonstration origin. Existing
sample vhosts are not silently deleted: remove or replace them deliberately, or
explicitly enable the protocol fixture in an isolated old test installation. New
databases show a neutral informational landing page with no sample backend.

New fields are not a promise of backward parser compatibility. Downgrade by
restoring both old code and the pre-upgrade private-data snapshot. No verified Rust
build is supplied; run check/test and server acceptance before rollout.

## 0.6.0: historical vhost migration note

0.6.0 originally shipped a one-way importer from the pre-vhost development format.
That importer is removed in 0.8.0. Current builds accept only `vhosts`, named
`listeners.endpoints` and `frontend.listener_ids`. Development databases containing
top-level route/backend documents or inline frontend listeners must be recreated or
manually rewritten before starting 0.8.0. No automatic compatibility conversion runs.

The dedicated Prometheus listener remains opt-in/disabled by default. Settings ->
Prometheus controls it. A restart is required for enable/bind/token-variable changes.
Certificate provider metadata remains additive and secrets remain outside exported
configuration.

## 0.5.0 behavior changes

Back up the database and configured PEM pairs before upgrading. The managed
updater replaces the full source tree, not live data. Build and test on staging.
No verified binary or fabricated Cargo.lock is supplied.

TLS bindings with omitted `http3` now enable UDP with defaults; older versions
omitted disabled options when serializing, so review ALL TLS bindings. To retain
TCP-only operation, explicitly save `http3: null` on the named TLS listener before
restarting the new process. Shared/wildcard
IP-port bindings must agree on all protocol flags and H3 limits. A UDP bind failure
is not silently ignored. HTTP/1.1 and optional HTTP/2 remain on TCP.

`diagnostic_headers.hide_mock_origin` was removed and is no longer accepted or
exported; use explicit response-header rules if needed. X-Mock-Origin is a normal upstream field; remove it with ordinary
response-header rules when desired. Request ID remains off by default. The removed 0.17 `diagnostic_headers.revision` field is no longer part of the configuration schema. Custom Server values
are NOT automatically rewritten to the new binary version.

Cache contents are process-local. Reserved admin-cookie removal now occurs before
cache eligibility; application cookies still bypass. Domain counters reset on
configuration publication. See CACHE_DIAGNOSTICS.md for explaining BYPASS.

ACME/RFC2136 is optional and adds external Python/Certbot/plugin requirements.
No DNS update or CA request happens simply by upgrading. Issuance is explicit,
staging-first and does not activate a certificate without import. Existing
`dev.sh backup` does not include the private `certs/acme` account/TSIG profiles;
back those up separately when issuance is idle. No automatic renewal is included.
See ACME_RFC2136.md for limitations and restoration responsibilities.


## Reported auth module compilation failure

0.0.1 used `src/auth.rs`. 0.0.2 moved the implementation into `src/auth/mod.rs` and
`src/auth/store.rs`. Overlay extraction can retain both layouts and cause Rust
E0761. Missing `Store::session_at`, `admin_record`, `end_session` and
`revoke_all_sessions` messages can follow because the implementation module did not
load. The 0.0.2 archive itself did not contain both auth layouts.

0.4.0 retains an explicit `#[path = "auth/mod.rs"]` declaration, including for direct
Cargo invocations. Preflight also detects mixed source trees. It recognizes the
exact original auth file by SHA-256 and moves it to
`.upgrade-backup/legacy-auth/auth.rs.0.0.1`. Unknown custom source is left untouched
and the development script stops with instructions. Never delete custom files just
to silence a compiler error.

```bash
bash dev.sh doctor
bash dev.sh repair
```

These commands do not require Rust. Repair is deliberately limited; a safe whole
source-tree update is preferable to extracting individual files over a checkout.

## Recommended source update

Extract the new archive beside, not inside, the installed project. First back up
the existing database using the new backup script. The destination must not exist:

```bash
unzip RustProxy-0.4.0.zip -d /home/mg/rustproxy-040-staging
cd /home/mg/rustproxy-040-staging/RustProxy
python3 scripts/backup.py --data-dir /home/mg/RustProxy/data \
  /home/mg/RustProxy-backup-before-0.4.0
```

A missing database means there is no database to back up. Do not initialize an empty
replacement accidentally. For data stored elsewhere, use the actual directory.

Stop RustProxy using its terminal or service manager. Then:

```bash
bash upgrade.sh /home/mg/RustProxy --stopped
# External data directory: also pass --data-dir /actual/private/data
cd /home/mg/RustProxy
bash dev.sh test
bash dev.sh
```

The updater checks a process lock on POSIX, requires the explicit `--stopped`
acknowledgement and rejects nested source/target paths and managed symlinks.
It preserves `.env`, `data`, `.git`, `target` and unmanaged files. Source, scripts,
tests, examples and documentation are replaced as whole managed paths. Old managed
files remain in `.upgrade-backup/before-0.4.0-<identifier>`. An interrupted copy
exception triggers source rollback. Abrupt power loss is not a transactional
filesystem update; inspect the backup and source tree before restarting.

The updater does not evaluate the old `.env` or discover arbitrary external data
paths. Pass `--data-dir` when necessary, and stop every process yourself before
changing source. It does not perform a database migration or backup automatically.
Custom source edits are backed up, not merged.

## Database compatibility

RustProxy 0.13.2 advances the SQLite schema from version 2 to **version 3** with one
additive `certificate_renewals` table for non-secret ACME observation state. The existing
0.0.1-to-0.0.2 auth migration still runs when needed. Configuration-document additions
remain defaulted/non-destructive. Existing accounts, policies, revisions and configuration
remain intact. Starting with 0.15.0 the application reads renewal observations only from this table;
historical JSON observation state is handled, when present, by the stopped source-upgrade remove
described above rather than by the running service.

The generated default certificate is created on first TLS-enabled startup, in
`<data-dir>/certs`. Its key is never overwritten by startup. A partial or invalid
pair causes a clear startup error; it is not silently discarded. Demo settings
never overwrite an existing configuration, including an empty existing one.

## Backup contents and restore

`./dev.sh backup NEW_DIRECTORY` uses SQLite's online backup API, so committed WAL
content is included. The result contains a private `config.sqlite3`, `backup.json`
with hashes/revision metadata and any configured/default certificate files under
`certs/`. Unreferenced staged certificate files are not included. Do not rotate or
manually replace certificate files while taking a backup. This is a consistent
SQLite snapshot plus file copies, not one cross-filesystem transaction.

Backups contain credential hashes, session records and private keys. Keep them
private, encrypted at rest as appropriate, and outside public web directories.
They do not contain `.env`, source edits or a generated dependency lockfile; preserve
those separately. Existing backup destinations and certificate symlinks are refused.

To restore, stop the server, verify backup hashes and keep the current data directory
as a separate recovery copy. Restore into a **new private directory**, including
`config.sqlite3` and `certs/` together, then point `RUST_PROXY_DATA_DIR` to it.
Use directory mode 700 and file mode 600. Do not copy stale `-wal` or `-shm` files
from another database beside the restored snapshot. Start the matching version and
run `./dev.sh admin logout-all` or reset the password to revoke restored sessions.
Verify routes, TLS identities and administration before resuming traffic.

## Downgrade

Older versions reject unknown configuration fields such as `tls`, `protocol` or
`tls_ca_pem`. Restoring old source alone is therefore not a safe downgrade after
saving new configuration. Restore the matching pre-upgrade database and environment
from a verified backup, with the server stopped. Do not edit password hashes or
schema versions manually.


## 0.0.6 -> 0.1.0 document and source changes

No new SQL schema version is introduced. Old documents default to inherited
listeners, disabled cache/compression and preserve-Server behavior. New optional
route cache policy does not change existing route behavior. Existing account,
revision, TLS and ACL state is retained.

Managed paths now include build.rs, langs, deploy and make_zip.py. Extra *.json
language packs from the installed langs directory are copied into the new tree
when the new distribution does not contain the same filename. Shipped packs are
updated, with old bytes preserved in the source backup. Invalid custom packs can
correctly block a build; review them with scripts/langs_check.py. No symlinked
language pack is followed. The updater never copies a database from the package.

The old lockfile is preserved in the managed source backup and a new dependency
graph is generated by Cargo on first build. No 0.1.0 resolved Cargo.lock is included
because Cargo was unavailable here. Keep the actual newly resolved file afterward.

Saving any 0.1.0 document serializes its new fields. Older binaries use strict
unknown-field rejection, so rolling back only the source/binary may fail startup.
Restore the pre-upgrade database/certificates into a new private directory and keep
the newer data separately. Do not directly edit live SQLite to remove fields.

A saved invalid listener address or missing low-port permission can prevent
startup. The online config CLI cannot repair an offline server. Keep a pre-change
backup and use the documented stopped-server restore path; remove persisted
listener overrides through Settings while the service is still running before
changing deployment assumptions. The cache itself is RAM only and is not backed up.

## 0.1.0 -> 0.2.0: compression, vhosts and pool behavior

The `SizeAbove` u16/u64 mismatch is fixed by replacing that predicate with a custom
u64 implementation. Do not use `try_into().unwrap()` or truncate the setting to
work around the old source: configured thresholds above 65535 must remain valid.
This source change removes that call, but is not a successful compiler result;
run `bash dev.sh check` and `bash dev.sh test` on a Rust-enabled staging machine.

SQL schema version stays 2; accounts, ACLs, certificates and revisions are retained.
New `vhosts` defaults empty; named TLS listeners default to H2/H3 capability when those fields are omitted. Legacy routes
continue on the global listeners and are never silently migrated or exposed on
additional IPs. The editor's explicit migration action converts one unambiguous
route group in a single revision and keeps the migrated route group's existing health behavior.

New vhosts default to TCP health checks and remain unavailable until
`rise` checks succeed. Plan the correct path/status/timeouts before migration.
Health/counter/affinity observations are ephemeral. They survive compatible edits,
not restarts; new/changed/re-enabled origins require new checks. Cache and idle
upstream pools are not carried across config publication. Admitted exchanges finish
with their old snapshot and are not replayed on a different backend.

HTTPS origins previously configured `http2` now retain HTTP/1.1 ALPN compatibility.
Plaintext `http2` is still strict h2c. Per-listener IP/port/TLS/HTTP2 changes require
restart. During an address move, hot routing can deny the previous destination
before the new socket is bound. Plan a maintenance window, verify assigned IPs,
privileges/certificates and retain an offline-restorable backup.

After saving any 0.2.0 document, 0.1.0 can reject new fields even without a vhost,
because `https_http2` is serialized. Downgrade requires matching pre-0.2.0 data,
source and environment. Never remove fields by editing a live SQLite database.
The archive contains no fabricated lockfile, executable, database or private key.
Generate and retain the real dependency graph before acceptance/deployment.

## 0.2.0 -> 0.3.0

The reported `json_internal!` recursion error is addressed by splitting
`status_payload` into small JSON values and inserting feature flags through a map.
Do not overlay only this function: this release also adds the `query` Axum feature,
`src/pages.rs`, `management.js`, new configuration fields and matching UI assets.
The earlier u64 compression predicate remains in place. Compilation has not been
executed in the packaging environment; this is not a successful-build assertion.

Existing database documents deserialize with empty/default `pages`, vhost `cache`
and `certificate_id` fields. No SQL schema migration is added. Per-vhost certificate
selection rewrites the effective SNI map in a snapshot, not the files. Upload is a
staging action: Save must publish a validated configuration before traffic uses it.
Canceling the editor leaves an unbound pair, not an active certificate change.

Use the complete backup/whole-source update procedure above. The existing backup
copies configured keys and fallback files. Unbound upload staging files are not in
the configuration and therefore are not part of that configured-certificate backup.
Retain them separately only when needed. Never delete a filename still referenced
by the configuration or currently serving snapshot; stop the process before manual
cleanup and preserve a backup of the full private certificate directory.

New defaults do not activate custom pages or vhost cache overrides. An explicitly
disabled global cache can still be overridden by one vhost; the global byte/entry/object
limits still bound it. Current releases rotate cache compatibility only for configuration
changes that can alter cached/origin semantics; unrelated control-plane publication preserves
reusable objects. The dedicated per-vhost Purge action remains selective.

Downgrade requires restoring the earlier database/configuration together with its
matching source: 0.2.0 rejects the new fields. Do not merely relabel a new document.
The source archive contains no fabricated lockfile. Resolve and retain Cargo.lock
on the first compiler-enabled build and run all tests before restarting production.

## 0.3.0 -> 0.4.0

Use the full-source update above, not an overlay of a few Rust files. This revision
adds Quinn/h3 dependencies, new modules and a new embedded integrations.js asset.
No successful compiler run or dependency resolution is claimed. Generate and retain
a real Cargo.lock, then run check and all tests before acceptance.

SQL schema stays 2. Proxy sockets are persisted only in `listeners.endpoints`; a new
database is seeded from startup CLI listener addresses. Existing development documents
using the removed listener format are rejected rather than migrated. Tinyauth and
vhost Server policy continue to use their normal defaults. New diagnostic defaults remove x-request-id from client responses unless it is enabled explicitly where needed. Access logs retain request IDs. The old revision diagnostic field/header behavior was removed in 0.18.2. Forwarding headers are
rebuilt from actual peers unless known proxy CIDRs are explicitly configured.

Reference named listener IDs per vhost instead of duplicating bindings. There is no
inline/global compatibility conversion path in 0.8.0. Scope names have no firewall meaning. Changed sockets, ALPN or H3 limits
require restart; opening UDP is necessary for H3. Saving intent does not prove
that an OS bind or UDP firewall rule will succeed.

The certificate directory/fallback is prepared even with HTTP-only startup, so
library uploads can be staged before enabling TLS. The generated fallback is
self-signed development material and does not automatically open an HTTPS socket.
Import the PEM pair in Certificates, save inventory, then select its ID in SSL.
Unbound staged files are not automatically backed up; preserve private files needed
for rollback and never remove any currently referenced certificate/key.

Tinyauth-protected vhosts bypass shared cache regardless of cache toggles. Failures
deny access instead of serving cached/static content. Configure its verifier URL,
public login URL and explicit private-origin permission before enabling it on an
existing service. Test login, refusal, logout/revocation and WebSockets on staging.

The examples are complete replacement documents for disposable test databases,
not merge patches. Saving a 0.4 document serializes new fields; older binaries may
reject them. Downgrade requires matching pre-upgrade database, certificates,
environment and source, with the process stopped. The source backup alone is not
a database/configuration rollback.