first commit

docs/data-import.md

@@ -0,0 +1,262 @@
+# Fresh installation and explicit Python import
+
+Version **0.0.1** starts with a clean installation. It does not update an old
+schema or move existing backup files during startup. Importing data from the
+Python/TypeScript application is a separate, explicit operation.
+
+## Choose one installation path
+
+For an empty installation, configure the new application's `.env` and run:
+
+```sh
+./start.sh
+```
+
+A database is initialized only when the selected database file does not exist.
+The complete current schema is in `src/schema.sql`. The initial account uses
+`DEFAULT_ADMIN_USERNAME` and `DEFAULT_ADMIN_PASSWORD`; if the password is omitted,
+a generated value is stored in `DATA_DIR/secrets/initial-admin-password`.
+Registration remains controlled by the new application's configuration.
+
+If you intend to import Python data, **run the import before starting the new
+server**. Starting the server first creates a database, and the importer will
+refuse to overwrite or merge it. Use a different new destination if a database
+already exists; do not delete a database merely to bypass this check.
+
+## Startup never upgrades an existing database
+
+Startup accepts an existing database only when it has the current `schema_info`
+marker, supported format version, required tables and columns, valid integrity,
+and the expected user/settings records. An older Python schema, an earlier Rust
+work-in-progress schema, an unknown version, or a pre-existing empty database file
+is rejected with instructions to select a fresh destination or use the explicit
+Python importer. No `ALTER TABLE`, persistent pre-upgrade snapshot, migration
+journal, automatic import, or backup relocation is performed.
+
+Before creating storage locks or secrets, the server performs a read-only
+preflight. SQLite inspects a private temporary copy of the database and its WAL
+or rollback journal, so opening an incompatible database cannot create a source
+`-shm` file. The temporary inspection copy is removed afterwards. The schema is
+checked again while the server holds its storage locks. Existing current records
+and file paths are left unchanged.
+
+## Import command
+
+Stop the old Python application, workers, and schedules. Keep its database, WAL,
+rollback journal, and backup files together. Make an independent copy of the old
+installation before switching services. This is an offline import, not a live
+replication mechanism.
+
+Use the helper from the new Rust project:
+
+```sh
+./import-python.sh /srv/old-mikrotik-app
+```
+
+Its equivalent application command is:
+
+```sh
+./start.sh import-python --source /srv/old-mikrotik-app
+```
+
+Optional source overrides are available when paths are customized or discovery
+finds more than one candidate:
+
+```sh
+./import-python.sh /srv/old-mikrotik-app \
+  --database docker-data/db/routeros_backup_next.db \
+  --backup-root docker-data
+```
+
+Both `--database` and `--backup-root` are resolved **relative to `--source`**, unless
+an absolute path is supplied. They select the old source, not the destination.
+The helper resolves its first directory argument relative to the caller's working
+directory before invoking the project's starter.
+
+The destination comes from the new application's configuration. An explicit
+example is:
+
+```sh
+DATA_DIR="$PWD/storage-imported" \
+DATABASE_PATH="$PWD/storage-imported/routeros_backup_next.db" \
+./import-python.sh /srv/old-mikrotik-app \
+  --database docker-data/db/routeros_backup_next.db \
+  --backup-root docker-data
+```
+
+After a successful import, use the same destination to start the server:
+
+```sh
+DATA_DIR="$PWD/storage-imported" \
+DATABASE_PATH="$PWD/storage-imported/routeros_backup_next.db" \
+./start.sh serve
+```
+
+The `migrate` command from previous revisions has been removed. The importer does
+not start HTTP or background schedules. It does not add a new administrator when
+users have been imported. Log in with an imported account and its existing password.
+
+## Source discovery
+
+Discovery follows the layouts present in the supplied Python/TypeScript project:
+
+| Source layout | Recognized database location |
+| --- | --- |
+| Shipped Compose bind mount `./docker-data:/app/storage` | `docker-data/db/routeros_backup_next.db` or `docker-data/routeros_backup_next.db` |
+| Local application working directory | `storage/routeros_backup_next.db` |
+| Local backend working directory | `backend/storage/routeros_backup_next.db` |
+| Explicit source configuration | SQLite `DATABASE_URL` and `DATA_DIR` in `.env` or `backend/.env` |
+
+The known data directories are also checked for the documented `db/` database
+subdirectory. Container paths under `/app/storage` are mapped to the shipped
+`docker-data` mount. A customized Compose volume mapping is not executed or
+interpreted as arbitrary code; supply explicit paths when it differs.
+
+Only actual existing candidates are considered. **Multiple different database
+files require `--database`**; discovery never chooses the first, newest, or largest
+file. A configured database or data directory that is unavailable produces an
+error instead of silently substituting another installation. Source overrides
+allow the operator to resolve that situation explicitly.
+
+The importer reads only `DATABASE_URL` and `DATA_DIR` from source environment
+files. It does not execute shell expressions, expand variables, import the Python
+application, or copy the source `.env`. Dynamic expressions and unsupported SQLite
+URL options require explicit source paths. Providing both overrides bypasses
+source environment-path discovery.
+
+For each backup record, the importer resolves the recorded path or its preserved
+filename within the selected or recognized source data roots. It supports the
+original flat layout and older device-name subdirectories. Missing files,
+multiple distinct matches, parent traversal, and symlinks escaping a selected
+root are errors. Choose an unambiguous `--backup-root` when necessary; no source
+file is silently skipped.
+
+## What is preserved
+
+The import creates a new database using the current schema and copies supported
+records logically from a private source snapshot. It preserves:
+
+- User, device, backup, settings, and operation-log IDs; device ownership.
+- Existing password hashes, usernames, preferences, timestamps, and backup metadata.
+- RouterOS/SwitchOS connection information, per-device credentials, and backup policies.
+- Global settings, schedules, SSH keys, SwitchOS credentials, SMTP settings/secrets,
+  and Pushover settings/secrets stored in SQLite.
+- Every referenced backup's bytes, original `file_name`, and existing checksum.
+
+Fields absent in an older supported Python schema receive current defaults in the
+new database only. A missing old `file_name` is derived from its recorded path.
+Missing/null historical timestamps use the current schema's default when that
+older schema did not store a usable value. A missing password hash, orphaned
+ownership, or unsupported source structure causes an error; import never assigns
+an arbitrary owner or replaces a missing password with an administrator password.
+An empty source settings table receives the normal default settings row.
+
+Imported SwitchOS records have `use_global_switchos_credentials = null` when the
+Python schema has no explicit source selector. This preserves the former per-field
+inheritance. Choosing global or device credentials later uses a complete pair
+from that source without deleting saved device credentials. Secrets are not
+returned in the public API or prefilled into the interface.
+
+Passlib PBKDF2-SHA256 and supported Werkzeug PBKDF2 hashes remain usable without a
+reset. New and changed passwords use Argon2id. The source environment's JWT secret
+is not copied. Unless `SECRET_KEY` is explicitly configured for the new server,
+its first startup creates a new `DATA_DIR/secrets/jwt.key`, and old sessions must
+log in again. The import itself does not generate a JWT key.
+
+## Destination layout and publication
+
+The destination must be separate from the source application and source backup
+roots. Its database must not exist, and its `backups/` tree must contain no existing
+files. Existing empty directories, the new configuration's private files, and
+storage locks do not authorize overwriting a database or backup. Relative paths,
+parent components, and existing symlinks are resolved before any directory or lock
+is created, using the same normalized paths for validation and subsequent writes.
+
+Imported and newly created files use stable numeric device IDs:
+
+| Backup type | Directory relative to `DATA_DIR` |
+| --- | --- |
+| RouterOS or SwitchOS binary | `backups/binary/<device-id>/` |
+| RouterOS text export | `backups/exports/<device-id>/` |
+
+An imported physical filename receives the backup ID as a prefix. Its metadata
+`file_name` remains unchanged for downloads and RouterOS uploads. Names are bounded
+and published without replacing existing files or symlinks. Renaming a device does
+not rename its numeric directory. Database paths to these files are relative to
+`DATA_DIR`.
+
+The importer reads the stopped source database and any `-wal` or `-journal` as
+ordinary files, copying them into private staging. It compares metadata and SHA-256
+before and after copying and refuses a detected change. SQLite opens only this
+private copy and includes committed WAL records. The source `-shm` is neither opened
+by SQLite nor recreated; staging reconstructs its own coordination files when
+needed. Keep the source stopped throughout the operation, because these checks
+are not a substitute for coordinating concurrent writers.
+
+All source rows and backup files are validated before the target database is
+published. Every copied file is checked against its recorded SHA-256 when present
+and against the copy's calculated SHA-256. Publication uses private temporary
+files, file synchronization, atomic no-clobber publication, and directory
+synchronization. The destination database is published last.
+
+Before database publication, error cleanup makes a best-effort attempt to remove
+files recorded as installed by that attempt; source files remain unchanged. If
+directory synchronization fails after database publication, the database and its
+referenced backups are retained, and the error asks the operator to inspect the
+destination. An interrupted process or filesystem I/O failure can leave staging
+or unreferenced files: inspect or choose
+another new destination instead of expecting automatic startup recovery. There is
+no migration journal or automatic resumption framework.
+
+The CLI prints a JSON summary with the selected source database and counts of
+users, devices, backup records, copied files, and logs. It never prints saved
+passwords or keys. Check that report, then start the new service, inspect device
+lists, preview an export, and download representative binary and text files.
+Do not run both installations' schedules for the same devices during cutover.
+
+## Rollback and operational limits
+
+The original installation remains the rollback source. To return to it, stop the
+new service and restart the old application on its unchanged data. New backups
+created after cutover exist only in the new installation; preserve them and its
+database before discarding or reverting anything. Do not exchange a SQLite main
+file under a running process or mix unrelated main/WAL/journal files.
+
+The importer does not copy operating-system permissions, service definitions,
+external environment secrets, or SSH host-trust files. Review the new environment
+configuration and host-key policy before first device contact. Imported SQLite
+secrets retain their original representation; target database/secret files use
+private permissions, and deployment storage must protect their contents.
+
+Regression tests cover fresh initialization/restart, rejection without source
+changes, explicit path selection and ambiguity, record/secret preservation, new
+file layout, checksums, locks, symlinks, committed WAL data, and source-sidecar
+immutability. Tests use generated fixtures; no production database or physical
+MikroTik was supplied. Current verification results are recorded separately in
+[validation.md](validation.md).
+
+## Import with Docker Compose only
+
+If the host does not have a Rust toolchain, run the importer from the same image as
+the server. Stop the old application and make sure the new application's target
+volume does not already contain a database. Mount the legacy project read-only:
+
+```sh
+docker compose build app
+docker compose run --rm \
+  -v /srv/old-mikrotik-app:/legacy:ro \
+  app import-python \
+  --source /legacy \
+  --database docker-data/db/routeros_backup_next.db \
+  --backup-root docker-data
+```
+
+After the importer prints a successful JSON report, start the service:
+
+```sh
+docker compose up -d
+```
+
+If the legacy directory uses a different layout, change only the source paths passed
+to `--database` and `--backup-root`. The target data remains in the new `/data`
+volume.