mikrotik-backup/docs/data-import.md

Fresh installation and explicit Python import

Version 0.0.1 starts with a clean installation. It does not update an old schema or move existing backup files during startup. Importing data from the Python/TypeScript application is a separate, explicit operation.

Choose one installation path

For an empty installation, configure the new application's .env and run:

./start.sh

A database is initialized only when the selected database file does not exist. The complete current schema is in src/schema.sql. The initial account uses DEFAULT_ADMIN_USERNAME and DEFAULT_ADMIN_PASSWORD; if the password is omitted, a generated value is stored in DATA_DIR/secrets/initial-admin-password. Registration remains controlled by the new application's configuration.

If you intend to import Python data, run the import before starting the new server. Starting the server first creates a database, and the importer will refuse to overwrite or merge it. Use a different new destination if a database already exists; do not delete a database merely to bypass this check.

Startup never upgrades an existing database

Startup accepts an existing database only when it has the current schema_info marker, supported format version, required tables and columns, valid integrity, and the expected user/settings records. An older Python schema, an earlier Rust work-in-progress schema, an unknown version, or a pre-existing empty database file is rejected with instructions to select a fresh destination or use the explicit Python importer. No ALTER TABLE, persistent pre-upgrade snapshot, migration journal, automatic import, or backup relocation is performed.

Before creating storage locks or secrets, the server performs a read-only preflight. SQLite inspects a private temporary copy of the database and its WAL or rollback journal, so opening an incompatible database cannot create a source -shm file. The temporary inspection copy is removed afterwards. The schema is checked again while the server holds its storage locks. Existing current records and file paths are left unchanged.

Import command

Stop the old Python application, workers, and schedules. Keep its database, WAL, rollback journal, and backup files together. Make an independent copy of the old installation before switching services. This is an offline import, not a live replication mechanism.

Use the helper from the new Rust project:

./import-python.sh /srv/old-mikrotik-app

Its equivalent application command is:

./start.sh import-python --source /srv/old-mikrotik-app

Optional source overrides are available when paths are customized or discovery finds more than one candidate:

./import-python.sh /srv/old-mikrotik-app \
  --database docker-data/db/routeros_backup_next.db \
  --backup-root docker-data

Both --database and --backup-root are resolved relative to --source, unless an absolute path is supplied. They select the old source, not the destination. The helper resolves its first directory argument relative to the caller's working directory before invoking the project's starter.

The destination comes from the new application's configuration. An explicit example is:

DATA_DIR="$PWD/storage-imported" \
DATABASE_PATH="$PWD/storage-imported/routeros_backup_next.db" \
./import-python.sh /srv/old-mikrotik-app \
  --database docker-data/db/routeros_backup_next.db \
  --backup-root docker-data

After a successful import, use the same destination to start the server:

DATA_DIR="$PWD/storage-imported" \
DATABASE_PATH="$PWD/storage-imported/routeros_backup_next.db" \
./start.sh serve

The migrate command from previous revisions has been removed. The importer does not start HTTP or background schedules. It does not add a new administrator when users have been imported. Log in with an imported account and its existing password.

Source discovery

Discovery follows the layouts present in the supplied Python/TypeScript project:

Source layout	Recognized database location
Shipped Compose bind mount ./docker-data:/app/storage	docker-data/db/routeros_backup_next.db or docker-data/routeros_backup_next.db
Local application working directory	storage/routeros_backup_next.db
Local backend working directory	backend/storage/routeros_backup_next.db
Explicit source configuration	SQLite DATABASE_URL and DATA_DIR in .env or backend/.env

The known data directories are also checked for the documented db/ database subdirectory. Container paths under /app/storage are mapped to the shipped docker-data mount. A customized Compose volume mapping is not executed or interpreted as arbitrary code; supply explicit paths when it differs.

Only actual existing candidates are considered. Multiple different database files require --database; discovery never chooses the first, newest, or largest file. A configured database or data directory that is unavailable produces an error instead of silently substituting another installation. Source overrides allow the operator to resolve that situation explicitly.

The importer reads only DATABASE_URL and DATA_DIR from source environment files. It does not execute shell expressions, expand variables, import the Python application, or copy the source .env. Dynamic expressions and unsupported SQLite URL options require explicit source paths. Providing both overrides bypasses source environment-path discovery.

For each backup record, the importer resolves the recorded path or its preserved filename within the selected or recognized source data roots. It supports the original flat layout and older device-name subdirectories. Missing files, multiple distinct matches, parent traversal, and symlinks escaping a selected root are errors. Choose an unambiguous --backup-root when necessary; no source file is silently skipped.

What is preserved

The import creates a new database using the current schema and copies supported records logically from a private source snapshot. It preserves:

• User, device, backup, settings, and operation-log IDs; device ownership.
• Existing password hashes, usernames, preferences, timestamps, and backup metadata.
• RouterOS/SwitchOS connection information, per-device credentials, and backup policies.
• Global settings, schedules, SSH keys, SwitchOS credentials, SMTP settings/secrets, and Pushover settings/secrets stored in SQLite.
• Every referenced backup's bytes, original file_name, and existing checksum.

Fields absent in an older supported Python schema receive current defaults in the new database only. A missing old file_name is derived from its recorded path. Missing/null historical timestamps use the current schema's default when that older schema did not store a usable value. A missing password hash, orphaned ownership, or unsupported source structure causes an error; import never assigns an arbitrary owner or replaces a missing password with an administrator password. An empty source settings table receives the normal default settings row.

Imported SwitchOS records have use_global_switchos_credentials = null when the Python schema has no explicit source selector. This preserves the former per-field inheritance. Choosing global or device credentials later uses a complete pair from that source without deleting saved device credentials. Secrets are not returned in the public API or prefilled into the interface.

Passlib PBKDF2-SHA256 and supported Werkzeug PBKDF2 hashes remain usable without a reset. New and changed passwords use Argon2id. The source environment's JWT secret is not copied. Unless SECRET_KEY is explicitly configured for the new server, its first startup creates a new DATA_DIR/secrets/jwt.key, and old sessions must log in again. The import itself does not generate a JWT key.

Destination layout and publication

The destination must be separate from the source application and source backup roots. Its database must not exist, and its backups/ tree must contain no existing files. Existing empty directories, the new configuration's private files, and storage locks do not authorize overwriting a database or backup. Relative paths, parent components, and existing symlinks are resolved before any directory or lock is created, using the same normalized paths for validation and subsequent writes.

Imported and newly created files use stable numeric device IDs:

Backup type	Directory relative to DATA_DIR
RouterOS or SwitchOS binary	backups/binary/<device-id>/
RouterOS text export	backups/exports/<device-id>/

An imported physical filename receives the backup ID as a prefix. Its metadata file_name remains unchanged for downloads and RouterOS uploads. Names are bounded and published without replacing existing files or symlinks. Renaming a device does not rename its numeric directory. Database paths to these files are relative to DATA_DIR.

The importer reads the stopped source database and any -wal or -journal as ordinary files, copying them into private staging. It compares metadata and SHA-256 before and after copying and refuses a detected change. SQLite opens only this private copy and includes committed WAL records. The source -shm is neither opened by SQLite nor recreated; staging reconstructs its own coordination files when needed. Keep the source stopped throughout the operation, because these checks are not a substitute for coordinating concurrent writers.

All source rows and backup files are validated before the target database is published. Every copied file is checked against its recorded SHA-256 when present and against the copy's calculated SHA-256. Publication uses private temporary files, file synchronization, atomic no-clobber publication, and directory synchronization. The destination database is published last.

Before database publication, error cleanup makes a best-effort attempt to remove files recorded as installed by that attempt; source files remain unchanged. If directory synchronization fails after database publication, the database and its referenced backups are retained, and the error asks the operator to inspect the destination. An interrupted process or filesystem I/O failure can leave staging or unreferenced files: inspect or choose another new destination instead of expecting automatic startup recovery. There is no migration journal or automatic resumption framework.

The CLI prints a JSON summary with the selected source database and counts of users, devices, backup records, copied files, and logs. It never prints saved passwords or keys. Check that report, then start the new service, inspect device lists, preview an export, and download representative binary and text files. Do not run both installations' schedules for the same devices during cutover.

Rollback and operational limits

The original installation remains the rollback source. To return to it, stop the new service and restart the old application on its unchanged data. New backups created after cutover exist only in the new installation; preserve them and its database before discarding or reverting anything. Do not exchange a SQLite main file under a running process or mix unrelated main/WAL/journal files.

The importer does not copy operating-system permissions, service definitions, external environment secrets, or SSH host-trust files. Review the new environment configuration and host-key policy before first device contact. Imported SQLite secrets retain their original representation; target database/secret files use private permissions, and deployment storage must protect their contents.

Regression tests cover fresh initialization/restart, rejection without source changes, explicit path selection and ambiguity, record/secret preservation, new file layout, checksums, locks, symlinks, committed WAL data, and source-sidecar immutability. Tests use generated fixtures; no production database or physical MikroTik was supplied. Current verification results are recorded separately in validation.md.

Import with Docker Compose only

If the host does not have a Rust toolchain, run the importer from the same image as the server. Stop the old application and make sure the new application's target volume does not already contain a database. Mount the legacy project read-only:

docker compose build app
docker compose run --rm \
  -v /srv/old-mikrotik-app:/legacy:ro \
  app import-python \
  --source /legacy \
  --database docker-data/db/routeros_backup_next.db \
  --backup-root docker-data

After the importer prints a successful JSON report, start the service:

docker compose up -d

If the legacy directory uses a different layout, change only the source paths passed to --database and --backup-root. The target data remains in the new /data volume.