mikrotik-backup/docs/data-import.md

# Fresh installation and explicit Python import

Version **0.0.1** starts with a clean installation. It does not update an old
schema or move existing backup files during startup. Importing data from the
Python/TypeScript application is a separate, explicit operation.

## Choose one installation path

For an empty installation, configure the new application's `.env` and run:

```sh
./start.sh
```

A database is initialized only when the selected database file does not exist.
The complete current schema is in `src/schema.sql`. The initial account uses
`DEFAULT_ADMIN_USERNAME` and `DEFAULT_ADMIN_PASSWORD`; if the password is omitted,
a generated value is stored in `DATA_DIR/secrets/initial-admin-password`.
Registration remains controlled by the new application's configuration.

If you intend to import Python data, **run the import before starting the new
server**. Starting the server first creates a database, and the importer will
refuse to overwrite or merge it. Use a different new destination if a database
already exists; do not delete a database merely to bypass this check.

## Startup never upgrades an existing database

Startup accepts an existing database only when it has the current `schema_info`
marker, supported format version, required tables and columns, valid integrity,
and the expected user/settings records. An older Python schema, an earlier Rust
work-in-progress schema, an unknown version, or a pre-existing empty database file
is rejected with instructions to select a fresh destination or use the explicit
Python importer. No `ALTER TABLE`, persistent pre-upgrade snapshot, migration
journal, automatic import, or backup relocation is performed.

Before creating storage locks or secrets, the server performs a read-only
preflight. SQLite inspects a private temporary copy of the database and its WAL
or rollback journal, so opening an incompatible database cannot create a source
`-shm` file. The temporary inspection copy is removed afterwards. The schema is
checked again while the server holds its storage locks. Existing current records
and file paths are left unchanged.

## Import command

Stop the old Python application, workers, and schedules. Keep its database, WAL,
rollback journal, and backup files together. Make an independent copy of the old
installation before switching services. This is an offline import, not a live
replication mechanism.

Use the helper from the new Rust project:

```sh
./import-python.sh /srv/old-mikrotik-app
```

Its equivalent application command is:

```sh
./start.sh import-python --source /srv/old-mikrotik-app
```

Optional source overrides are available when paths are customized or discovery
finds more than one candidate:

```sh
./import-python.sh /srv/old-mikrotik-app \
  --database docker-data/db/routeros_backup_next.db \
  --backup-root docker-data
```

Both `--database` and `--backup-root` are resolved **relative to `--source`**, unless
an absolute path is supplied. They select the old source, not the destination.
The helper resolves its first directory argument relative to the caller's working
directory before invoking the project's starter.

The destination comes from the new application's configuration. An explicit
example is:

```sh
DATA_DIR="$PWD/storage-imported" \
DATABASE_PATH="$PWD/storage-imported/routeros_backup_next.db" \
./import-python.sh /srv/old-mikrotik-app \
  --database docker-data/db/routeros_backup_next.db \
  --backup-root docker-data
```

After a successful import, use the same destination to start the server:

```sh
DATA_DIR="$PWD/storage-imported" \
DATABASE_PATH="$PWD/storage-imported/routeros_backup_next.db" \
./start.sh serve
```

The `migrate` command from previous revisions has been removed. The importer does
not start HTTP or background schedules. It does not add a new administrator when
users have been imported. Log in with an imported account and its existing password.

## Source discovery

Discovery follows the layouts present in the supplied Python/TypeScript project:

| Source layout | Recognized database location |
| --- | --- |
| Shipped Compose bind mount `./docker-data:/app/storage` | `docker-data/db/routeros_backup_next.db` or `docker-data/routeros_backup_next.db` |
| Local application working directory | `storage/routeros_backup_next.db` |
| Local backend working directory | `backend/storage/routeros_backup_next.db` |
| Explicit source configuration | SQLite `DATABASE_URL` and `DATA_DIR` in `.env` or `backend/.env` |

The known data directories are also checked for the documented `db/` database
subdirectory. Container paths under `/app/storage` are mapped to the shipped
`docker-data` mount. A customized Compose volume mapping is not executed or
interpreted as arbitrary code; supply explicit paths when it differs.

Only actual existing candidates are considered. **Multiple different database
files require `--database`**; discovery never chooses the first, newest, or largest
file. A configured database or data directory that is unavailable produces an
error instead of silently substituting another installation. Source overrides
allow the operator to resolve that situation explicitly.

The importer reads only `DATABASE_URL` and `DATA_DIR` from source environment
files. It does not execute shell expressions, expand variables, import the Python
application, or copy the source `.env`. Dynamic expressions and unsupported SQLite
URL options require explicit source paths. Providing both overrides bypasses
source environment-path discovery.

For each backup record, the importer resolves the recorded path or its preserved
filename within the selected or recognized source data roots. It supports the
original flat layout and older device-name subdirectories. Missing files,
multiple distinct matches, parent traversal, and symlinks escaping a selected
root are errors. Choose an unambiguous `--backup-root` when necessary; no source
file is silently skipped.

## What is preserved

The import creates a new database using the current schema and copies supported
records logically from a private source snapshot. It preserves:

- User, device, backup, settings, and operation-log IDs; device ownership.
- Existing password hashes, usernames, preferences, timestamps, and backup metadata.
- RouterOS/SwitchOS connection information, per-device credentials, and backup policies.
- Global settings, schedules, SSH keys, SwitchOS credentials, SMTP settings/secrets,
  and Pushover settings/secrets stored in SQLite.
- Every referenced backup's bytes, original `file_name`, and existing checksum.

Fields absent in an older supported Python schema receive current defaults in the
new database only. A missing old `file_name` is derived from its recorded path.
Missing/null historical timestamps use the current schema's default when that
older schema did not store a usable value. A missing password hash, orphaned
ownership, or unsupported source structure causes an error; import never assigns
an arbitrary owner or replaces a missing password with an administrator password.
An empty source settings table receives the normal default settings row.

Imported SwitchOS records have `use_global_switchos_credentials = null` when the
Python schema has no explicit source selector. This preserves the former per-field
inheritance. Choosing global or device credentials later uses a complete pair
from that source without deleting saved device credentials. Secrets are not
returned in the public API or prefilled into the interface.

Passlib PBKDF2-SHA256 and supported Werkzeug PBKDF2 hashes remain usable without a
reset. New and changed passwords use Argon2id. The source environment's JWT secret
is not copied. Unless `SECRET_KEY` is explicitly configured for the new server,
its first startup creates a new `DATA_DIR/secrets/jwt.key`, and old sessions must
log in again. The import itself does not generate a JWT key.

## Destination layout and publication

The destination must be separate from the source application and source backup
roots. Its database must not exist, and its `backups/` tree must contain no existing
files. Existing empty directories, the new configuration's private files, and
storage locks do not authorize overwriting a database or backup. Relative paths,
parent components, and existing symlinks are resolved before any directory or lock
is created, using the same normalized paths for validation and subsequent writes.

Imported and newly created files use stable numeric device IDs:

| Backup type | Directory relative to `DATA_DIR` |
| --- | --- |
| RouterOS or SwitchOS binary | `backups/binary/<device-id>/` |
| RouterOS text export | `backups/exports/<device-id>/` |

An imported physical filename receives the backup ID as a prefix. Its metadata
`file_name` remains unchanged for downloads and RouterOS uploads. Names are bounded
and published without replacing existing files or symlinks. Renaming a device does
not rename its numeric directory. Database paths to these files are relative to
`DATA_DIR`.

The importer reads the stopped source database and any `-wal` or `-journal` as
ordinary files, copying them into private staging. It compares metadata and SHA-256
before and after copying and refuses a detected change. SQLite opens only this
private copy and includes committed WAL records. The source `-shm` is neither opened
by SQLite nor recreated; staging reconstructs its own coordination files when
needed. Keep the source stopped throughout the operation, because these checks
are not a substitute for coordinating concurrent writers.

All source rows and backup files are validated before the target database is
published. Every copied file is checked against its recorded SHA-256 when present
and against the copy's calculated SHA-256. Publication uses private temporary
files, file synchronization, atomic no-clobber publication, and directory
synchronization. The destination database is published last.

Before database publication, error cleanup makes a best-effort attempt to remove
files recorded as installed by that attempt; source files remain unchanged. If
directory synchronization fails after database publication, the database and its
referenced backups are retained, and the error asks the operator to inspect the
destination. An interrupted process or filesystem I/O failure can leave staging
or unreferenced files: inspect or choose
another new destination instead of expecting automatic startup recovery. There is
no migration journal or automatic resumption framework.

The CLI prints a JSON summary with the selected source database and counts of
users, devices, backup records, copied files, and logs. It never prints saved
passwords or keys. Check that report, then start the new service, inspect device
lists, preview an export, and download representative binary and text files.
Do not run both installations' schedules for the same devices during cutover.

## Rollback and operational limits

The original installation remains the rollback source. To return to it, stop the
new service and restart the old application on its unchanged data. New backups
created after cutover exist only in the new installation; preserve them and its
database before discarding or reverting anything. Do not exchange a SQLite main
file under a running process or mix unrelated main/WAL/journal files.

The importer does not copy operating-system permissions, service definitions,
external environment secrets, or SSH host-trust files. Review the new environment
configuration and host-key policy before first device contact. Imported SQLite
secrets retain their original representation; target database/secret files use
private permissions, and deployment storage must protect their contents.

Regression tests cover fresh initialization/restart, rejection without source
changes, explicit path selection and ambiguity, record/secret preservation, new
file layout, checksums, locks, symlinks, committed WAL data, and source-sidecar
immutability. Tests use generated fixtures; no production database or physical
MikroTik was supplied. Current verification results are recorded separately in
[validation.md](validation.md).

## Import with Docker Compose only

If the host does not have a Rust toolchain, run the importer from the same image as
the server. Stop the old application and make sure the new application's target
volume does not already contain a database. Mount the legacy project read-only:

```sh
docker compose build app
docker compose run --rm \
  -v /srv/old-mikrotik-app:/legacy:ro \
  app import-python \
  --source /legacy \
  --database docker-data/db/routeros_backup_next.db \
  --backup-root docker-data
```

After the importer prints a successful JSON report, start the service:

```sh
docker compose up -d
```

If the legacy directory uses a different layout, change only the source paths passed
to `--database` and `--backup-root`. The target data remains in the new `/data`
volume.