263 lines
13 KiB
Markdown
263 lines
13 KiB
Markdown
# Fresh installation and explicit Python import
|
|
|
|
Version **0.0.1** starts with a clean installation. It does not update an old
|
|
schema or move existing backup files during startup. Importing data from the
|
|
Python/TypeScript application is a separate, explicit operation.
|
|
|
|
## Choose one installation path
|
|
|
|
For an empty installation, configure the new application's `.env` and run:
|
|
|
|
```sh
|
|
./start.sh
|
|
```
|
|
|
|
A database is initialized only when the selected database file does not exist.
|
|
The complete current schema is in `src/schema.sql`. The initial account uses
|
|
`DEFAULT_ADMIN_USERNAME` and `DEFAULT_ADMIN_PASSWORD`; if the password is omitted,
|
|
a generated value is stored in `DATA_DIR/secrets/initial-admin-password`.
|
|
Registration remains controlled by the new application's configuration.
|
|
|
|
If you intend to import Python data, **run the import before starting the new
|
|
server**. Starting the server first creates a database, and the importer will
|
|
refuse to overwrite or merge it. Use a different new destination if a database
|
|
already exists; do not delete a database merely to bypass this check.
|
|
|
|
## Startup never upgrades an existing database
|
|
|
|
Startup accepts an existing database only when it has the current `schema_info`
|
|
marker, supported format version, required tables and columns, valid integrity,
|
|
and the expected user/settings records. An older Python schema, an earlier Rust
|
|
work-in-progress schema, an unknown version, or a pre-existing empty database file
|
|
is rejected with instructions to select a fresh destination or use the explicit
|
|
Python importer. No `ALTER TABLE`, persistent pre-upgrade snapshot, migration
|
|
journal, automatic import, or backup relocation is performed.
|
|
|
|
Before creating storage locks or secrets, the server performs a read-only
|
|
preflight. SQLite inspects a private temporary copy of the database and its WAL
|
|
or rollback journal, so opening an incompatible database cannot create a source
|
|
`-shm` file. The temporary inspection copy is removed afterwards. The schema is
|
|
checked again while the server holds its storage locks. Existing current records
|
|
and file paths are left unchanged.
|
|
|
|
## Import command
|
|
|
|
Stop the old Python application, workers, and schedules. Keep its database, WAL,
|
|
rollback journal, and backup files together. Make an independent copy of the old
|
|
installation before switching services. This is an offline import, not a live
|
|
replication mechanism.
|
|
|
|
Use the helper from the new Rust project:
|
|
|
|
```sh
|
|
./import-python.sh /srv/old-mikrotik-app
|
|
```
|
|
|
|
Its equivalent application command is:
|
|
|
|
```sh
|
|
./start.sh import-python --source /srv/old-mikrotik-app
|
|
```
|
|
|
|
Optional source overrides are available when paths are customized or discovery
|
|
finds more than one candidate:
|
|
|
|
```sh
|
|
./import-python.sh /srv/old-mikrotik-app \
|
|
--database docker-data/db/routeros_backup_next.db \
|
|
--backup-root docker-data
|
|
```
|
|
|
|
Both `--database` and `--backup-root` are resolved **relative to `--source`**, unless
|
|
an absolute path is supplied. They select the old source, not the destination.
|
|
The helper resolves its first directory argument relative to the caller's working
|
|
directory before invoking the project's starter.
|
|
|
|
The destination comes from the new application's configuration. An explicit
|
|
example is:
|
|
|
|
```sh
|
|
DATA_DIR="$PWD/storage-imported" \
|
|
DATABASE_PATH="$PWD/storage-imported/routeros_backup_next.db" \
|
|
./import-python.sh /srv/old-mikrotik-app \
|
|
--database docker-data/db/routeros_backup_next.db \
|
|
--backup-root docker-data
|
|
```
|
|
|
|
After a successful import, use the same destination to start the server:
|
|
|
|
```sh
|
|
DATA_DIR="$PWD/storage-imported" \
|
|
DATABASE_PATH="$PWD/storage-imported/routeros_backup_next.db" \
|
|
./start.sh serve
|
|
```
|
|
|
|
The `migrate` command from previous revisions has been removed. The importer does
|
|
not start HTTP or background schedules. It does not add a new administrator when
|
|
users have been imported. Log in with an imported account and its existing password.
|
|
|
|
## Source discovery
|
|
|
|
Discovery follows the layouts present in the supplied Python/TypeScript project:
|
|
|
|
| Source layout | Recognized database location |
|
|
| --- | --- |
|
|
| Shipped Compose bind mount `./docker-data:/app/storage` | `docker-data/db/routeros_backup_next.db` or `docker-data/routeros_backup_next.db` |
|
|
| Local application working directory | `storage/routeros_backup_next.db` |
|
|
| Local backend working directory | `backend/storage/routeros_backup_next.db` |
|
|
| Explicit source configuration | SQLite `DATABASE_URL` and `DATA_DIR` in `.env` or `backend/.env` |
|
|
|
|
The known data directories are also checked for the documented `db/` database
|
|
subdirectory. Container paths under `/app/storage` are mapped to the shipped
|
|
`docker-data` mount. A customized Compose volume mapping is not executed or
|
|
interpreted as arbitrary code; supply explicit paths when it differs.
|
|
|
|
Only actual existing candidates are considered. **Multiple different database
|
|
files require `--database`**; discovery never chooses the first, newest, or largest
|
|
file. A configured database or data directory that is unavailable produces an
|
|
error instead of silently substituting another installation. Source overrides
|
|
allow the operator to resolve that situation explicitly.
|
|
|
|
The importer reads only `DATABASE_URL` and `DATA_DIR` from source environment
|
|
files. It does not execute shell expressions, expand variables, import the Python
|
|
application, or copy the source `.env`. Dynamic expressions and unsupported SQLite
|
|
URL options require explicit source paths. Providing both overrides bypasses
|
|
source environment-path discovery.
|
|
|
|
For each backup record, the importer resolves the recorded path or its preserved
|
|
filename within the selected or recognized source data roots. It supports the
|
|
original flat layout and older device-name subdirectories. Missing files,
|
|
multiple distinct matches, parent traversal, and symlinks escaping a selected
|
|
root are errors. Choose an unambiguous `--backup-root` when necessary; no source
|
|
file is silently skipped.
|
|
|
|
## What is preserved
|
|
|
|
The import creates a new database using the current schema and copies supported
|
|
records logically from a private source snapshot. It preserves:
|
|
|
|
- User, device, backup, settings, and operation-log IDs; device ownership.
|
|
- Existing password hashes, usernames, preferences, timestamps, and backup metadata.
|
|
- RouterOS/SwitchOS connection information, per-device credentials, and backup policies.
|
|
- Global settings, schedules, SSH keys, SwitchOS credentials, SMTP settings/secrets,
|
|
and Pushover settings/secrets stored in SQLite.
|
|
- Every referenced backup's bytes, original `file_name`, and existing checksum.
|
|
|
|
Fields absent in an older supported Python schema receive current defaults in the
|
|
new database only. A missing old `file_name` is derived from its recorded path.
|
|
Missing/null historical timestamps use the current schema's default when that
|
|
older schema did not store a usable value. A missing password hash, orphaned
|
|
ownership, or unsupported source structure causes an error; import never assigns
|
|
an arbitrary owner or replaces a missing password with an administrator password.
|
|
An empty source settings table receives the normal default settings row.
|
|
|
|
Imported SwitchOS records have `use_global_switchos_credentials = null` when the
|
|
Python schema has no explicit source selector. This preserves the former per-field
|
|
inheritance. Choosing global or device credentials later uses a complete pair
|
|
from that source without deleting saved device credentials. Secrets are not
|
|
returned in the public API or prefilled into the interface.
|
|
|
|
Passlib PBKDF2-SHA256 and supported Werkzeug PBKDF2 hashes remain usable without a
|
|
reset. New and changed passwords use Argon2id. The source environment's JWT secret
|
|
is not copied. Unless `SECRET_KEY` is explicitly configured for the new server,
|
|
its first startup creates a new `DATA_DIR/secrets/jwt.key`, and old sessions must
|
|
log in again. The import itself does not generate a JWT key.
|
|
|
|
## Destination layout and publication
|
|
|
|
The destination must be separate from the source application and source backup
|
|
roots. Its database must not exist, and its `backups/` tree must contain no existing
|
|
files. Existing empty directories, the new configuration's private files, and
|
|
storage locks do not authorize overwriting a database or backup. Relative paths,
|
|
parent components, and existing symlinks are resolved before any directory or lock
|
|
is created, using the same normalized paths for validation and subsequent writes.
|
|
|
|
Imported and newly created files use stable numeric device IDs:
|
|
|
|
| Backup type | Directory relative to `DATA_DIR` |
|
|
| --- | --- |
|
|
| RouterOS or SwitchOS binary | `backups/binary/<device-id>/` |
|
|
| RouterOS text export | `backups/exports/<device-id>/` |
|
|
|
|
An imported physical filename receives the backup ID as a prefix. Its metadata
|
|
`file_name` remains unchanged for downloads and RouterOS uploads. Names are bounded
|
|
and published without replacing existing files or symlinks. Renaming a device does
|
|
not rename its numeric directory. Database paths to these files are relative to
|
|
`DATA_DIR`.
|
|
|
|
The importer reads the stopped source database and any `-wal` or `-journal` as
|
|
ordinary files, copying them into private staging. It compares metadata and SHA-256
|
|
before and after copying and refuses a detected change. SQLite opens only this
|
|
private copy and includes committed WAL records. The source `-shm` is neither opened
|
|
by SQLite nor recreated; staging reconstructs its own coordination files when
|
|
needed. Keep the source stopped throughout the operation, because these checks
|
|
are not a substitute for coordinating concurrent writers.
|
|
|
|
All source rows and backup files are validated before the target database is
|
|
published. Every copied file is checked against its recorded SHA-256 when present
|
|
and against the copy's calculated SHA-256. Publication uses private temporary
|
|
files, file synchronization, atomic no-clobber publication, and directory
|
|
synchronization. The destination database is published last.
|
|
|
|
Before database publication, error cleanup makes a best-effort attempt to remove
|
|
files recorded as installed by that attempt; source files remain unchanged. If
|
|
directory synchronization fails after database publication, the database and its
|
|
referenced backups are retained, and the error asks the operator to inspect the
|
|
destination. An interrupted process or filesystem I/O failure can leave staging
|
|
or unreferenced files: inspect or choose
|
|
another new destination instead of expecting automatic startup recovery. There is
|
|
no migration journal or automatic resumption framework.
|
|
|
|
The CLI prints a JSON summary with the selected source database and counts of
|
|
users, devices, backup records, copied files, and logs. It never prints saved
|
|
passwords or keys. Check that report, then start the new service, inspect device
|
|
lists, preview an export, and download representative binary and text files.
|
|
Do not run both installations' schedules for the same devices during cutover.
|
|
|
|
## Rollback and operational limits
|
|
|
|
The original installation remains the rollback source. To return to it, stop the
|
|
new service and restart the old application on its unchanged data. New backups
|
|
created after cutover exist only in the new installation; preserve them and its
|
|
database before discarding or reverting anything. Do not exchange a SQLite main
|
|
file under a running process or mix unrelated main/WAL/journal files.
|
|
|
|
The importer does not copy operating-system permissions, service definitions,
|
|
external environment secrets, or SSH host-trust files. Review the new environment
|
|
configuration and host-key policy before first device contact. Imported SQLite
|
|
secrets retain their original representation; target database/secret files use
|
|
private permissions, and deployment storage must protect their contents.
|
|
|
|
Regression tests cover fresh initialization/restart, rejection without source
|
|
changes, explicit path selection and ambiguity, record/secret preservation, new
|
|
file layout, checksums, locks, symlinks, committed WAL data, and source-sidecar
|
|
immutability. Tests use generated fixtures; no production database or physical
|
|
MikroTik was supplied. Current verification results are recorded separately in
|
|
[validation.md](validation.md).
|
|
|
|
## Import with Docker Compose only
|
|
|
|
If the host does not have a Rust toolchain, run the importer from the same image as
|
|
the server. Stop the old application and make sure the new application's target
|
|
volume does not already contain a database. Mount the legacy project read-only:
|
|
|
|
```sh
|
|
docker compose build app
|
|
docker compose run --rm \
|
|
-v /srv/old-mikrotik-app:/legacy:ro \
|
|
app import-python \
|
|
--source /legacy \
|
|
--database docker-data/db/routeros_backup_next.db \
|
|
--backup-root docker-data
|
|
```
|
|
|
|
After the importer prints a successful JSON report, start the service:
|
|
|
|
```sh
|
|
docker compose up -d
|
|
```
|
|
|
|
If the legacy directory uses a different layout, change only the source paths passed
|
|
to `--database` and `--backup-root`. The target data remains in the new `/data`
|
|
volume.
|