mikrotik-backup/docs/validation.md

Validation report — revision 5 working copy

Date: 28 October 2026. This section applies to the revision 5 changes only; the older revision 4 and revision 3 results below remain historical evidence for their respective source revisions.

Area	Current result	Scope
JavaScript syntax	PASS	node --check web/js/*.js
JavaScript unit tests	16/16 PASS	node web/tests/core.test.cjs
Rust compile/tests	NOT RUN in this environment	cargo / rustc are not installed in the available runtime; no current Rust pass is claimed
Browser smoke/live tests	NOT RUN in this environment	Playwright is not installed; the test files could not start
SwitchOS protocol comparison	REVIEWED	Compared directly with the supplied Python SwosBetaService; local requests.HTTPDigestAuth output was also checked for header formatting
Physical SwitchOS device	NOT TESTED	Requires the user's device/firmware for final acceptance

The revision 5 patch changes SwitchOS authentication/probing, settings layout, and RSC diff payload/rendering. Digest handling now accepts a changed nonce/challenge for /backup.swb after homepage authentication and re-signs redirects for their final URI. A physical-device check remains required to confirm the original 502 scenario is resolved on the affected firmware.

---

Validation report — 0.0.1, revision 4

Date: 27 October 2026. This report separates completed revision 4 verification from historical results. Historical checks are explicitly labeled and are not presented as newly executed revision 4 checks.

The distribution contains source only: Rust code, Cargo.lock, interface sources, tests, and the patched library under vendor/. It does not ship an executable or bin/, target/, or storage/ directories. The input archives contained no production database or device credentials.

Current revision 4 results

Revision 4 replaces automatic schema/data/layout conversion with fresh initialization and an explicit Python importer. It also embeds API documentation directly from checked-in input files and translates project documentation into English.

Area	Result	Evidence and scope
Rust dev/test build and suite	111/111 PASS	95 unit tests, 11 API contract tests, 3 import-launcher tests, and 2 OpenAPI tests; no failures or skipped tests
Fresh installation and incompatible database	PASS	Fresh schema creation and restart; old/foreign/empty existing databases rejected without conversion or backup relocation
Explicit Python importer	PASS	Unit regressions plus six actual CLI/HTTP acceptance scenarios; source selection, unchanged originals, preserved records/files, and refusal to overwrite a destination
Import launcher	3/3 PASS	Validation before build, argument/path preservation including spaces and CDPATH, and import failure propagation
Source launchers and ignore rules	16/16 PASS	Isolated Cargo/Docker stubs, build-before-start, errors, arguments, working directories, target-directory overrides, and private/generated file exclusions
API documentation embedding	PASS	/docs, /openapi.json, local versioned assets, matching content hashes, custom prefix, and English documentation
Release-profile check	PASS	cargo check --release --locked --offline after removing all six previous generated documentation files; no OUT_DIR/docs.* files recreated
Formatting and static analysis	PASS	Package-scoped formatting check and cargo clippy --all-targets --locked --offline -- -D warnings
JavaScript unit tests	16/16 PASS	Existing interface behavior, translations, forms, and credential-source controls
Chromium with actual Rust and a SwOS fixture	8/8 PASS	Global credentials, backup bytes/layout, HTML rejection, English Swagger rendering, public requests, and OAuth-authorized requests
Chromium with actual Rust navigation	PASS	Eight routes, settings redirects, responsive layout, CSP, no API/JavaScript errors, and sign-out
Dependency integrity	PASS	All 91 libssh2 checksum entries and 5 Swagger UI assets verified; Cargo.lock unchanged from the documented historical audit
Release executable build	Not performed for revision 4	The deployment launcher builds release source on the target host
Docker / systemd execution	Not performed for revision 4	No container or service-manager execution result claimed

The actual import acceptance run used a generated database matching the supplied Python SQLAlchemy models and the original Compose directory layout. Its Passlib PBKDF2-SHA256 password hash was generated independently with Python's standard library. The run verified:

1. Startup rejects that original database without changing any source file or moving a backup.
2. Explicit import discovers the Compose database and creates a separate target; it neither executes nor copies the source .env.
3. IDs, ownership, password hashes, database secrets, logs, filenames, checksums, and every backup's bytes are preserved in the current schema and layout.
4. A second import refuses the existing target without overwriting it.
5. The imported account can sign in and download all three fixture backups after the first start and a subsequent restart.
6. Ordinary startup in another fresh directory creates an empty installation, without importing the nearby Python data.

Unit regressions additionally retain committed WAL data with an existing or missing source SHM file, compare source sidecar bytes before/after import, reject ambiguous or overlapping paths, and verify validation precedes secret/lock writes. No production database was used. Filesystem interruption and synchronization failures remain subject to the explicit cleanup limits in data-import.md.

The browser run executed public /health and an authenticated device-list request through the actual English Swagger UI. All documentation assets stayed on the same origin, with no JavaScript or CSP errors. The build/profile check and tests ran with the old generated documentation files removed; the updated documentation handler embeds checked-in input files directly.

Evidence is recorded in revision4-release-check.log, revision4-rust-tests.log, revision4-clippy.log, revision4-format.log, revision4-ui-core.log, revision4-live-import.json, revision4-live-switchos.json, revision4-ui-live.log, revision4-source-start-checks.json, and revision4-vendor-verification.json. These are acceptance-environment reports, not files promised in the source archive. The project contains the Rust, launcher, and interface regression tests.

The installation sequence is documented in data-import.md. There is no automatic upgrade, startup conversion, or migration recovery framework.

Historical revision 3 results

Revision 3 used actual dev/test builds with Rust 1.99.1, edition 2024. It did not run a revision 3 release build, Docker container, or physical MikroTik acceptance test.

Area	Historical revision 3 result	Scope and method
Dev/test compilation	PASS	Actual application and test compilation with Rust 1.99.1, edition 2024
Rust unit tests	91/91 PASS	Data, then-current migrations, security, files, diff, cron, adapters, and notifications
HTTP contract tests	11/11 PASS	Actual Axum router and SQLite, authorization, file operations, and SwitchOS credential-source modes
OpenAPI/documentation tests	2/2 PASS	All 36 operations, route/custom-prefix compatibility, and local Swagger assets without a CDN
Total Rust tests	104/104 PASS	91 unit + 11 API + 2 OpenAPI; zero failures or skipped tests
JavaScript tests	16/16 PASS	UI logic, forms, schedules, translations, and interface regressions
Chromium workflow with controlled API	PASS	113 requests to a local fixture API, full interface workflow, no JavaScript errors
Final formatting	PASS	cargo fmt --package mikrotik-backup -- --check
Final Rust static analysis	PASS	cargo clippy --all-targets --locked -- -D warnings
Chromium + actual Rust, Swagger UI, and SwOS	8/8 PASS	Browser, running backend, SQLite, and an independent local HTTP Digest SwOS fixture
Chromium + actual Rust navigation	PASS	Eight routes, old-route redirects, responsiveness, CSP, no API/JS errors, and sign-out
Vendored integrity recheck	PASS	All 91 entries in vendor/libssh2-sys/SHA256SUMS and 5 Swagger UI files matched; Cargo.lock unchanged
Revision 3 release build	Not performed	The deployment launcher builds release on the target host
Revision 3 Docker execution	Not performed	No image/container execution result established

These results were recorded in revision3-rust-tests.log, revision3-ui-core.log, revision3-ui-workflow.log, revision3-ui-live.log, revision3-live-switchos.json, revision3-clippy.log, revision3-format.log, and revision3-vendor-verification.json in the acceptance tooling directory. These are verification-environment logs, not files promised as part of the source archive. The Rust and JavaScript test sources are included in the project.

Historical migration, file, and concurrency checks

The automatic migration behavior in this subsection belonged to revision 3 and has been removed in revision 4. It is retained here only to identify the scope of the historical 91-unit-test result.

Those tests preserved IDs, ownership, password hashes, settings, and secrets while upgrading old schemas. Migration 002 kept use_global_switchos_credentials = null for existing devices. Migration 003 added a persistent backup-layout journal. Fresh backups, imports, and startup conversions used DATA_DIR/backups/binary/<router_id>/ and DATA_DIR/backups/exports/<router_id>/; renaming a device did not change the directory ID, and the original file_name was preserved.

Checks covered a database snapshot before path changes, file copying before metadata transactions, SHA-256, and idempotent restart. Fault cases interrupted the logical process after plan persistence, after verified copying, and between file publication and journal acknowledgement. Restart did not overwrite unrelated files, even with identical bytes. Missing/modified sources, corrupted copies, tampered journal paths, and snapshot symlinks prevented partial activation of changed records.

Separate cases covered source-preserving import, target conflicts, storage/database locks, symlinks outside the selected source, and a dangling destination database symlink. Sixteen threads concurrently created initial device/type directories; a directory encountered after a race was checked again for symlinks.

Retention and deletion operated on active files and preserved originals retained by that revision, even if the newer active copy was absent. Tests forced both a SQLite DELETE rejection and a failed compensating rename; the remaining copy had to survive as a recovery file. Code review also examined directory synchronization after imported database publication and backup preservation following a later failure. No final fsync failure, physical power loss, or storage hardware failure was injected.

Revision 4 uses a new destination and an explicit import instead of this startup conversion mechanism. Historical results do not validate the replacement until its own checks have completed.

Historical credentials, SwOS adapter, and HTTP contract checks

Revision 3 API tests covered all three SwitchOS credential modes: a complete global pair, a complete local pair, and legacy field-by-field fallback. Cases included creation, partial PUT, ordinary saves that preserve legacy semantics, global changes without rewriting device records, and returning from global mode to local mode without losing the stored password. They verified no implicit fallback in explicit modes, a clear missing-username error, empty-password support, and masked responses.

Adapter tests used local HTTP servers with controlled responses. They covered Digest, Basic, and authentication retries when a public HTML page did not return a 401 challenge. Checks included the correct system and backup endpoints, the Digest request URI, a global credential pair, explicit URL ports, and actual response-byte transfer and validation. HTML/login pages could neither be saved as backups nor produce a positive availability result. A missing endpoint could not be replaced with a public homepage; redirects did not forward credentials; the timeout covered the whole operation. These were adapter/protocol tests, not hardware acceptance for a particular SwOS model.

Other repeated cases covered authorization, ownership, JSON/form login, preferences, password changes, and the bounded password-operation pool. Independent fixtures verified Passlib/Werkzeug passwords and HS256 compatibility, invalid exp/nbf rejection, secret preservation/clearing, download bytes, RSC previews, and ZIP. Unified diff passed 32 Python difflib fixtures. Local SMTP and Pushover tests did not send to production recipients; the SMTP fixture did not establish STARTTLS compatibility with an external provider.

The two documentation tests covered all 36 API operations, actual route coverage, a custom prefix, and documentation assets served without a CDN. A completed Chromium run against an actual Rust process loaded all 36 operations, executed public /health, signed in through Authorize, and fetched an authenticated device list. There were no external-server requests, JavaScript errors, or CSP violations.

That same run created a SwitchOS device through the form using global credentials without copying secrets into the device. The connection test and backup used Digest with a /sys.b challenge and the correct /backup.swb URI. Downloaded bytes, SHA-256, and backups/binary/<router_id>/ were checked, including valid bytes mislabeled text/html. Rotating the global username/password took effect without editing the device. Actual HTML was rejected with HTTP 502, no backup record was added, and the connection test did not report availability. The fixture device, backups, and global test credentials were removed afterward.

Historical revision 3 interface checks

The mocked-API Chromium workflow made 113 requests. It covered login and registration, device forms, secret preservation, the global → local → global → local sequence, saving legacy null mode, downloads, binary upload, and bulk deletion. It checked RSC-only diff actions, hidden SwitchOS export/unsupported actions, toast errors, log filters, preferences, and password changes through the form.

Checks also covered flat daily/weekly/custom automation controls, old-route redirects, responsive layout, themes, escaping, and sign-out. A separate actual-Rust run visited /devices, /files, /diff-configs, /logs, /settings, /schedule, /change-password, and /. It confirmed the automation and account redirects, the flat automation layout, a mobile view without horizontal page scrolling, and sign-out. Neither workflow had JavaScript errors; the live-server run also had no API or CSP errors.

Historical revision 2 checks

These separate runs are not described as rerun in revision 3 or revision 4. The ordinary test suite reran its included SSH unit tests during revision 3; the five scenarios below were a separate acceptance run with an actual application process.

Area	Historical revision 2 result	Scope
SSH trust over an actual local protocol	5/5 PASS	Default accept-new, repeated pin verification, changed-key rejection, invalid private key after host verification, and four concurrent first contacts
Launchers and project rules	16/16 PASS	Bash with isolated Cargo/Docker substitutes; build-before-start, errors, arguments, paths with spaces, CARGO_TARGET_DIR, Compose, .env; 31 ignored paths and 25 retained source paths
Chromium + actual Rust	PASS	Eight then-current routes, responsive layout, CSP, sign-out, password change through Settings, and signing in again
RustSec audit	PASS	270 dependencies, 0 reported vulnerabilities, 0 warnings, no ignored advisories
Vendor integrity	PASS	All 91 SHA256SUMS entries matched

Revision 3's Cargo.lock was unchanged from that audit. This does not mean the advisory database was fetched again or a new scan was performed. The historical scan and its metadata remain in dependency-audit.json and dependency-audit-metadata.json.

SSH cases used loopback, generated test keys, and real handshake/authentication. Launcher checks used substitute CLIs; they neither built release nor ran Docker.

Historical initial acceptance — before revision 2

Initial acceptance included a release build and actual local SSH/SFTP/SCP sessions emulating RouterOS responses. It covered password, own-key and global-key authentication, identity/resource, exports, binary backup, download, upload, SHA-256, and remote temporary-file cleanup. Truncated SCP, negative ACK, and nonzero final exit status were rejected. These runs are not new revision 3 or revision 4 tests.

A historical scheduler check removed a controlled expired file and its SQLite record. Other checks covered restart, persistence, healthcheck, check-config, rejecting a second instance, and SIGTERM during an incomplete request; SQLite reported integrity_check = ok. A separate libssh2 C test exercised short/oversized AES-GCM blocks and overflowing username lengths against earlier debug and release archives. Patch reproduction confirmed all seven modified C files matched.

At that time, the launcher could run a bundled executable without Cargo. That path has been removed. Current start.sh builds the source; the historical release artifact is not included in this source archive. Backport details are in dependency-security.md.

Limits and reproduction

• Hardware and data: no physical MikroTik or unavailable production database was tested. Local fixtures do not establish compatibility with every model/firmware.
• Network: local SMTP/Pushover checks do not verify external provider accounts or STARTTLS. The ICMP test permits an explicit socket-permission failure; PASS alone does not prove an echo reply when the environment denied socket creation.
• Deployment: no revision 4 release executable was linked or run, and Docker and systemd were not executed. The release-profile check, dev/test compilation, and live debug-server acceptance are separate completed checks.
• Dependency security: the historical RustSec result applies to a specific advisory set, date, and lockfile, not every possible vulnerability. System OpenSSL, zlib, and glibc updates and new advisories remain relevant.

To reproduce local verification gates:

cargo fmt --package mikrotik-backup -- --check
cargo clippy --all-targets --locked -- -D warnings
cargo test --locked
cargo check --release --locked
node web/tests/core.test.cjs

Run cargo audit --file Cargo.lock --json for a new dependency scan. Running ./start.sh builds release and starts the application on the deployment host. cargo check --release checks the release profile but does not produce a tested release executable. These commands are instructions, not additional recorded passes. ui.md describes browser checks. Fixture servers and acceptance tools are not required for normal application operation.

Revision 6 targeted SwitchOS change

The source adds a regression fixture for a SwitchOS server that sends a Digest challenge together with Set-Cookie and returns an HTML login page if the cookie is missing from the authenticated /backup.swb retry. The adapter now preserves that cookie, drains the challenge response before retrying, forces HTTP/1.x, and starts the backup request with fresh authentication state.

The current execution environment does not contain cargo, rustc, or rustfmt, so the Rust test suite and compiler checks were not executed for revision 6 here. The new test is included for execution on the build/deployment host with cargo test --locked.

Revision 7 targeted UI, diff, container, and importer review

The browser diff now requests a compact JSON response that omits the duplicated unified text and HTML documents. The UI renders at most 200 structured rows per page, filters before rendering, and fetches the full unified .diff text only when the user explicitly downloads it. Semantic RSC analysis is included alongside the line diff; it ignores comments/export metadata and command ordering, groups changes by RouterOS section, pairs recognizable commands, and flags selected security/network/service sections.

Current JavaScript verification in the patch environment:

• node --test web/tests/core.test.cjs: 16/16 PASS.
• node --check web/js/*.js: PASS.
• Playwright browser acceptance: NOT RUN because the Playwright Node module is not installed.
• Rust compiler/tests: NOT RUN because cargo and rustc are unavailable.
• Docker build/runtime: NOT RUN because the Docker CLI/daemon is unavailable.

The supplied Python/TypeScript SQLAlchemy models were compared with src/schema.sql. All legacy columns in users, routers, backups, operation_logs, and global_settings are represented in the Rust schema; the Rust routers table adds use_global_switchos_credentials. The importer copies supported source columns into a new target, validates referenced backup files and recorded checksums, and publishes the database last. No production database was supplied, so this is a source/schema review plus existing importer regression coverage, not a production-data migration run.