gru/mikrotik-backup
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

first commit

Browse Source
This commit is contained in:
Mateusz Gruszczyński committed 2026-10-28 09:16:54 +01:00
commit 32a7ba8e8f
168 files changed
+123856
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/validation.md
+329
View File
@@ -0,0 +1,329 @@
# Validation report — revision 5 working copy
Date: **28 October 2026**. This section applies to the revision 5 changes only; the
older revision 4 and revision 3 results below remain historical evidence for their
respective source revisions.
| Area | Current result | Scope |
| --- | --- | --- |
| JavaScript syntax | **PASS** | `node --check web/js/*.js` |
| JavaScript unit tests | **16/16 PASS** | `node web/tests/core.test.cjs` |
| Rust compile/tests | **NOT RUN in this environment** | `cargo` / `rustc` are not installed in the available runtime; no current Rust pass is claimed |
| Browser smoke/live tests | **NOT RUN in this environment** | Playwright is not installed; the test files could not start |
| SwitchOS protocol comparison | **REVIEWED** | Compared directly with the supplied Python `SwosBetaService`; local `requests.HTTPDigestAuth` output was also checked for header formatting |
| Physical SwitchOS device | **NOT TESTED** | Requires the user's device/firmware for final acceptance |
The revision 5 patch changes SwitchOS authentication/probing, settings layout, and
RSC diff payload/rendering. Digest handling now accepts a changed nonce/challenge for
`/backup.swb` after homepage authentication and re-signs redirects for their final URI.
A physical-device check remains required to confirm the original 502 scenario is
resolved on the affected firmware.
---
# Validation report — 0.0.1, revision 4
Date: **27 October 2026**. This report separates completed revision 4 verification
from historical results. Historical checks are explicitly labeled and are not
presented as newly executed revision 4 checks.
The distribution contains **source only**: Rust code, `Cargo.lock`, interface sources,
tests, and the patched library under `vendor/`. It does not ship an executable or
`bin/`, `target/`, or `storage/` directories. The input archives contained no production
database or device credentials.
## Current revision 4 results
Revision 4 replaces automatic schema/data/layout conversion with fresh initialization
and an explicit Python importer. It also embeds API documentation directly from
checked-in input files and translates project documentation into English.
| Area | Result | Evidence and scope |
| --- | --- | --- |
| Rust dev/test build and suite | **111/111 PASS** | 95 unit tests, 11 API contract tests, 3 import-launcher tests, and 2 OpenAPI tests; no failures or skipped tests |
| Fresh installation and incompatible database | **PASS** | Fresh schema creation and restart; old/foreign/empty existing databases rejected without conversion or backup relocation |
| Explicit Python importer | **PASS** | Unit regressions plus six actual CLI/HTTP acceptance scenarios; source selection, unchanged originals, preserved records/files, and refusal to overwrite a destination |
| Import launcher | **3/3 PASS** | Validation before build, argument/path preservation including spaces and CDPATH, and import failure propagation |
| Source launchers and ignore rules | **16/16 PASS** | Isolated Cargo/Docker stubs, build-before-start, errors, arguments, working directories, target-directory overrides, and private/generated file exclusions |
| API documentation embedding | **PASS** | `/docs`, `/openapi.json`, local versioned assets, matching content hashes, custom prefix, and English documentation |
| Release-profile check | **PASS** | `cargo check --release --locked --offline` after removing all six previous generated documentation files; no `OUT_DIR/docs.*` files recreated |
| Formatting and static analysis | **PASS** | Package-scoped formatting check and `cargo clippy --all-targets --locked --offline -- -D warnings` |
| JavaScript unit tests | **16/16 PASS** | Existing interface behavior, translations, forms, and credential-source controls |
| Chromium with actual Rust and a SwOS fixture | **8/8 PASS** | Global credentials, backup bytes/layout, HTML rejection, English Swagger rendering, public requests, and OAuth-authorized requests |
| Chromium with actual Rust navigation | **PASS** | Eight routes, settings redirects, responsive layout, CSP, no API/JavaScript errors, and sign-out |
| Dependency integrity | **PASS** | All 91 libssh2 checksum entries and 5 Swagger UI assets verified; Cargo.lock unchanged from the documented historical audit |
| Release executable build | Not performed for revision 4 | The deployment launcher builds release source on the target host |
| Docker / systemd execution | Not performed for revision 4 | No container or service-manager execution result claimed |
The actual import acceptance run used a generated database matching the supplied
Python SQLAlchemy models and the original Compose directory layout. Its Passlib
PBKDF2-SHA256 password hash was generated independently with Python's standard
library. The run verified:
1. Startup rejects that original database without changing any source file or
moving a backup.
2. Explicit import discovers the Compose database and creates a separate target;
it neither executes nor copies the source `.env`.
3. IDs, ownership, password hashes, database secrets, logs, filenames, checksums,
and every backup's bytes are preserved in the current schema and layout.
4. A second import refuses the existing target without overwriting it.
5. The imported account can sign in and download all three fixture backups after
the first start and a subsequent restart.
6. Ordinary startup in another fresh directory creates an empty installation,
without importing the nearby Python data.
Unit regressions additionally retain committed WAL data with an existing or
missing source SHM file, compare source sidecar bytes before/after import, reject
ambiguous or overlapping paths, and verify validation precedes secret/lock writes.
No production database was used. Filesystem interruption and synchronization
failures remain subject to the explicit cleanup limits in [data-import.md](data-import.md).
The browser run executed public `/health` and an authenticated device-list request
through the actual English Swagger UI. All documentation assets stayed on the
same origin, with no JavaScript or CSP errors. The build/profile check and tests
ran with the old generated documentation files removed; the updated documentation
handler embeds checked-in input files directly.
Evidence is recorded in `revision4-release-check.log`, `revision4-rust-tests.log`,
`revision4-clippy.log`, `revision4-format.log`, `revision4-ui-core.log`,
`revision4-live-import.json`, `revision4-live-switchos.json`, `revision4-ui-live.log`,
`revision4-source-start-checks.json`, and `revision4-vendor-verification.json`.
These are acceptance-environment reports, not files promised in the source archive.
The project contains the Rust, launcher, and interface regression tests.
The installation sequence is documented in [data-import.md](data-import.md).
There is no automatic upgrade, startup conversion, or migration recovery framework.
## Historical revision 3 results
Revision 3 used actual dev/test builds with Rust 1.99.1, edition 2024. It did not run
a revision 3 release build, Docker container, or physical MikroTik acceptance test.
| Area | Historical revision 3 result | Scope and method |
| --- | --- | --- |
| Dev/test compilation | PASS | Actual application and test compilation with Rust 1.99.1, edition 2024 |
| Rust unit tests | **91/91 PASS** | Data, then-current migrations, security, files, diff, cron, adapters, and notifications |
| HTTP contract tests | **11/11 PASS** | Actual Axum router and SQLite, authorization, file operations, and SwitchOS credential-source modes |
| OpenAPI/documentation tests | **2/2 PASS** | All 36 operations, route/custom-prefix compatibility, and local Swagger assets without a CDN |
| Total Rust tests | **104/104 PASS** | 91 unit + 11 API + 2 OpenAPI; zero failures or skipped tests |
| JavaScript tests | **16/16 PASS** | UI logic, forms, schedules, translations, and interface regressions |
| Chromium workflow with controlled API | PASS | **113 requests** to a local fixture API, full interface workflow, no JavaScript errors |
| Final formatting | **PASS** | `cargo fmt --package mikrotik-backup -- --check` |
| Final Rust static analysis | **PASS** | `cargo clippy --all-targets --locked -- -D warnings` |
| Chromium + actual Rust, Swagger UI, and SwOS | **8/8 PASS** | Browser, running backend, SQLite, and an independent local HTTP Digest SwOS fixture |
| Chromium + actual Rust navigation | **PASS** | Eight routes, old-route redirects, responsiveness, CSP, no API/JS errors, and sign-out |
| Vendored integrity recheck | **PASS** | All **91 entries** in `vendor/libssh2-sys/SHA256SUMS` and **5 Swagger UI files** matched; `Cargo.lock` unchanged |
| Revision 3 release build | Not performed | The deployment launcher builds release on the target host |
| Revision 3 Docker execution | Not performed | No image/container execution result established |
These results were recorded in `revision3-rust-tests.log`, `revision3-ui-core.log`,
`revision3-ui-workflow.log`, `revision3-ui-live.log`, `revision3-live-switchos.json`,
`revision3-clippy.log`, `revision3-format.log`, and
`revision3-vendor-verification.json` in the acceptance tooling directory. These are
verification-environment logs, not files promised as part of the source archive.
The Rust and JavaScript test sources are included in the project.
### Historical migration, file, and concurrency checks
**The automatic migration behavior in this subsection belonged to revision 3 and
has been removed in revision 4.** It is retained here only to identify the scope of
the historical 91-unit-test result.
Those tests preserved IDs, ownership, password hashes, settings, and secrets while
upgrading old schemas. Migration 002 kept
`use_global_switchos_credentials = null` for existing devices. Migration 003 added
a persistent backup-layout journal. Fresh backups, imports, and startup conversions
used `DATA_DIR/backups/binary/<router_id>/` and
`DATA_DIR/backups/exports/<router_id>/`; renaming a device did not change the directory
ID, and the original `file_name` was preserved.
Checks covered a database snapshot before path changes, file copying before metadata
transactions, SHA-256, and idempotent restart. Fault cases interrupted the logical
process after plan persistence, after verified copying, and between file publication
and journal acknowledgement. Restart did not overwrite unrelated files, even with
identical bytes. Missing/modified sources, corrupted copies, tampered journal paths,
and snapshot symlinks prevented partial activation of changed records.
Separate cases covered source-preserving import, target conflicts, storage/database
locks, symlinks outside the selected source, and a dangling destination database
symlink. **Sixteen threads** concurrently created initial device/type directories;
a directory encountered after a race was checked again for symlinks.
Retention and deletion operated on active files and preserved originals retained by
that revision, even if the newer active copy was absent. Tests forced both a SQLite
DELETE rejection and a failed compensating rename; the remaining copy had to survive
as a recovery file. Code review also examined directory synchronization after imported
database publication and backup preservation following a later failure. No final
fsync failure, physical power loss, or storage hardware failure was injected.
Revision 4 uses a new destination and an explicit import instead of this startup
conversion mechanism. Historical results do not validate the replacement until its
own checks have completed.
### Historical credentials, SwOS adapter, and HTTP contract checks
Revision 3 API tests covered all three SwitchOS credential modes: a complete global
pair, a complete local pair, and legacy field-by-field fallback. Cases included
creation, partial PUT, ordinary saves that preserve legacy semantics, global changes
without rewriting device records, and returning from global mode to local mode
without losing the stored password. They verified no implicit fallback in explicit
modes, a clear missing-username error, empty-password support, and masked responses.
Adapter tests used **local HTTP servers with controlled responses**. They covered
Digest, Basic, and authentication retries when a public HTML page did not return a
401 challenge. Checks included the correct system and backup endpoints, the Digest
request URI, a global credential pair, explicit URL ports, and actual response-byte
transfer and validation. HTML/login pages could neither be saved as backups nor
produce a positive availability result. A missing endpoint could not be replaced
with a public homepage; redirects did not forward credentials; the timeout covered
the whole operation. These were adapter/protocol tests, not hardware acceptance for
a particular SwOS model.
Other repeated cases covered authorization, ownership, JSON/form login, preferences,
password changes, and the bounded password-operation pool. Independent fixtures
verified Passlib/Werkzeug passwords and HS256 compatibility, invalid `exp`/`nbf`
rejection, secret preservation/clearing, download bytes, RSC previews, and ZIP.
Unified diff passed **32 Python difflib fixtures**. Local SMTP and Pushover tests did
not send to production recipients; the SMTP fixture did not establish STARTTLS
compatibility with an external provider.
The two documentation tests covered all **36 API operations**, actual route coverage,
a custom prefix, and documentation assets served without a CDN. A completed Chromium
run against an actual Rust process loaded all 36 operations, executed public
`/health`, signed in through **Authorize**, and fetched an authenticated device list.
There were no external-server requests, JavaScript errors, or CSP violations.
That same run created a SwitchOS device through the form using global credentials
without copying secrets into the device. The connection test and backup used Digest
with a `/sys.b` challenge and the correct `/backup.swb` URI. Downloaded bytes, SHA-256,
and `backups/binary/<router_id>/` were checked, including valid bytes mislabeled
`text/html`. Rotating the global username/password took effect without editing the
device. Actual HTML was rejected with HTTP 502, no backup record was added, and the
connection test did not report availability. The fixture device, backups, and global
test credentials were removed afterward.
### Historical revision 3 interface checks
The mocked-API Chromium workflow made **113 requests**. It covered login and
registration, device forms, secret preservation, the global → local → global → local
sequence, saving legacy `null` mode, downloads, binary upload, and bulk deletion.
It checked RSC-only diff actions, hidden SwitchOS export/unsupported actions,
toast errors, log filters, preferences, and password changes through the form.
Checks also covered flat daily/weekly/custom automation controls, old-route redirects,
responsive layout, themes, escaping, and sign-out. A separate actual-Rust run visited
`/devices`, `/files`, `/diff-configs`, `/logs`, `/settings`, `/schedule`,
`/change-password`, and `/`. It confirmed the automation and account redirects,
the flat automation layout, a mobile view without horizontal page scrolling, and
sign-out. Neither workflow had JavaScript errors; the live-server run also had no
API or CSP errors.
## Historical revision 2 checks
These separate runs are **not described as rerun in revision 3 or revision 4**.
The ordinary test suite reran its included SSH unit tests during revision 3; the five
scenarios below were a separate acceptance run with an actual application process.
| Area | Historical revision 2 result | Scope |
| --- | --- | --- |
| SSH trust over an actual local protocol | **5/5 PASS** | Default `accept-new`, repeated pin verification, changed-key rejection, invalid private key after host verification, and four concurrent first contacts |
| Launchers and project rules | **16/16 PASS** | Bash with isolated Cargo/Docker substitutes; build-before-start, errors, arguments, paths with spaces, `CARGO_TARGET_DIR`, Compose, `.env`; 31 ignored paths and 25 retained source paths |
| Chromium + actual Rust | PASS | Eight then-current routes, responsive layout, CSP, sign-out, password change through Settings, and signing in again |
| RustSec audit | PASS | **270 dependencies, 0 reported vulnerabilities, 0 warnings**, no ignored advisories |
| Vendor integrity | PASS | All **91 SHA256SUMS entries** matched |
Revision 3's `Cargo.lock` was unchanged from that audit. This does not mean the
advisory database was fetched again or a new scan was performed. The historical scan
and its metadata remain in [dependency-audit.json](dependency-audit.json) and
[dependency-audit-metadata.json](dependency-audit-metadata.json).
SSH cases used loopback, generated test keys, and real handshake/authentication.
Launcher checks used substitute CLIs; **they neither built release nor ran Docker**.
## Historical initial acceptance — before revision 2
Initial acceptance included a release build and actual local SSH/SFTP/SCP sessions
emulating RouterOS responses. It covered password, own-key and global-key
authentication, identity/resource, exports, binary backup, download, upload, SHA-256,
and remote temporary-file cleanup. Truncated SCP, negative ACK, and nonzero final
exit status were rejected. These runs are not new revision 3 or revision 4 tests.
A historical scheduler check removed a controlled expired file and its SQLite record.
Other checks covered restart, persistence, `healthcheck`, `check-config`, rejecting
a second instance, and SIGTERM during an incomplete request; SQLite reported
`integrity_check = ok`. A separate libssh2 C test exercised short/oversized AES-GCM
blocks and overflowing username lengths against earlier debug and release archives.
Patch reproduction confirmed all seven modified C files matched.
At that time, the launcher could run a bundled executable without Cargo. **That path
has been removed.** Current `start.sh` builds the source; the historical release
artifact is not included in this source archive. Backport details are in
[dependency-security.md](dependency-security.md).
## Limits and reproduction
- **Hardware and data:** no physical MikroTik or unavailable production database was
tested. Local fixtures do not establish compatibility with every model/firmware.
- **Network:** local SMTP/Pushover checks do not verify external provider accounts
or STARTTLS. The ICMP test permits an explicit socket-permission failure; PASS
alone does not prove an echo reply when the environment denied socket creation.
- **Deployment:** no revision 4 release executable was linked or run, and Docker
and systemd were not executed. The release-profile check, dev/test compilation,
and live debug-server acceptance are separate completed checks.
- **Dependency security:** the historical RustSec result applies to a specific
advisory set, date, and lockfile, not every possible vulnerability. System
OpenSSL, zlib, and glibc updates and new advisories remain relevant.
To reproduce local verification gates:
```bash
cargo fmt --package mikrotik-backup -- --check
cargo clippy --all-targets --locked -- -D warnings
cargo test --locked
cargo check --release --locked
node web/tests/core.test.cjs
```
Run `cargo audit --file Cargo.lock --json` for a new dependency scan. Running
`./start.sh` builds release and starts the application on the deployment host.
`cargo check --release` checks the release profile but does not produce a tested
release executable. These commands are instructions, not additional recorded passes.
[ui.md](ui.md) describes browser checks. Fixture servers and acceptance tools are
not required for normal application operation.
## Revision 6 targeted SwitchOS change
The source adds a regression fixture for a SwitchOS server that sends a Digest
challenge together with `Set-Cookie` and returns an HTML login page if the cookie is
missing from the authenticated `/backup.swb` retry. The adapter now preserves that
cookie, drains the challenge response before retrying, forces HTTP/1.x, and starts the
backup request with fresh authentication state.
The current execution environment does not contain `cargo`, `rustc`, or `rustfmt`, so
the Rust test suite and compiler checks were **not executed for revision 6 here**. The
new test is included for execution on the build/deployment host with `cargo test --locked`.
## Revision 7 targeted UI, diff, container, and importer review
The browser diff now requests a compact JSON response that omits the duplicated unified
text and HTML documents. The UI renders at most 200 structured rows per page, filters
before rendering, and fetches the full unified `.diff` text only when the user explicitly
downloads it. Semantic RSC analysis is included alongside the line diff; it ignores
comments/export metadata and command ordering, groups changes by RouterOS section,
pairs recognizable commands, and flags selected security/network/service sections.
Current JavaScript verification in the patch environment:
- `node --test web/tests/core.test.cjs`: **16/16 PASS**.
- `node --check web/js/*.js`: **PASS**.
- Playwright browser acceptance: **NOT RUN** because the Playwright Node module is not installed.
- Rust compiler/tests: **NOT RUN** because `cargo` and `rustc` are unavailable.
- Docker build/runtime: **NOT RUN** because the Docker CLI/daemon is unavailable.
The supplied Python/TypeScript SQLAlchemy models were compared with `src/schema.sql`.
All legacy columns in `users`, `routers`, `backups`, `operation_logs`, and
`global_settings` are represented in the Rust schema; the Rust `routers` table adds
`use_global_switchos_credentials`. The importer copies supported source columns into
a new target, validates referenced backup files and recorded checksums, and publishes
the database last. No production database was supplied, so this is a source/schema
review plus existing importer regression coverage, not a production-data migration run.