first commit

2026-08-13 15:58:52 +02:00
commit adfdb0b86c
100 changed files with 6216 additions and 0 deletions
@@ -0,0 +1,490 @@
+# RouterOS TZSP + Suricata IDS
+
+Project version: `0.3.2`
+
+A lightweight IDS stack designed to run as a **single container on MikroTik RouterOS**.
+RouterOS mirrors selected traffic with TZSP, the container decodes the frames into a TAP interface, Suricata analyzes them, and the Python service stores EVE alerts in SQLite and exposes a small web dashboard.
+
+## Architecture
+
+```text
+VLAN / RouterOS traffic
+        |
+        v
+RouterOS Packet Sniffer
+        |
+        | TZSP UDP/37008
+        v
+single RouterOS container
+  Debian slim
+  + Python TZSP receiver
+  + TAP suritap0
+  + Suricata IDS
+  + EVE JSON watcher
+  + SQLite
+  + Web UI :8080
+  + optional RouterOS REST blocking
+```
+
+The RouterOS deployment workflow is:
+
+```text
+Docker/Podman build host
+        |
+        | build for RouterOS CPU architecture
+        v
+container image
+        |
+        | docker save / podman save
+        v
+*.tar
+        |
+        | SCP
+        v
+RouterOS external disk
+        |
+        | /container/add file=...
+        v
+running IDS container
+```
+
+Docker Compose is only provided for Linux integration testing. RouterOS receives one saved container image as a TAR archive.
+
+---
+
+## Local Web UI development without Docker
+
+The web dashboard can be started locally without Docker, Suricata, TAP, `/dev/net/tun`, or root privileges.
+
+Requirements:
+
+- Linux or macOS
+- Python 3
+- Python `venv` support
+
+Start the development dashboard:
+
+```bash
+./dev.sh
+```
+
+The script will:
+
+1. create `.venv/` if needed,
+2. install dependencies from `requirements.txt`,
+3. create `data/dev/`,
+4. start the web-only application.
+
+Open:
+
+```text
+http://127.0.0.1:8080
+```
+
+This mode intentionally does **not** start Suricata, TZSP capture, TAP, or RouterOS integration.
+
+### Status API
+
+The dashboard and external monitoring systems can use:
+
+```text
+GET /api/status
+```
+
+Example:
+
+```bash
+curl http://127.0.0.1:8080/api/status
+```
+
+The response includes:
+
+- overall application status and uptime,
+- Web UI/API status,
+- TZSP receiver status,
+- TAP interface status,
+- Suricata process status and PID,
+- EVE JSON watcher status,
+- RouterOS REST integration status,
+- listening/outbound port information,
+- runtime packet, alert, and block counters.
+
+`GET /api/health` is kept as a compatibility alias and returns the same status payload.
+
+### Add a sample alert
+
+To start the dashboard with one demo alert in an empty development database:
+
+```bash
+DEV_SEED_DATA=true ./dev.sh
+```
+
+Development database:
+
+```text
+data/dev/ids.db
+```
+
+To use another port:
+
+```bash
+WEB_PORT=8090 ./dev.sh
+```
+
+To listen on all local interfaces:
+
+```bash
+WEB_BIND=0.0.0.0 ./dev.sh
+```
+
+Do not expose the development server directly to an untrusted network.
+
+---
+
+## Local tests
+
+Install dependencies and run tests:
+
+```bash
+./dev.sh --test
+```
+
+Or run tests before starting the dashboard:
+
+```bash
+RUN_TESTS=true ./dev.sh
+```
+
+The project currently uses only the Python standard library, including `unittest` for tests. `requirements.txt` is kept as the canonical dependency file so future packages can be installed automatically by `dev.sh`.
+
+---
+
+## Docker integration test
+
+Prepare local environment configuration:
+
+```bash
+cp .env.example .env
+```
+
+Build and start:
+
+```bash
+docker compose up -d --build
+```
+
+Dashboard:
+
+```text
+http://127.0.0.1:8080
+```
+
+Send the built-in TZSP test packet:
+
+```bash
+./scripts/selftest.sh
+```
+
+The test rule should generate:
+
+```text
+LOCAL TZSP PIPELINE TEST
+SID 1000001
+```
+
+Stop the stack:
+
+```bash
+docker compose down
+```
+
+---
+
+## RouterOS requirements
+
+Before deployment, verify that the router has:
+
+- a RouterOS version and package set that supports containers,
+- the `container` package installed,
+- container and sniffer functionality enabled in device mode,
+- SSH access from the build machine,
+- enough storage for the image, container root directory, Suricata rules, logs, and SQLite database,
+- preferably an external disk instead of small internal flash storage.
+
+Useful RouterOS checks:
+
+```routeros
+/system/resource/print
+/system/package/print
+/system/device-mode/print
+/disk/print
+/container/print
+```
+
+---
+
+## Automated RouterOS deployment
+
+Create the deployment configuration:
+
+```bash
+cp deploy-routeros.env.example deploy-routeros.env
+```
+
+Edit at least:
+
+```dotenv
+ROUTER_HOST=192.168.88.1
+ROUTER_USER=admin
+ROUTER_ARCH=auto
+ROUTER_DISK=disk1
+ROUTER_SCP_DIR=disk1
+VLAN_ID=100
+MONITORED_NETWORKS=192.168.100.0/24
+AUTO_BLOCK=false
+```
+
+Deploy:
+
+```bash
+./scripts/deploy-routeros.sh
+```
+
+The deployer performs the following sequence:
+
+```text
+SSH architecture detection
+  -> build image for the detected CPU
+  -> docker save / podman save to build/*.tar
+  -> calculate SHA256
+  -> generate deployment .rsc
+  -> SCP TAR to RouterOS
+  -> SCP .rsc to RouterOS
+  -> create bridge/VETH/NAT/mounts/environment
+  -> /container/add file=<image.tar>
+  -> wait for extraction
+  -> start the container
+  -> optionally configure/start TZSP sniffer
+  -> print container status and logs
+```
+
+For SSH key authentication set:
+
+```dotenv
+ROUTER_IDENTITY_FILE=/home/user/.ssh/id_ed25519
+```
+
+---
+
+## Build RouterOS image without deployment
+
+ARM64:
+
+```bash
+./scripts/build-routeros.sh arm64
+```
+
+AMD64/x86_64:
+
+```bash
+./scripts/build-routeros.sh amd64
+```
+
+ARMv7/armhf:
+
+```bash
+./scripts/build-routeros.sh arm
+```
+
+Generated files are written to `build/`, for example:
+
+```text
+build/routeros-suricata-tzsp-arm64.tar
+build/routeros-suricata-tzsp-arm64.tar.sha256
+```
+
+---
+
+## Manual SCP and RouterOS import
+
+Example:
+
+```bash
+./scripts/build-routeros.sh arm64
+scp build/routeros-suricata-tzsp-arm64.tar admin@192.168.88.1:disk1/
+```
+
+RouterOS templates are located in `routeros/`:
+
+```text
+01-container-network.rsc
+02-sniffer-vlan100.rsc
+03-rest-and-firewall.rsc
+04-container-import-amd64.rsc
+04-container-import-arm64.rsc
+04-container-import-arm.rsc
+rollback.rsc
+```
+
+After extraction, inspect and start the container:
+
+```routeros
+/container/print detail
+/container/start suricata-ids
+/log/print where message~"suricata|TZSP|IDS"
+```
+
+---
+
+## Persistent data on RouterOS
+
+The deployment keeps application state outside the image root directory:
+
+```text
+<disk>/containers/suricata-ids-data  -> /data
+<disk>/containers/suricata-ids-logs  -> /var/log/suricata
+<disk>/containers/suricata-ids-rules -> /var/lib/suricata
+```
+
+This preserves SQLite data, logs, and downloaded Suricata rules when the application image is replaced.
+
+---
+
+## TZSP capture
+
+The default deployment configuration can configure RouterOS Packet Sniffer to stream VLAN traffic to the container address on UDP port `37008`.
+
+Relevant settings in `deploy-routeros.env`:
+
+```dotenv
+CONFIGURE_SNIFFER=true
+START_SNIFFER=true
+VLAN_ID=100
+```
+
+If the router already uses Packet Sniffer for another purpose, disable automatic sniffer configuration:
+
+```dotenv
+CONFIGURE_SNIFFER=false
+START_SNIFFER=false
+```
+
+Then configure the capture manually.
+
+Hardware-offloaded bridge traffic may require additional verification on the specific RouterOS device because some switched traffic can bypass software capture paths.
+
+---
+
+## TAP and Suricata
+
+Inside the container the application creates:
+
+```text
+suritap0
+```
+
+The pipeline is:
+
+```text
+TZSP datagram
+  -> Python decoder
+  -> Ethernet frame
+  -> TAP suritap0
+  -> Suricata
+  -> eve.json
+  -> Python EVE watcher
+  -> SQLite / Web UI
+```
+
+If the RouterOS container cannot create the TAP interface, the application will fail early with an error related to `/dev/net/tun`, `TUNSETIFF`, or permissions. This is the main platform-specific capability to validate on the target router.
+
+---
+
+## RouterOS REST reaction
+
+Automatic blocking is intentionally disabled by default:
+
+```dotenv
+AUTO_BLOCK=false
+```
+
+Observation mode should be used first. After validating alerts and false positives, the optional reaction engine can add selected addresses to a RouterOS firewall address list through REST.
+
+Relevant settings:
+
+```dotenv
+AUTO_BLOCK=true
+AUTO_BLOCK_MAX_SEVERITY=1
+ROUTEROS_URL=https://172.31.255.1
+ROUTEROS_USER=suricata-api
+ROUTEROS_PASSWORD=CHANGE_ME
+ROUTEROS_ADDRESS_LIST=IDS-BLOCK
+BLOCK_TIMEOUT=1h
+```
+
+Do not enable automatic blocking until the monitored networks, exclusion list, REST credentials, firewall rule placement, and alert policy have been reviewed.
+
+---
+
+## Important files
+
+```text
+Dockerfile
+Dockerfile / Debian slim runtime image
+
+dev.sh
+Local web-only development launcher
+
+requirements.txt
+Local Python development/test dependencies
+
+app/main.py
+Full RouterOS/container application entry point
+
+app/dev_web.py
+Local web-only entry point
+
+app/tzsp.py
+TZSP receiver and decoder
+
+app/tap.py
+TAP interface handling
+
+app/eve.py
+Suricata EVE JSON watcher
+
+app/policy.py
+Alert/blocking policy
+
+app/routeros.py
+RouterOS REST client
+
+app/store.py
+SQLite alert storage
+
+app/webui.py
+Dashboard and JSON API
+
+scripts/build-routeros.sh
+Build and save image to TAR
+
+scripts/deploy-routeros.sh
+Build, SCP, import, and start on RouterOS
+
+routeros/
+RouterOS configuration templates
+```
+
+---
+
+## Safety defaults
+
+The default configuration is observation-oriented:
+
+```dotenv
+AUTO_BLOCK=false
+UPDATE_RULES_ON_START=false
+ROUTEROS_PASSWORD=CHANGE_ME
+```
+
+Keep automatic firewall actions disabled until the capture path and alert quality are validated on the real network.