routeros-suricata-tzsp/README.md

# RouterOS TZSP + Suricata IDS

Project version: `0.3.2`

A lightweight IDS stack designed to run as a **single container on MikroTik RouterOS**.
RouterOS mirrors selected traffic with TZSP, the container decodes the frames into a TAP interface, Suricata analyzes them, and the Python service stores EVE alerts in SQLite and exposes a small web dashboard.

## Architecture

```text
VLAN / RouterOS traffic
        |
        v
RouterOS Packet Sniffer
        |
        | TZSP UDP/37008
        v
single RouterOS container
  Debian slim
  + Python TZSP receiver
  + TAP suritap0
  + Suricata IDS
  + EVE JSON watcher
  + SQLite
  + Web UI :8080
  + optional RouterOS REST blocking
```

The RouterOS deployment workflow is:

```text
Docker/Podman build host
        |
        | build for RouterOS CPU architecture
        v
container image
        |
        | docker save / podman save
        v
*.tar
        |
        | SCP
        v
RouterOS external disk
        |
        | /container/add file=...
        v
running IDS container
```

Docker Compose is only provided for Linux integration testing. RouterOS receives one saved container image as a TAR archive.

---

## Local Web UI development without Docker

The web dashboard can be started locally without Docker, Suricata, TAP, `/dev/net/tun`, or root privileges.

Requirements:

- Linux or macOS
- Python 3
- Python `venv` support

Start the development dashboard:

```bash
./dev.sh
```

The script will:

1. create `.venv/` if needed,
2. install dependencies from `requirements.txt`,
3. create `data/dev/`,
4. start the web-only application.

Open:

```text
http://127.0.0.1:8080
```

This mode intentionally does **not** start Suricata, TZSP capture, TAP, or RouterOS integration.

### Status API

The dashboard and external monitoring systems can use:

```text
GET /api/status
```

Example:

```bash
curl http://127.0.0.1:8080/api/status
```

The response includes:

- overall application status and uptime,
- Web UI/API status,
- TZSP receiver status,
- TAP interface status,
- Suricata process status and PID,
- EVE JSON watcher status,
- RouterOS REST integration status,
- listening/outbound port information,
- runtime packet, alert, and block counters.

`GET /api/health` is kept as a compatibility alias and returns the same status payload.

### Add a sample alert

To start the dashboard with one demo alert in an empty development database:

```bash
DEV_SEED_DATA=true ./dev.sh
```

Development database:

```text
data/dev/ids.db
```

To use another port:

```bash
WEB_PORT=8090 ./dev.sh
```

To listen on all local interfaces:

```bash
WEB_BIND=0.0.0.0 ./dev.sh
```

Do not expose the development server directly to an untrusted network.

---

## Local tests

Install dependencies and run tests:

```bash
./dev.sh --test
```

Or run tests before starting the dashboard:

```bash
RUN_TESTS=true ./dev.sh
```

The project currently uses only the Python standard library, including `unittest` for tests. `requirements.txt` is kept as the canonical dependency file so future packages can be installed automatically by `dev.sh`.

---

## Docker integration test

Prepare local environment configuration:

```bash
cp .env.example .env
```

Build and start:

```bash
docker compose up -d --build
```

Dashboard:

```text
http://127.0.0.1:8080
```

Send the built-in TZSP test packet:

```bash
./scripts/selftest.sh
```

The test rule should generate:

```text
LOCAL TZSP PIPELINE TEST
SID 1000001
```

Stop the stack:

```bash
docker compose down
```

---

## RouterOS requirements

Before deployment, verify that the router has:

- a RouterOS version and package set that supports containers,
- the `container` package installed,
- container and sniffer functionality enabled in device mode,
- SSH access from the build machine,
- enough storage for the image, container root directory, Suricata rules, logs, and SQLite database,
- preferably an external disk instead of small internal flash storage.

Useful RouterOS checks:

```routeros
/system/resource/print
/system/package/print
/system/device-mode/print
/disk/print
/container/print
```

---

## Automated RouterOS deployment

Create the deployment configuration:

```bash
cp deploy-routeros.env.example deploy-routeros.env
```

Edit at least:

```dotenv
ROUTER_HOST=192.168.88.1
ROUTER_USER=admin
ROUTER_ARCH=auto
ROUTER_DISK=disk1
ROUTER_SCP_DIR=disk1
VLAN_ID=100
MONITORED_NETWORKS=192.168.100.0/24
AUTO_BLOCK=false
```

Deploy:

```bash
./scripts/deploy-routeros.sh
```

The deployer performs the following sequence:

```text
SSH architecture detection
  -> build image for the detected CPU
  -> docker save / podman save to build/*.tar
  -> calculate SHA256
  -> generate deployment .rsc
  -> SCP TAR to RouterOS
  -> SCP .rsc to RouterOS
  -> create bridge/VETH/NAT/mounts/environment
  -> /container/add file=<image.tar>
  -> wait for extraction
  -> start the container
  -> optionally configure/start TZSP sniffer
  -> print container status and logs
```

For SSH key authentication set:

```dotenv
ROUTER_IDENTITY_FILE=/home/user/.ssh/id_ed25519
```

---

## Build RouterOS image without deployment

ARM64:

```bash
./scripts/build-routeros.sh arm64
```

AMD64/x86_64:

```bash
./scripts/build-routeros.sh amd64
```

ARMv7/armhf:

```bash
./scripts/build-routeros.sh arm
```

Generated files are written to `build/`, for example:

```text
build/routeros-suricata-tzsp-arm64.tar
build/routeros-suricata-tzsp-arm64.tar.sha256
```

---

## Manual SCP and RouterOS import

Example:

```bash
./scripts/build-routeros.sh arm64
scp build/routeros-suricata-tzsp-arm64.tar admin@192.168.88.1:disk1/
```

RouterOS templates are located in `routeros/`:

```text
01-container-network.rsc
02-sniffer-vlan100.rsc
03-rest-and-firewall.rsc
04-container-import-amd64.rsc
04-container-import-arm64.rsc
04-container-import-arm.rsc
rollback.rsc
```

After extraction, inspect and start the container:

```routeros
/container/print detail
/container/start suricata-ids
/log/print where message~"suricata|TZSP|IDS"
```

---

## Persistent data on RouterOS

The deployment keeps application state outside the image root directory:

```text
<disk>/containers/suricata-ids-data  -> /data
<disk>/containers/suricata-ids-logs  -> /var/log/suricata
<disk>/containers/suricata-ids-rules -> /var/lib/suricata
```

This preserves SQLite data, logs, and downloaded Suricata rules when the application image is replaced.

---

## TZSP capture

The default deployment configuration can configure RouterOS Packet Sniffer to stream VLAN traffic to the container address on UDP port `37008`.

Relevant settings in `deploy-routeros.env`:

```dotenv
CONFIGURE_SNIFFER=true
START_SNIFFER=true
VLAN_ID=100
```

If the router already uses Packet Sniffer for another purpose, disable automatic sniffer configuration:

```dotenv
CONFIGURE_SNIFFER=false
START_SNIFFER=false
```

Then configure the capture manually.

Hardware-offloaded bridge traffic may require additional verification on the specific RouterOS device because some switched traffic can bypass software capture paths.

---

## TAP and Suricata

Inside the container the application creates:

```text
suritap0
```

The pipeline is:

```text
TZSP datagram
  -> Python decoder
  -> Ethernet frame
  -> TAP suritap0
  -> Suricata
  -> eve.json
  -> Python EVE watcher
  -> SQLite / Web UI
```

If the RouterOS container cannot create the TAP interface, the application will fail early with an error related to `/dev/net/tun`, `TUNSETIFF`, or permissions. This is the main platform-specific capability to validate on the target router.

---

## RouterOS REST reaction

Automatic blocking is intentionally disabled by default:

```dotenv
AUTO_BLOCK=false
```

Observation mode should be used first. After validating alerts and false positives, the optional reaction engine can add selected addresses to a RouterOS firewall address list through REST.

Relevant settings:

```dotenv
AUTO_BLOCK=true
AUTO_BLOCK_MAX_SEVERITY=1
ROUTEROS_URL=https://172.31.255.1
ROUTEROS_USER=suricata-api
ROUTEROS_PASSWORD=CHANGE_ME
ROUTEROS_ADDRESS_LIST=IDS-BLOCK
BLOCK_TIMEOUT=1h
```

Do not enable automatic blocking until the monitored networks, exclusion list, REST credentials, firewall rule placement, and alert policy have been reviewed.

---

## Important files

```text
Dockerfile
Dockerfile / Debian slim runtime image

dev.sh
Local web-only development launcher

requirements.txt
Local Python development/test dependencies

app/main.py
Full RouterOS/container application entry point

app/dev_web.py
Local web-only entry point

app/tzsp.py
TZSP receiver and decoder

app/tap.py
TAP interface handling

app/eve.py
Suricata EVE JSON watcher

app/policy.py
Alert/blocking policy

app/routeros.py
RouterOS REST client

app/store.py
SQLite alert storage

app/webui.py
Dashboard and JSON API

scripts/build-routeros.sh
Build and save image to TAR

scripts/deploy-routeros.sh
Build, SCP, import, and start on RouterOS

routeros/
RouterOS configuration templates
```

---

## Safety defaults

The default configuration is observation-oriented:

```dotenv
AUTO_BLOCK=false
UPDATE_RULES_ON_START=false
ROUTEROS_PASSWORD=CHANGE_ME
```

Keep automatic firewall actions disabled until the capture path and alert quality are validated on the real network.