gru/routeros-suricata-tzsp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

worked poc

Browse Source
This commit is contained in:
Mateusz Gruszczyński
2026-08-14 11:33:01 +02:00
parent adfdb0b86c
commit fc3a2944b2
94 changed files with 2931 additions and 3412 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+313 -47
View File
@@ -1,6 +1,6 @@
# RouterOS TZSP + Suricata IDS
Project version: `0.3.2`
Project version: `0.5.3`
A lightweight IDS stack designed to run as a **single container on MikroTik RouterOS**.
RouterOS mirrors selected traffic with TZSP, the container decodes the frames into a TAP interface, Suricata analyzes them, and the Python service stores EVE alerts in SQLite and exposes a small web dashboard.
@@ -52,6 +52,249 @@ Docker Compose is only provided for Linux integration testing. RouterOS receives
---
## Quick start - Docker Compose on Linux
Use these steps for the local full-stack test with TZSP, TAP and Suricata.
### 1. Check prerequisites
You need:
- Linux,
- Docker Engine,
- Docker Compose v2 (`docker compose`),
- `/dev/net/tun` available on the host.
Check them:
```bash
docker --version
docker compose version
test -c /dev/net/tun && echo "TUN/TAP: OK" || echo "TUN/TAP: MISSING"
```
If `/dev/net/tun` is missing on Linux, try:
```bash
sudo modprobe tun
```
Then check `/dev/net/tun` again.
### 2. Create the runtime configuration
From the project directory:
```bash
cp .env.example .env
```
For a first test the defaults can be used. Before monitoring a real network, review at least:
```dotenv
SURICATA_HOME_NET=[192.168.0.0/16,10.0.0.0/8,172.16.0.0/12]
MONITORED_NETWORKS=192.168.100.0/24
ALERT_MAX_SEVERITY=2
ALERT_DEDUP_WINDOW_SECONDS=300
ADMIN_TOKEN=<long-random-token>
AUTO_BLOCK=false
```
Keep `AUTO_BLOCK=false` until alerts are verified.
### 3. Build and start
```bash
docker compose up -d --build
```
Check container status:
```bash
docker compose ps
```
Follow logs:
```bash
docker compose logs -f ids
```
You can also use the container name directly:
```bash
docker logs -f routeros-suricata-tzsp
```
Do not run plain `docker logs -f` - Docker requires a container name.
### 4. Run the end-to-end self-test
```bash
./scripts/selftest.sh
```
Expected result:
```text
SELFTEST OK: Suricata emitted marked TZSP pipeline test alert(s); UI filtering remains enabled
```
The self-test uses reserved SID `1000001` and a unique payload marker. Normal ICMP/ping traffic cannot match it, and SID `1000001` is filtered from SQLite/UI by default.
### 5. Open the dashboard
```text
http://127.0.0.1:8080
```
Status API:
```bash
curl http://127.0.0.1:8080/api/status
```
### 6. Stop the stack
```bash
docker compose down
```
### One-command first start
The helper script checks Docker, Compose and `/dev/net/tun`, creates `.env` if needed, starts the stack and runs the self-test:
```bash
./scripts/first-run.sh
```
If scripts are not executable after unpacking an archive:
```bash
chmod +x dev.sh scripts/*.sh
./scripts/first-run.sh
```
### Rebuild after updating older images
Version `0.3.2` could restart continuously with:
```text
chown: invalid user: 'suricata:suricata'
```
Version `0.3.3` added the `suricata` system account. Version `0.3.4` also fixes a second startup issue where the `suricata -T` configuration check could create root-owned `eve.json`, `fast.log`, and `stats.log`, causing the real Suricata process to fail with `Permission denied`. It also relocates the Unix command socket into `/run/suricata/`. Rebuild the image completely:
```bash
docker compose down
docker compose build --no-cache
docker compose up -d
docker compose ps
docker compose logs -f ids
```
---
## Alert tuning and false-positive control
Version `0.5.2` uses two tuning layers. Raw Suricata EVE stays on disk, while the incident database/UI defaults to severity `1-2` and collapses repeated identical SID/source/destination tuples for five minutes. This prevents informational events from flooding the dashboard without changing the raw sensor log.
```dotenv
ALERT_MAX_SEVERITY=2
ALERT_DEDUP_WINDOW_SECONDS=300
ALERT_IGNORE_SIDS=
ALERT_IGNORE_CATEGORIES=
```
For a known false positive, prefer sensor-level tuning in `/data/suricata/threshold.config`. The dashboard can add a full SID suppression with **Suppress SID**, or the file can be edited directly from the rule-management panel. Examples:
```text
suppress gen_id 1, sig_id 1234567
threshold gen_id 1, sig_id 1234567, type limit, track by_src, count 1, seconds 300
```
Do not globally suppress a rule just because it fired once. First verify the SID, endpoint, direction and expected application behavior.
---
## Adding new Suricata detections
Suricata is signature/rule driven; it does not need model training to learn a new network detection. Add environment-specific signatures to `/data/suricata/custom.rules`, or use **Custom Suricata signatures** in the dashboard. Save performs a `suricata -T` validation first and only then writes the file and requests a live rule reload. Runtime loads persisted `/data/suricata/*.rules`, so additional rule files can be placed beside `custom.rules` without rebuilding the image.
Example local rule:
```text
alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"LOCAL suspicious URI marker"; flow:established,to_server; http.uri; content:"/admin/export"; nocase; classtype:web-application-activity; priority:2; sid:1000100; rev:1;)
```
Use unique local SIDs. SID `1000001` is reserved for the marked pipeline self-test, built-in production detections use `1000101-1000108`, and site-specific rules should use `1001000+`.
Vendor rules are managed with `suricata-update`. A baseline ET/Open ruleset and a current OISF source index are baked into the image. Docker Compose and RouterOS deployments persist `/var/lib/suricata`, including enabled source definitions, downloaded feeds and the source index. An empty first-run mount is seeded from the image baseline. `scripts/update-rules.sh` applies persisted `/data/suricata/disable.conf`, `enable.conf`, and `modify.conf`.
The **Rules** page now has a **Signature sources** table backed by the official OISF `suricata-update` catalog. The UI lists free sources, shows vendor/license/tags/status, refreshes the OISF index, enables or disables parameter-free feeds, and downloads all active feeds on demand. ET/Open remains the default source and cannot be accidentally disabled from the panel. Feeds that require credentials or parameters are displayed but must be configured manually instead of prompting through the web UI.
Every feed update is transactional at the merged-rules level: the existing `suricata.rules` is backed up, new signatures are downloaded, the complete Suricata configuration is tested with `suricata -T`, and only a validated ruleset is kept. If download or validation fails, the previous known-good rules are restored. The periodic updater uses the same active-source set and runs every `RULE_UPDATE_INTERVAL_HOURS` when the interval is greater than zero.
```dotenv
UPDATE_RULES_ON_START=false
RULE_UPDATE_INTERVAL_HOURS=24
```
---
## Built-in production detections
The image now ships with a conservative local baseline in addition to the ET/Open snapshot baked by `suricata-update`. The local baseline is intentionally rate-limited so one packet does not create an incident. It covers repeated SSH, RDP and WinBox connection attempts, high-rate SYN scanning, ICMP sweeps, external SMB access, unusually long DNS labels and outbound Telnet.
The baseline uses SIDs `1000101-1000108`. The deterministic pipeline self-test remains SID `1000001`, but it only matches the exact payload marker generated by `scripts/send_test_tzsp.py` and is ignored by the incident database. This keeps the end-to-end test available without turning ordinary ping traffic into alerts.
ET/Open is still the main vendor signature source. `suricata-update` is the supported manager for refreshing it; the image seeds the persistent `/var/lib/suricata` volume on first start.
---
## Upload a ready RouterOS image without deploying it
The upload helper requires an already-built TAR and does exactly one job:
```bash
cp deploy-routeros.env.example deploy-routeros.env
./scripts/upload-routeros-image.sh build/routeros-suricata-tzsp-arm64.tar
```
It performs SCP upload plus a read-only file-list verification. It does **not** detect architecture, build an image, run `/container/add`, import an `.rsc`, change RouterOS configuration, or start a container.
---
## Database, storage and maintenance
The dashboard detects SQLite and persistent storage separately. It shows DB path, schema version, row count, DB/WAL size, filesystem usage and Suricata log size. SQLite uses WAL mode and performs a small schema migration automatically when upgrading from older project versions.
Administrative actions are disabled until `ADMIN_TOKEN` is set. The dashboard then provides:
- clear stored alert incidents,
- truncate active Suricata `eve.json`, `fast.log`, `stats.log`, and `suricata.log`,
- compact SQLite with `VACUUM`,
- reset runtime counters,
- validate/save/reload custom rules and `threshold.config`,
- refresh the official OISF source catalog, enable/disable supported free feeds, and download/update the active vendor rulesets.
Set a long random admin token and keep port `8080` on a management-only network. The UI does not provide TLS termination.
---
## Extended statistics
The dashboard reports alert hits vs deduplicated incidents, 1h/24h activity, top signatures, top sources, severity distribution data, filter/dedup/error counters, block attempts/results and the latest Suricata EVE `stats` counters such as decoder/capture/drop values when emitted by the installed Suricata configuration.
---
## Dashboard sections
The web UI is split into top-menu sections: **Overview**, **Incidents**, **Statistics**, **System**, **Rules**, and **Maintenance**. Incident timestamps are stored in UTC and rendered in the browser's local timezone. Repeated events are aggregated by SID, source, destination, protocol and destination port within the configured deduplication window.
---
## Local Web UI development without Docker
The web dashboard can be started locally without Docker, Suricata, TAP, `/dev/net/tun`, or root privileges.
@@ -100,14 +343,13 @@ curl http://127.0.0.1:8080/api/status
The response includes:
- overall application status and uptime,
- Web UI/API status,
- TZSP receiver status,
- TAP interface status,
- Suricata process status and PID,
- EVE JSON watcher status,
- RouterOS REST integration status,
- listening/outbound port information,
- runtime packet, alert, and block counters.
- Web UI/API, TZSP, TAP, Suricata and EVE watcher state,
- SQLite existence/path/size/WAL/schema/row count,
- persistent filesystem usage and Suricata log size,
- managed custom-rule and threshold/suppression status,
- RouterOS REST integration status and ports,
- runtime packet/error/filter/dedup/block counters,
- the latest numeric Suricata EVE `stats` counters.
`GET /api/health` is kept as a compatibility alias and returns the same status payload.
@@ -185,12 +427,7 @@ Send the built-in TZSP test packet:
./scripts/selftest.sh
```
The test rule should generate:
```text
LOCAL TZSP PIPELINE TEST
SID 1000001
```
The test rule should generate raw EVE alert SID `1000001` with signature `LOCAL TEST TZSP PIPELINE MARKER`. It is intentionally filtered from the incident database/UI.
Stop the stack:
@@ -231,43 +468,28 @@ Create the deployment configuration:
cp deploy-routeros.env.example deploy-routeros.env
```
Edit at least:
```dotenv
ROUTER_HOST=192.168.88.1
ROUTER_USER=admin
ROUTER_ARCH=auto
ROUTER_DISK=disk1
ROUTER_SCP_DIR=disk1
VLAN_ID=100
MONITORED_NETWORKS=192.168.100.0/24
AUTO_BLOCK=false
```
Deploy:
Upload a ready image first:
```bash
./scripts/deploy-routeros.sh
./scripts/upload-routeros-image.sh build/routeros-suricata-tzsp-arm64.tar
```
The deployer performs the following sequence:
Then deploy by giving the **RouterOS-side TAR path** directly:
```bash
./scripts/deploy-routeros.sh routeros-suricata-tzsp-arm64.tar
```
The deployer no longer builds, detects image architecture, renames, or re-uploads the image. For project version `0.5.3` it creates:
```text
SSH architecture detection
-> build image for the detected CPU
-> docker save / podman save to build/*.tar
-> calculate SHA256
-> generate deployment .rsc
-> SCP TAR to RouterOS
-> SCP .rsc to RouterOS
-> create bridge/VETH/NAT/mounts/environment
-> /container/add file=<image.tar>
-> wait for extraction
-> start the container
-> optionally configure/start TZSP sniffer
-> print container status and logs
name=suricata_0.5.3
file=routeros-suricata-tzsp-arm64.tar
root-dir=/containers/suricata_0.5.3/root
```
The remaining deployment work is unchanged: bridge/VETH/NAT, environment, persistent mounts, optional RouterOS REST user/firewall integration, TZSP sniffer configuration, image extraction wait, container start, and final status. Existing containers are not removed. Re-running deployment for the same version stops with `Container suricata_0.5.3 already exists`.
For SSH key authentication set:
```dotenv
@@ -311,7 +533,7 @@ Example:
```bash
./scripts/build-routeros.sh arm64
scp build/routeros-suricata-tzsp-arm64.tar admin@192.168.88.1:disk1/
scp build/routeros-suricata-tzsp-arm64.tar admin@192.168.88.1:/
```
RouterOS templates are located in `routeros/`:
@@ -346,7 +568,7 @@ The deployment keeps application state outside the image root directory:
<disk>/containers/suricata-ids-rules -> /var/lib/suricata
```
This preserves SQLite data, logs, and downloaded Suricata rules when the application image is replaced.
This preserves SQLite data, custom signatures, threshold/suppression configuration, Suricata-update filters, raw logs, and downloaded vendor rules when the application image is replaced.
---
@@ -456,6 +678,15 @@ Suricata EVE JSON watcher
app/policy.py
Alert/blocking policy
app/tuning.py
Second-stage alert severity/category/SID filter
app/rules.py
Validated custom rules, suppressions and vendor-rule updates
app/maintenance.py
Storage detection and safe maintenance helpers
app/routeros.py
RouterOS REST client
@@ -483,8 +714,43 @@ The default configuration is observation-oriented:
```dotenv
AUTO_BLOCK=false
ALERT_MAX_SEVERITY=2
UPDATE_RULES_ON_START=false
ROUTEROS_PASSWORD=CHANGE_ME
ADMIN_TOKEN=
```
Keep automatic firewall actions disabled until the capture path and alert quality are validated on the real network.
Keep automatic firewall actions disabled until the capture path and alert quality are validated on the real network. Set `ADMIN_TOKEN` before enabling dashboard maintenance/rule-management actions, and restrict the Web UI to a trusted management network.
---
## Image-only upgrade on an already configured RouterOS
After the first deployment, when `veth-ids`, bridge/NAT, TZSP sniffer, `IDS_ENV` and `IDS_MOUNTS` already exist, do not run the full deploy just to change the image.
1. Upload the ready TAR only:
```bash
./scripts/upload-routeros-image.sh build/routeros-suricata-tzsp-arm64.tar
```
2. Swap the Suricata container only:
```bash
./scripts/upgrade-routeros-container.sh routeros-suricata-tzsp-arm64.tar
```
For version `0.5.3` the second command creates:
```text
name=suricata_0.5.3
file=routeros-suricata-tzsp-arm64.tar
root-dir=/containers/suricata_0.5.3/root
interface=veth-ids
envlist=IDS_ENV
mountlists=IDS_MOUNTS
```
The upgrade helper does not create or modify the bridge, IP addresses, NAT, veth, TZSP/sniffer, firewall, REST user, envlist definitions, or mount definitions. It disables `start-on-boot` on older `suricata_*` containers, stops the running old Suricata container, creates the new versioned container, waits for image extraction, and starts it. Older containers are kept stopped for rollback.
Persistent `/data`, Suricata logs and vendor rules continue to use the existing mount list, so they survive the version change.