gru/routeros-suricata-tzsp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

poc4 wit rust

Browse Source
This commit is contained in:
Mateusz Gruszczyński
2026-08-16 15:34:53 +02:00
parent e5d344622e
commit 40474cdc59
29 changed files with 2692 additions and 159 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+126 -43
View File
@@ -1,6 +1,37 @@
# MikroSuricata
Project version: `0.9.1`
Project version: `0.11.2`
## What changed in 0.11.2
- Replaced full-traffic RouterOS Packet Sniffer capture with a **hybrid TZSP path**: routed IPv4 uses `/ip firewall mangle action=sniff-tzsp`, while Packet Sniffer streams only `filter-mac-protocol=!ip` traffic such as ARP, IPv6, LLDP and PPPoE.
- The capture deploy no longer assumes a production bridge, VLAN or interface name. `TZSP_L2_INTERFACE=` defaults to RouterOS `filter-interface=all`; set it only when a site intentionally wants to constrain the low-volume non-IPv4 stream.
- Packet Sniffer TZSP destination now tries the current `streaming-server=IP:PORT` syntax first and automatically falls back to legacy split `streaming-server`/`streaming-port` syntax.
- The project-owned IPv4 mangle rule is identified by the exact comment `MikroSuricata TZSP IPv4`; deployment replaces only that rule and leaves unrelated mangle rules untouched. RouterOS `sniff-tzsp` continues processing subsequent mangle rules after cloning the packet.
- Added `scripts/configure-routeros-tzsp-hybrid.sh` to migrate an existing installation without rebuilding the container, including `DRY_RUN=true` preview mode.
- Replaced the VLAN-specific RouterOS template with `routeros/02-tzsp-hybrid.rsc` and added detailed capture coverage, limitations and migration documentation in `docs/routeros-tzsp-capture.md`.
## What changed in 0.11.0
- Reworked the Rust TZSP data-plane for RouterOS containers that cannot raise kernel `rmem_max`: UDP receive and TZSP decode/TAP injection now run on **separate threads**.
- The RX thread does only blocking `recvmmsg(MSG_WAITFORONE)` into reusable preallocated batches. It never waits for TAP, Suricata, Python, Redis or the UI.
- Added a bounded **userspace packet queue** (`TZSP_QUEUE_MB`, default 64 MiB). With the default 256-packet batch and 12 KiB slots this provides roughly half a second of burst absorption at 1 Gbit/s even when the kernel socket is limited to ~416 KiB.
- If downstream processing falls behind after the userspace queue fills, the RX thread keeps draining UDP into a scratch batch and records explicit `queue_dropped_datagrams` instead of silently turning all loss into kernel UDP drops.
- Increased the default receive batch from 64 to **256** and made the per-datagram buffer configurable (`TZSP_DATAGRAM_BYTES`, default 12288; up to 65535). Oversized/truncated datagrams are counted explicitly.
- Added live capture diagnostics: TZSP ingress rate, inspected rate, capture efficiency, userspace queue fill, kernel drops/s, queue drops/s, truncated packets and RX/TAP worker health.
- The overview now labels the main value as **Inspected throughput** and separately shows TZSP ingress, so a RouterOS mirror limitation can be distinguished from a Suricata/TAP bottleneck immediately.
- Prometheus exports the new Rust queue/capture-health gauges and drop counters.
## What changed in 0.10.0
- Moved the complete TZSP packet hot-path out of Python into a dedicated **Rust data-plane process**. Rust now owns batched UDP receive (`recvmmsg`), TZSP decoding, direction accounting and TAP injection; packet bytes no longer cross into Python.
- Added configurable UDP receive buffering and batched receive. In 0.11.0 the hot path was further split into dedicated RX/TAP threads with a userspace queue.
- Added 1 Hz Unix-socket telemetry from Rust to the Python control plane. The dashboard now receives live throughput independently from retained Redis history, so **Throughput now** is current instead of reusing a stale historical payload.
- Added receiver observability to the UI/health/Prometheus path: actual receive-buffer size, Rust PID, telemetry age and kernel UDP drop counter.
- Fixed the overview refresh path so throughput history is refreshed during periodic UI refreshes, not only at initial load/range changes.
- The old `FORENSIC_PCAP_MODE=blocks` pre-event ring required Python to process every frame. With the isolated Rust data-plane it is treated as a legacy setting and safely falls back to Suricata alert PCAP capture. The new default is `alerts`.
- Docker now compiles and tests the Rust receiver in a dedicated build stage and copies only the stripped receiver binary into the runtime image. The existing RouterOS amd64/arm64/armv7 image workflow is preserved.
## What changed in 0.9.1
@@ -70,7 +101,7 @@ A lightweight IDS stack designed to run as a **single container on MikroTik Rout
- Security analytics add anomalies, NXDOMAIN counts, encrypted/cleartext session counts, local/remote endpoint inventory, passive IP/MAC asset observations, top signatures/severities and JA4/JA3/HASSH fingerprint inventory.
- The built-in local ruleset adds NXDOMAIN/DNS-rate signals, outbound SMB/SMTP/FTP policy detections, exposed database-service probes and an internal administrative/lateral-movement burst detector.
RouterOS mirrors selected traffic with TZSP, the container decodes the frames into a TAP interface, Suricata analyzes them, and the Python service stores EVE alerts in SQLite and exposes a small web dashboard.
RouterOS mirrors traffic with a hybrid TZSP path: routed IPv4 is cloned by an IPv4 mangle `sniff-tzsp` rule, while Packet Sniffer handles non-IPv4 Ethernet traffic. A dedicated Rust process decodes and injects the frames into TAP, Suricata analyzes them, and the Python control plane handles EVE/SQLite/NDR/API/UI without sitting in the packet hot-path.
## NDR / MikroTik-specific options
@@ -93,28 +124,39 @@ All NDR state, IOC data, Redis persistence, Suricata logs/rules and forensic PCA
## Architecture
```text
VLAN / RouterOS traffic
|
v
RouterOS Packet Sniffer
|
| TZSP UDP/37008
v
RouterOS routed IPv4 ----> mangle sniff-tzsp ---------+
| TZSP UDP/37008
RouterOS non-IPv4 -----> Packet Sniffer (!ip) --------+
v
single RouterOS container
Debian slim
+ Python TZSP receiver
+ Rust TZSP data-plane (recvmmsg -> TZSP -> TAP)
+ TAP suritap0
+ Suricata IDS
+ EVE JSON watcher
+ Python control plane / EVE JSON watcher
+ SQLite alerts / assets / NDR incidents / sessions
+ MikroSuricata behavior + correlation engine
+ local IOC datasets (IP/domain/SHA256/JA3/JA4/HASSH)
+ Redis/RAM bounded traffic history
+ WebSocket live stream
+ Redis traffic history
+ Rust -> Python 1 Hz Unix telemetry
+ WebSocket live throughput / event stream
+ Web UI :8080
+ optional RouterOS REST blocking
```
The capture path is intentionally independent from the UI/control plane:
```text
RouterOS TZSP UDP
|
v
Rust receiver -- recvmmsg() --> TZSP decode --> TAP write --> Suricata
|
+-- 1 Hz counters only --> Unix datagram --> Python --> Redis / WebSocket / Prometheus
```
If Redis, the browser or an analytics request is slow, it cannot block UDP receive/TAP injection.
The RouterOS deployment workflow is:
```text
@@ -342,7 +384,7 @@ Every feed update is transactional at the merged-rules level: the existing `suri
```dotenv
UPDATE_RULES_ON_START=false
RULE_UPDATE_INTERVAL_HOURS=24
FORENSIC_PCAP_MODE=blocks
FORENSIC_PCAP_MODE=alerts
FORENSIC_PCAP_WINDOW_SECONDS=60
FORENSIC_PCAP_MEMORY_MB=64
FORENSIC_PCAP_MAX_FILES=32
@@ -584,15 +626,15 @@ Then deploy by giving the **RouterOS-side TAR path** directly:
./scripts/deploy-routeros.sh routeros-suricata-tzsp-arm64.tar
```
The deployer no longer builds, detects image architecture, renames, or re-uploads the image. For project version `0.9.0` it creates:
The deployer no longer builds, detects image architecture, renames, or re-uploads the image. For project version `0.11.2` it creates:
```text
name=suricata_0.9.0
name=suricata_0.11.2
file=routeros-suricata-tzsp-arm64.tar
root-dir=/containers/suricata_0.9.0/root
root-dir=/containers/suricata_0.11.2/root
```
The remaining deployment work is unchanged: bridge/VETH/NAT, environment, persistent mounts, optional RouterOS REST user/firewall integration, TZSP sniffer configuration, image extraction wait, container start, and final status. Existing containers are not removed. Re-running deployment for the same version stops with `Container suricata_0.9.0 already exists`.
The remaining deployment work is unchanged: private container bridge/VETH/NAT, environment, persistent mounts, optional RouterOS REST user/firewall integration, hybrid TZSP capture configuration, image extraction wait, container start, and final status. Existing containers are not removed. Re-running deployment for the same version stops with `Container suricata_0.11.2 already exists`.
For SSH key authentication set:
@@ -644,7 +686,7 @@ RouterOS templates are located in `routeros/`:
```text
01-container-network.rsc
02-sniffer-vlan100.rsc
02-tzsp-hybrid.rsc
03-rest-and-firewall.rsc
04-container-import-amd64.rsc
04-container-import-arm64.rsc
@@ -686,26 +728,50 @@ Replacing or restarting the application container does not remove any of these f
## TZSP capture
The default deployment configuration can configure RouterOS Packet Sniffer to stream VLAN traffic to the container address on UDP port `37008`.
The default RouterOS deployment uses a hybrid capture model:
Relevant settings in `deploy-routeros.env`:
```dotenv
CONFIGURE_SNIFFER=true
START_SNIFFER=true
VLAN_ID=100
```text
routed IPv4 -> /ip firewall mangle action=sniff-tzsp
non-IPv4 L2 -> /tool sniffer filter-mac-protocol=!ip
\_______________________________/
TZSP UDP/37008
```
If the router already uses Packet Sniffer for another purpose, disable automatic sniffer configuration:
Relevant `deploy-routeros.env` settings:
```dotenv
CONFIGURE_SNIFFER=false
START_SNIFFER=false
TZSP_PORT=37008
CONFIGURE_TZSP_CAPTURE=true
CONFIGURE_IPV4_MANGLE=true
CONFIGURE_L2_SNIFFER=true
START_L2_SNIFFER=true
TZSP_L2_INTERFACE=
TZSP_L2_MAC_PROTOCOL=!ip
```
Then configure the capture manually.
No production bridge/VLAN name is assumed. Empty `TZSP_L2_INTERFACE` maps to RouterOS `filter-interface=all`. To intentionally constrain non-IPv4 capture, set an exact interface name, for example `TZSP_L2_INTERFACE=bridge-core`.
Hardware-offloaded bridge traffic may require additional verification on the specific RouterOS device because some switched traffic can bypass software capture paths.
The IPv4 rule is added to `chain=forward` and is owned by the comment `MikroSuricata TZSP IPv4`. Only that project-owned rule is replaced; unrelated mangle rules are left intact. `sniff-tzsp` clones the packet and then continues to subsequent mangle rules.
The Packet Sniffer complement uses `filter-mac-protocol=!ip`. RouterOS defines `ip` as IPv4 EtherType `0x0800` and `ipv6` separately as `0x86DD`, so this stream includes ARP, IPv6, LLDP, PPPoE and other non-IPv4 Ethernet protocols. `filter-stream=yes` prevents the sniffer's own TZSP stream from being recaptured.
Current RouterOS builds may expose the target as `streaming-server=IP:PORT`; older builds/documentation may expose `streaming-server` plus `streaming-port`. Deployment tries the combined form first and falls back automatically.
To migrate an already running installation without changing the container image:
```bash
./scripts/configure-routeros-tzsp-hybrid.sh
```
Preview without modifying RouterOS:
```bash
DRY_RUN=true ./scripts/configure-routeros-tzsp-hybrid.sh
```
Packet Sniffer is global RouterOS state. If another administrative workflow owns `/tool/sniffer`, use `CONFIGURE_L2_SNIFFER=false` and manage the low-volume L2 complement separately. Hardware-offloaded bridge-only traffic may also be invisible to Packet Sniffer on some devices.
Full coverage details, caveats, migration behavior and verification commands are in [`docs/routeros-tzsp-capture.md`](docs/routeros-tzsp-capture.md).
---
@@ -721,15 +787,29 @@ The pipeline is:
```text
TZSP datagram
-> Python decoder
-> Rust batched UDP receiver
-> Rust TZSP decoder
-> Ethernet frame
-> TAP suritap0
-> Suricata
-> eve.json
-> Python EVE watcher
-> Python control plane / EVE watcher
-> SQLite / Web UI
```
The Rust receiver emits only a compact 1 Hz telemetry sample to Python. That sample drives the live throughput card and is persisted asynchronously for historical throughput charts.
Useful receive-path tuning:
```dotenv
TZSP_RCVBUF_BYTES=33554432
TZSP_BATCH_SIZE=256
TZSP_QUEUE_MB=64
TZSP_DATAGRAM_BYTES=12288
```
On RouterOS the kernel may cap the actual socket buffer around a few hundred KiB regardless of `TZSP_RCVBUF_BYTES`. The 0.11.0 receiver therefore does not rely on sysctl: a dedicated RX thread drains UDP into the preallocated userspace queue. The UI exposes both kernel drops and userspace queue drops, plus TZSP ingress versus traffic actually injected into Suricata.
If the RouterOS container cannot create the TAP interface, the application will fail early with an error related to `/dev/net/tun`, `TUNSETIFF`, or permissions. This is the main platform-specific capability to validate on the target router.
---
@@ -778,11 +858,14 @@ Full RouterOS/container application entry point
app/dev_web.py
Local web-only entry point
app/tzsp.py
TZSP receiver and decoder
rust/tzsp-receiver/
Production Rust TZSP receiver: batched UDP receive, decoder, counters and TAP injection
app/tap.py
TAP interface handling
app/tzsp_rust.py
Rust process supervision plus 1 Hz telemetry bridge; no packet bytes enter Python
app/tzsp.py / app/tap.py
Legacy decoder/TAP helpers retained for focused tests; not used by the production packet path
app/eve.py
Suricata EVE JSON watcher
@@ -862,7 +945,7 @@ The default configuration is observation-oriented:
AUTO_BLOCK=false
ALERT_MAX_SEVERITY=2
UPDATE_RULES_ON_START=false
FORENSIC_PCAP_MODE=blocks
FORENSIC_PCAP_MODE=alerts
ROUTEROS_PASSWORD=CHANGE_ME
ADMIN_USERNAME=admin
ADMIN_PASSWORD=
@@ -875,7 +958,7 @@ Keep automatic firewall actions disabled until the capture path and alert qualit
## Image-only upgrade on an already configured RouterOS
After the first deployment, when `veth-ids`, bridge/NAT, TZSP sniffer, `IDS_ENV` and `IDS_MOUNTS` already exist, do not run the full deploy just to change the image.
After the first deployment, when the VETH/private bridge/NAT, hybrid TZSP capture, `IDS_ENV` and `IDS_MOUNTS` already exist, do not run the full deploy just to change the image.
1. Upload the ready TAR only:
@@ -889,18 +972,18 @@ After the first deployment, when `veth-ids`, bridge/NAT, TZSP sniffer, `IDS_ENV`
./scripts/upgrade-routeros-container.sh routeros-suricata-tzsp-arm64.tar
```
For version `0.9.0` the second command creates:
For version `0.11.2` the second command creates:
```text
name=suricata_0.9.0
name=suricata_0.11.2
file=routeros-suricata-tzsp-arm64.tar
root-dir=/containers/suricata_0.9.0/root
root-dir=/containers/suricata_0.11.2/root
interface=veth-ids
envlist=IDS_ENV
mountlists=IDS_MOUNTS
```
The upgrade helper does not modify the bridge, IP addresses, NAT, veth, TZSP/sniffer, firewall, REST user or envlist definitions. It stops older `suricata_*` containers, normalizes `IDS_MOUNTS` to the single `<disk>/containers/suricata-data -> /data` mapping, creates the new versioned container, waits for image extraction and starts it. Older containers are kept stopped for rollback.
The upgrade helper does not modify the bridge, IP addresses, NAT, VETH, hybrid TZSP capture, firewall, REST user or envlist definitions. It stops older `suricata_*` containers, normalizes `IDS_MOUNTS` to the single `<disk>/containers/suricata-data -> /data` mapping, creates the new versioned container, waits for image extraction and starts it. Older containers are kept stopped for rollback.
All new mutable state is written below `/data`, so subsequent image upgrades need only that one persistent mount.