routeros-suricata-tzsp/README.md

# MikroSuricata

Project version: `0.11.3`


## What changed in 0.11.3

- Reworked traffic retention so **Redis is only a short-lived ingestion buffer**. A dedicated archive worker commits normalized EVE events and throughput samples to SQLite and removes them from Redis only after the SQLite transaction succeeds.
- Dashboard analytics and chart snapshots are now calculated from the **SQLite traffic archive** and persisted in SQLite. Multi-hour views no longer scan large Redis sorted sets.
- Added an exact **5 hour (`18000` seconds)** dashboard range. Requested ranges are no longer rounded to the nearest predefined analytics window.
- Added bounded-memory analytics for long windows: endpoint/application cardinality is aggregated inside SQLite and a separate snapshot worker processes one requested window at a time instead of retaining large Python flow/IP sets. Archiving continues while a long 24h snapshot is being calculated.
- Managed Redis now defaults to a **128 MiB hard memory limit**, `noeviction`, no RDB snapshots and no AOF. The defaults prevent Redis from growing until the RouterOS container is killed by OOM; durable traffic history lives in `/data/ids.db`.
- Added `TRAFFIC_ARCHIVE_INTERVAL_SECONDS`, `TRAFFIC_ARCHIVE_LAG_SECONDS` and `TRAFFIC_ARCHIVE_BATCH_SIZE` controls. Defaults move committed data out of Redis every 5 seconds with a 10 second live-buffer lag.
- Upgrades with a large legacy Redis backlog drain it in bounded batches before publishing fresh SQLite snapshots, preventing partially migrated history from appearing in charts.


## What changed in 0.11.2

- Replaced full-traffic RouterOS Packet Sniffer capture with a **hybrid TZSP path**: routed IPv4 uses `/ip firewall mangle action=sniff-tzsp`, while Packet Sniffer streams only `filter-mac-protocol=!ip` traffic such as ARP, IPv6, LLDP and PPPoE.
- The capture deploy no longer assumes a production bridge, VLAN or interface name. `TZSP_L2_INTERFACE=` defaults to RouterOS `filter-interface=all`; set it only when a site intentionally wants to constrain the low-volume non-IPv4 stream.
- Packet Sniffer TZSP destination now tries the current `streaming-server=IP:PORT` syntax first and automatically falls back to legacy split `streaming-server`/`streaming-port` syntax.
- The project-owned IPv4 mangle rule is identified by the exact comment `MikroSuricata TZSP IPv4`; deployment replaces only that rule and leaves unrelated mangle rules untouched. RouterOS `sniff-tzsp` continues processing subsequent mangle rules after cloning the packet.
- Added `scripts/configure-routeros-tzsp-hybrid.sh` to migrate an existing installation without rebuilding the container, including `DRY_RUN=true` preview mode.
- Replaced the VLAN-specific RouterOS template with `routeros/02-tzsp-hybrid.rsc` and added detailed capture coverage, limitations and migration documentation in `docs/routeros-tzsp-capture.md`.

## What changed in 0.11.0

- Reworked the Rust TZSP data-plane for RouterOS containers that cannot raise kernel `rmem_max`: UDP receive and TZSP decode/TAP injection now run on **separate threads**.
- The RX thread does only blocking `recvmmsg(MSG_WAITFORONE)` into reusable preallocated batches. It never waits for TAP, Suricata, Python, Redis or the UI.
- Added a bounded **userspace packet queue** (`TZSP_QUEUE_MB`, default 64 MiB). With the default 256-packet batch and 12 KiB slots this provides roughly half a second of burst absorption at 1 Gbit/s even when the kernel socket is limited to ~416 KiB.
- If downstream processing falls behind after the userspace queue fills, the RX thread keeps draining UDP into a scratch batch and records explicit `queue_dropped_datagrams` instead of silently turning all loss into kernel UDP drops.
- Increased the default receive batch from 64 to **256** and made the per-datagram buffer configurable (`TZSP_DATAGRAM_BYTES`, default 12288; up to 65535). Oversized/truncated datagrams are counted explicitly.
- Added live capture diagnostics: TZSP ingress rate, inspected rate, capture efficiency, userspace queue fill, kernel drops/s, queue drops/s, truncated packets and RX/TAP worker health.
- The overview now labels the main value as **Inspected throughput** and separately shows TZSP ingress, so a RouterOS mirror limitation can be distinguished from a Suricata/TAP bottleneck immediately.
- Prometheus exports the new Rust queue/capture-health gauges and drop counters.

## What changed in 0.10.0

- Moved the complete TZSP packet hot-path out of Python into a dedicated **Rust data-plane process**. Rust now owns batched UDP receive (`recvmmsg`), TZSP decoding, direction accounting and TAP injection; packet bytes no longer cross into Python.
- Added configurable UDP receive buffering and batched receive. In 0.11.0 the hot path was further split into dedicated RX/TAP threads with a userspace queue.
- Added 1 Hz Unix-socket telemetry from Rust to the Python control plane. The dashboard now receives live throughput independently from retained Redis history, so **Throughput now** is current instead of reusing a stale historical payload.
- Added receiver observability to the UI/health/Prometheus path: actual receive-buffer size, Rust PID, telemetry age and kernel UDP drop counter.
- Fixed the overview refresh path so throughput history is refreshed during periodic UI refreshes, not only at initial load/range changes.
- The old `FORENSIC_PCAP_MODE=blocks` pre-event ring required Python to process every frame. With the isolated Rust data-plane it is treated as a legacy setting and safely falls back to Suricata alert PCAP capture. The new default is `alerts`.
- Docker now compiles and tests the Rust receiver in a dedicated build stage and copies only the stripped receiver binary into the runtime image. The existing RouterOS amd64/arm64/armv7 image workflow is preserved.


## What changed in 0.9.1

- Fixed a startup regression where `main.py` passed `backup_manager` to `EVEWatcher` even though the watcher has no such constructor argument. Backup management remains correctly attached to the web/admin layer.
- Added a regression test that statically validates keyword arguments used to construct `EVEWatcher` against its Python constructor signature.
- Made the local cleartext-FTP SYN policy rule explicitly `flow:to_server,stateless` and bumped SID `1000113` to revision 2, eliminating the Suricata 8 direction warning seen during validation/startup.

## What changed in 0.9.0

- Added conservative **MITRE ATT&CK** annotations to NDR evidence and correlated incidents. Mapping is limited to techniques supported by network-visible evidence and is backfilled for existing NDR event history during the schema migration.
- Expanded autonomous behavioral analytics with established-asset domain/fingerprint baselines, NXDOMAIN bursts, repeated high-entropy DNS tunnel candidates and outbound transfer-volume detection.
- Added **Adaptive Rule Intelligence** based on observed alert volume, duplication/concentration and incident correlation. It never disables signatures automatically; operators can explicitly apply a standard global Suricata `limit` threshold to high-noise candidates.
- Added persistent **ruleset snapshots and rollback**. Local rules, threshold configuration, merged vendor rules and enabled `suricata-update` source state are captured before rule changes/vendor updates, with bounded retention.
- Added persistent **IDS-state backups** under `/data/backups` using SQLite online backup. Backups include the database, custom Suricata state, enabled source definitions and merged rules while excluding Redis runtime data, EVE logs and the forensic PCAP ring.
- Added an SQLite **administrative audit trail** for login attempts and IDS-management operations including IOC changes, incident state changes, threshold/rule operations, backups and maintenance actions.
- Extended the Intelligence, Rules and System views with ATT&CK evidence, noisy-rule analysis, one-click threshold proposals, ruleset rollback, backup download/delete and audit history.
- The new 0.9.0 functionality is self-contained in the IDS container and persistent `/data`; it does **not add any new RouterOS configuration/firewall modification path**. Existing optional RouterOS integrations are unchanged.

## What changed in 0.8.1

- Fixed RouterOS/container rebuild persistence for signature-feed selections. All `suricata-update` source-management commands (`list-sources`, `update-sources`, `enable-source`, `disable-source`) now use the same persistent `-D /data/lib/suricata` directory as rule downloads, so enabled source definitions survive image replacement together with the single `/data` mount.
- Added bulk signature-source selection and a sequential download queue in **Signature Feeds**. Operators can select visible sources or all parameter-free free sources, queue them together, and the IDS enables each source before performing one merged download, `suricata -T` validation and live reload.
- Queue status and per-source progress/errors are visible in the feed table. A failed source does not prevent the remaining selected sources from being processed; the merged ruleset still keeps the previous known-good file if final download or validation fails.

A lightweight IDS stack designed to run as a **single container on MikroTik RouterOS**.

## What changed in 0.8.0

- Added a persistent **MikroSuricata NDR correlation engine** on top of Suricata: multi-stage incidents, risk score 0-100, bounded evidence timelines and incident triage.
- Added **asset intelligence** persisted in SQLite and enriched from RouterOS ARP/DHCP through the RouterOS v7 REST API. Baselines track applications, outbound ports, identities, domains and fingerprints.
- Added behavioral detections for new services on established assets, periodic beaconing, DGA/high-entropy DNS bursts, internal lateral fan-out, outbound scans of sensitive services, unusually large outbound transfers and repeated IP/MAC identity changes consistent with ARP spoofing or address conflicts.
- Added persistent local **threat intelligence** for IP, domain, SHA-256, JA3, JA4 and HASSH. IP/domain/JA3/JA4/HASSH are materialized as Suricata datasets; malicious SHA-256 lists are matched natively on supported file protocols.
- Added Suricata 8 `xbits` correlation for scan -> administrative access and internal probe -> SMB/RDP/SSH/WinRM/WinBox sequences.
- Forensic PCAP now defaults to `FORENSIC_PCAP_MODE=blocks`: packets stay in a bounded RAM pre-event ring and a PCAP is persisted only after a successful RouterOS block. `alerts`, `all` and `off` modes remain selectable by environment variables.
- Added MikroTik-specific detection for repeated RouterOS API/API-SSL access on TCP 8728/8729, in addition to WinBox/SSH/RDP and existing edge rules.
- Added sensor-quality health monitoring for capture drops, Suricata alert-queue overflow and TZSP/TAP injection errors.
- Managed Redis now uses **AOF everysec + RDB** persistence under the same `/data` volume.
- Added optional asynchronous high-risk incident webhooks (`NOTIFY_WEBHOOK_URL`, default disabled) with risk threshold and anti-spam escalation logic.
- Expanded the Intelligence UI with incident status actions, evidence, asset inventory, IOC management and the forensic PCAP ring.
- RouterOS deployment now forwards the NDR, Redis AOF, inventory and optional notification settings from `deploy-routeros.env` into `IDS_ENV`, so the same controls work without manual container edits.

## What changed in 0.7.2

- Fixed the Suricata 8.0.6 EVE profile: removed unsupported `llmnr` and `ftp-data` logger entries. `FTP_DATA` events remain supported through the `ftp` EVE logger.
- Normalized the IKE EVE entry to the Suricata 8.0.6 logger syntax.
- Removed the visual `M` logo mark from both the sidebar header and authentication modal.
- Removed the non-existent LLMNR EVE event filter from Live Sessions.

## What changed in 0.7.1

- All mutable state now uses a **single persistent `/data` mount**. SQLite, Redis, EVE logs, Suricata-update state/vendor rules, custom rules, sessions and analytics snapshots are kept below `/data`.
- Docker Compose now creates only `routeros-suricata-data`; the migration helper can merge the old 0.7.0 log/rule volumes into `/data/logs/suricata` and `/data/lib/suricata` without overwriting initialized targets.
- RouterOS deploy and upgrade helpers normalize `IDS_MOUNTS` to one mapping: `<disk>/containers/suricata-data -> /data`.
- `suricata-update` uses its `-D /data/lib/suricata` data directory and runtime Suricata uses that persistent rule path directly.

## What changed in 0.7.0

- Dashboard authentication now uses a normal username/password modal. The browser keeps only an `HttpOnly` session cookie; sessions are stored in SQLite and survive container restarts.
- Traffic analytics are materialized into SQLite for **15m / 1h / 6h / 24h** and refreshed periodically, so charts can render immediately while Redis is starting or reconnecting.
- Redis, SQLite, sessions and chart snapshots live under persistent `/data`; Docker Compose uses named volumes and RouterOS keeps the existing `IDS_MOUNTS` across image-only upgrades.
- Chart rendering is visibility-aware and re-runs on tab/view changes, resize, visibility changes and layout observation.
- Mobile mode now uses an off-canvas navigation drawer, responsive metrics/panels/forms and horizontally scrollable investigation tables.
- The System icon no longer depends on a font glyph; it is an inline SVG.
- The image now installs Suricata 8 from Debian trixie-backports and loads a project-owned EVE profile so package updates do not silently remove required telemetry.
- EVE telemetry explicitly enables DNS v3, `community_id`, Ethernet metadata, SHA-256 file hashes, JA3/JA4, SSH HASSH, ARP/DHCP, plus Windows/AD and application protocols such as SMB, RDP, Kerberos, DCERPC, LDAP, QUIC and HTTP/2/DoH2.
- Security analytics add anomalies, NXDOMAIN counts, encrypted/cleartext session counts, local/remote endpoint inventory, passive IP/MAC asset observations, top signatures/severities and JA4/JA3/HASSH fingerprint inventory.
- The built-in local ruleset adds NXDOMAIN/DNS-rate signals, outbound SMB/SMTP/FTP policy detections, exposed database-service probes and an internal administrative/lateral-movement burst detector.

RouterOS mirrors traffic with a hybrid TZSP path: routed IPv4 is cloned by an IPv4 mangle `sniff-tzsp` rule, while Packet Sniffer handles non-IPv4 Ethernet traffic. A dedicated Rust process decodes and injects the frames into TAP, Suricata analyzes them, and the Python control plane handles EVE/SQLite/NDR/API/UI without sitting in the packet hot-path.

## NDR / MikroTik-specific options

The defaults are conservative: NDR analysis is enabled, but automatic NDR blocking and outbound webhooks are disabled until explicitly configured. Useful environment variables:

```text
NDR_ENABLED=true
NDR_CORRELATION_WINDOW_SECONDS=1800
BEHAVIOR_MIN_OBSERVATIONS=50
NDR_AUTO_BLOCK=false
NDR_AUTO_BLOCK_RISK=92
ROUTEROS_INVENTORY_INTERVAL_SECONDS=300
NOTIFY_WEBHOOK_URL=
NOTIFY_MIN_RISK=80
REDIS_MAXMEMORY_MB=128
REDIS_SNAPSHOT_SECONDS=0
REDIS_AOF=false
TRAFFIC_ARCHIVE_INTERVAL_SECONDS=5
TRAFFIC_ARCHIVE_LAG_SECONDS=10
TRAFFIC_ARCHIVE_BATCH_SIZE=1000
```

All durable NDR state, IOC data, traffic history/chart snapshots, Suricata logs/rules and forensic PCAP rotation remain below the single persistent `/data` mount. Redis is a bounded transient ingest buffer by default; SQLite holds the durable traffic archive.

## Architecture

```text
RouterOS routed IPv4 ----> mangle sniff-tzsp ---------+
                                                      | TZSP UDP/37008
RouterOS non-IPv4 -----> Packet Sniffer (!ip) --------+
                                                      v
single RouterOS container
  Debian slim
  + Rust TZSP data-plane (recvmmsg -> TZSP -> TAP)
  + TAP suritap0
  + Suricata IDS
  + Python control plane / EVE JSON watcher
  + SQLite alerts / assets / NDR incidents / sessions / traffic archive
  + MikroSuricata behavior + correlation engine
  + local IOC datasets (IP/domain/SHA256/JA3/JA4/HASSH)
  + bounded Redis ingest buffer
  + SQLite dashboard snapshots
  + Rust -> Python 1 Hz Unix telemetry
  + WebSocket live throughput / event stream
  + Web UI :8080
  + optional RouterOS REST blocking
```

The capture path is intentionally independent from the UI/control plane:

```text
RouterOS TZSP UDP
      |
      v
Rust receiver -- recvmmsg() --> TZSP decode --> TAP write --> Suricata
      |
      +-- 1 Hz counters only --> Unix datagram --> Python --> short Redis buffer / SQLite / WebSocket / Prometheus
```

If Redis, the browser or an analytics request is slow, it cannot block UDP receive/TAP injection.

The RouterOS deployment workflow is:

```text
Docker/Podman build host
        |
        | build for RouterOS CPU architecture
        v
container image
        |
        | docker save / podman save
        v
*.tar
        |
        | SCP
        v
RouterOS external disk
        |
        | /container/add file=...
        v
running IDS container
```

Docker Compose is only provided for Linux integration testing. RouterOS receives one saved container image as a TAR archive.

---

## Quick start - Docker Compose on Linux

Use these steps for the local full-stack test with TZSP, TAP and Suricata.

### 1. Check prerequisites

You need:

- Linux,
- Docker Engine,
- Docker Compose v2 (`docker compose`),
- `/dev/net/tun` available on the host.

Check them:

```bash
docker --version
docker compose version
test -c /dev/net/tun && echo "TUN/TAP: OK" || echo "TUN/TAP: MISSING"
```

If `/dev/net/tun` is missing on Linux, try:

```bash
sudo modprobe tun
```

Then check `/dev/net/tun` again.

### 2. Create the runtime configuration

From the project directory:

```bash
cp .env.example .env
```

For a first test the defaults can be used. Before monitoring a real network, review at least:

```dotenv
SURICATA_HOME_NET=[192.168.0.0/16,10.0.0.0/8,172.16.0.0/12]
MONITORED_NETWORKS=192.168.0.0/16,10.0.0.0/8,172.16.0.0/12
ALERT_MAX_SEVERITY=2
ALERT_DEDUP_WINDOW_SECONDS=300
ADMIN_USERNAME=admin
ADMIN_PASSWORD=<long-unique-password>
SESSION_HOURS=168
ANALYTICS_SNAPSHOT_INTERVAL_SECONDS=60
AUTO_BLOCK=false
```

Keep `AUTO_BLOCK=false` until alerts are verified.

### 3. Build and start

```bash
docker compose up -d --build
```

Compose uses one named volume, `routeros-suricata-data`, mounted at `/data`. SQLite, Redis, chart snapshots, sessions, Suricata logs, local configuration and downloaded vendor rules all live below that mount. RouterOS uses the same single-mount layout: `disk1/containers/suricata-data -> /data` through `IDS_MOUNTS`. Rebuilding or replacing the container therefore leaves all mutable IDS data outside the image root.

When upgrading a Docker installation from **0.6.1 or older**, migrate the old bind-mounted `./data`, `./logs` and `./data/vendor-rules` before the first 0.8.1 start:

```bash
docker compose build
./scripts/migrate-docker-volumes.sh
docker compose up -d
```

The migration keeps the existing `routeros-suricata-data` volume and folds legacy log/rule volumes into `/data/logs/suricata` and `/data/lib/suricata` only when those target directories are empty. `./scripts/first-run.sh` performs this step automatically. If you start Compose manually, set a strong `ADMIN_PASSWORD` in `.env`; `first-run.sh` generates one automatically for a new `.env`.

Check container status:

```bash
docker compose ps
```

Follow logs:

```bash
docker compose logs -f ids
```

You can also use the container name directly:

```bash
docker logs -f routeros-suricata-tzsp
```

Do not run plain `docker logs -f` - Docker requires a container name.

### 4. Run the end-to-end self-test

```bash
./scripts/selftest.sh
```

Expected result:

```text
SELFTEST OK: Suricata emitted marked TZSP pipeline test alert(s); UI filtering remains enabled
```

The self-test uses reserved SID `1000001` and a unique payload marker. Normal ICMP/ping traffic cannot match it, and SID `1000001` is filtered from SQLite/UI by default.

### 5. Open the dashboard

```text
http://127.0.0.1:8080
```

Status API:

```bash
curl http://127.0.0.1:8080/api/status
```

### 6. Stop the stack

```bash
docker compose down
```

### One-command first start

The helper script checks Docker, Compose and `/dev/net/tun`, creates `.env` if needed, starts the stack and runs the self-test:

```bash
./scripts/first-run.sh
```

If scripts are not executable after unpacking an archive:

```bash
chmod +x dev.sh scripts/*.sh
./scripts/first-run.sh
```

### Rebuild after updating older images

Version `0.3.2` could restart continuously with:

```text
chown: invalid user: 'suricata:suricata'
```

Version `0.3.3` added the `suricata` system account. Version `0.3.4` also fixes a second startup issue where the `suricata -T` configuration check could create root-owned `eve.json`, `fast.log`, and `stats.log`, causing the real Suricata process to fail with `Permission denied`. It also relocates the Unix command socket into `/run/suricata/`. Rebuild the image completely:

```bash
docker compose down
docker compose build --no-cache
docker compose up -d
docker compose ps
docker compose logs -f ids
```

---

## Alert tuning and false-positive control

Version `0.5.2` uses two tuning layers. Raw Suricata EVE stays on disk, while the incident database/UI defaults to severity `1-2` and collapses repeated identical SID/source/destination tuples for five minutes. This prevents informational events from flooding the dashboard without changing the raw sensor log.

```dotenv
ALERT_MAX_SEVERITY=2
ALERT_DEDUP_WINDOW_SECONDS=300
ALERT_IGNORE_SIDS=
ALERT_IGNORE_CATEGORIES=
```

For a known false positive, prefer sensor-level tuning in `/data/suricata/threshold.config`. The dashboard can add a full SID suppression with **Suppress SID**, or the file can be edited directly from the rule-management panel. Examples:

```text
suppress gen_id 1, sig_id 1234567
threshold gen_id 1, sig_id 1234567, type limit, track by_src, count 1, seconds 300
```

Do not globally suppress a rule just because it fired once. First verify the SID, endpoint, direction and expected application behavior.

---

## Adding new Suricata detections

Suricata is signature/rule driven; it does not need model training to learn a new network detection. Add environment-specific signatures to `/data/suricata/custom.rules`, or use **Custom Suricata signatures** in the dashboard. Save performs a `suricata -T` validation first and only then writes the file and requests a live rule reload. Runtime loads persisted `/data/suricata/*.rules`, so additional rule files can be placed beside `custom.rules` without rebuilding the image.

Example local rule:

```text
alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"LOCAL suspicious URI marker"; flow:established,to_server; http.uri; content:"/admin/export"; nocase; classtype:web-application-activity; priority:2; sid:1000100; rev:1;)
```

Use unique local SIDs. SID `1000001` is reserved for the marked pipeline self-test, built-in production detections use `1000101-1000115`, and site-specific rules should use `1001000+`.

Vendor rules are managed with `suricata-update`. A baseline ET/Open ruleset and a current OISF source index are baked into the image. Runtime rule state is written with `suricata-update -D /data/lib/suricata`, so downloaded feeds, source definitions and caches are inside the single persistent `/data` mount. An empty first-run data directory is seeded from the image baseline. `scripts/update-rules.sh` applies persisted `/data/suricata/disable.conf`, `enable.conf`, and `modify.conf`.

The dedicated **Signature Feeds** page has a provider table backed by the official OISF `suricata-update` catalog. It also shows the current active merged-rule count and the configured automatic-update interval. It can also add/remove a signature source directly by HTTP(S) URL when that feed is not present in the public catalog. The UI lists sources, shows vendor/license/tags/status, refreshes the OISF index, enables or disables parameter-free feeds, and downloads all active feeds on demand. Multiple parameter-free sources can be selected and queued together; they are enabled sequentially and then rebuilt/validated once. All source-management commands and rule downloads use `-D /data/lib/suricata`, so source definitions survive RouterOS container rebuilds with the same `/data` mount. ET/Open remains the default source and cannot be accidentally disabled from the panel. The **Rules** page also exposes a read-only, searchable, paginated view of the merged `/data/lib/suricata/rules/suricata.rules` file. The merged-rule browser is collapsed by default to keep the page compact while leaving the active-rule counter visible in its header.

Every feed update is transactional at the merged-rules level: the existing `suricata.rules` is backed up, new signatures are downloaded, the complete Suricata configuration is tested with `suricata -T`, and only a validated ruleset is kept. If download or validation fails, the previous known-good rules are restored. The periodic updater uses the same active-source set and runs every `RULE_UPDATE_INTERVAL_HOURS` when the interval is greater than zero. The default is `24`; set `RULE_UPDATE_INTERVAL_HOURS=0` to disable automatic updates.

```dotenv
UPDATE_RULES_ON_START=false
RULE_UPDATE_INTERVAL_HOURS=24
FORENSIC_PCAP_MODE=alerts
FORENSIC_PCAP_WINDOW_SECONDS=60
FORENSIC_PCAP_MEMORY_MB=64
FORENSIC_PCAP_MAX_FILES=32
FORENSIC_PCAP_MAX_TOTAL_MB=512
```

---

## Built-in production detections

The image ships with a conservative local baseline in addition to the ET/Open snapshot baked by `suricata-update`. Local rules are rate-limited so ordinary single packets do not become incidents. The baseline covers repeated SSH/RDP/WinBox attempts, SYN scans, ICMP sweeps, inbound and outbound SMB policy violations, unusually long or high-rate DNS activity, NXDOMAIN bursts, outbound Telnet/FTP/direct SMTP, external database-service probes and an internal RDP/SMB lateral-movement burst signal.

The production baseline uses SIDs `1000101-1000116`, with NDR state rules `1000120-1000123` and managed threat-intelligence rules starting at `1000201`. The deterministic pipeline self-test remains SID `1000001`, but it only matches the exact payload marker generated by `scripts/send_test_tzsp.py` and is ignored by the incident database. Environment-specific exceptions should be handled with `threshold.config` or suppression rather than weakening the entire sensor.

ET/Open is still the main vendor signature source. `suricata-update` is the supported manager for refreshing it; the image seeds `/data/lib/suricata` inside the persistent data volume on first start.

---

## Upload a ready RouterOS image without deploying it

The upload helper requires an already-built TAR and does exactly one job:

```bash
cp deploy-routeros.env.example deploy-routeros.env
./scripts/upload-routeros-image.sh build/routeros-suricata-tzsp-arm64.tar
```

It performs SCP upload plus a read-only file-list verification. It does **not** detect architecture, build an image, run `/container/add`, import an `.rsc`, change RouterOS configuration, or start a container.

---

## Database, storage and maintenance

The dashboard detects SQLite and persistent storage separately. It shows DB path, schema version, row count, DB/WAL size, filesystem usage and Suricata log size. SQLite uses WAL mode and performs a small schema migration automatically when upgrading from older project versions.

Administrative actions require a dashboard session. Configure `ADMIN_USERNAME` and a long unique `ADMIN_PASSWORD`; the server issues an `HttpOnly`, `SameSite=Strict` cookie and stores only a hash of the random session token in SQLite. Set `SESSION_COOKIE_SECURE=true` when the dashboard itself is served over HTTPS. `ADMIN_TOKEN` is accepted only as a migration fallback and is no longer stored by the browser.

Authenticated maintenance includes clearing incident/history data, SQLite `VACUUM`, runtime counter reset, RouterOS block-list actions, validated custom-rule/threshold edits, live Suricata rule reloads and managed signature-feed updates. Keep port `8080` on a trusted management network or place the dashboard behind HTTPS.

SQLite also stores the normalized traffic archive and rolling dashboard summaries. Standard warm ranges are `900`, `3600`, `18000`, `21600` and `86400` seconds (15m / 1h / 5h / 6h / 24h), while arbitrary requested ranges are materialized on demand. The archive worker first commits old Redis entries to SQLite, then deletes those exact Redis members; chart calculations therefore do not require multi-hour Redis history.

Redis is deliberately transient in 0.11.3. New deployments use `REDIS_MAXMEMORY_MB=128`, `REDIS_SNAPSHOT_SECONDS=0` and `REDIS_AOF=false`. If an existing RouterOS `IDS_ENV` from an older release still explicitly sets `REDIS_AOF=true` or a non-zero `REDIS_SNAPSHOT_SECONDS`, change those values to the new defaults when you want the fully transient Redis model.

---

## Extended statistics

The dashboard reports alert hits vs deduplicated incidents, selected-window activity, top signatures, severity distribution, protocol anomalies, NXDOMAINs, encrypted vs cleartext sessions, local clients vs remote peers, JA4/JA3/HASSH fingerprints, passive IP/MAC assets, filter/dedup/error counters, block attempts/results and Suricata EVE `stats` counters such as decoder/capture/drop values when emitted by the installed configuration. Live-history records remain searchable by IP/port, signature, `flow_id`, `community_id` and transaction ID.


---

## Dashboard sections

The web UI sections are **Overview**, **Live Sessions**, **Security**, **Intelligence**, **Blocks**, **Reports**, **Signature Feeds**, **Rules** and **System**. **Security** is split into Incidents / Analytics / Telemetry subtabs, while **Intelligence** is split into Incidents / Assets / Threat intel / Forensics so large inventories stay one click away instead of far down the page. **System** includes a dedicated Redis status card with connection/runtime, persistence, retained event and writer-health information. Incident timestamps are stored in UTC and rendered in the browser's local timezone. Repeated alerts are aggregated by SID, source, destination, protocol and destination port within the configured deduplication window.

---

## Local Web UI development without Docker

The web dashboard can be started locally without Docker, Suricata, TAP, `/dev/net/tun`, or root privileges.

Requirements:

- Linux or macOS
- Python 3
- Python `venv` support

Start the development dashboard:

```bash
./dev.sh
```

The script will:

1. create `.venv/` if needed,
2. install dependencies from `requirements.txt`,
3. create `data/dev/`,
4. start the web-only application.

Open:

```text
http://127.0.0.1:8080
```

This mode intentionally does **not** start Suricata, TZSP capture, TAP, or RouterOS integration.

### Status API

The dashboard and external monitoring systems can use:

```text
GET /api/status
```

Example:

```bash
curl http://127.0.0.1:8080/api/status
```

The response includes:

- overall application status and uptime,
- Web UI/API, TZSP, TAP, Suricata and EVE watcher state,
- SQLite existence/path/size/WAL/schema/row count,
- persistent filesystem usage and Suricata log size,
- managed custom-rule and threshold/suppression status,
- RouterOS REST integration status and ports,
- runtime packet/error/filter/dedup/block counters,
- the latest numeric Suricata EVE `stats` counters.

`GET /api/health` is kept as a compatibility alias and returns the same status payload.

### Add a sample alert

To start the dashboard with one demo alert in an empty development database:

```bash
DEV_SEED_DATA=true ./dev.sh
```

Development database:

```text
data/dev/ids.db
```

To use another port:

```bash
WEB_PORT=8090 ./dev.sh
```

To listen on all local interfaces:

```bash
WEB_BIND=0.0.0.0 ./dev.sh
```

Do not expose the development server directly to an untrusted network.

---

## Local tests

Install dependencies and run tests:

```bash
./dev.sh --test
```

Or run tests before starting the dashboard:

```bash
RUN_TESTS=true ./dev.sh
```

The project currently uses only the Python standard library, including `unittest` for tests. `requirements.txt` is kept as the canonical dependency file so future packages can be installed automatically by `dev.sh`.

---

## Docker integration test

Prepare local environment configuration:

```bash
cp .env.example .env
```

Build and start:

```bash
docker compose up -d --build
```

Dashboard:

```text
http://127.0.0.1:8080
```

Send the built-in TZSP test packet:

```bash
./scripts/selftest.sh
```

The test rule should generate raw EVE alert SID `1000001` with signature `LOCAL TEST TZSP PIPELINE MARKER`. It is intentionally filtered from the incident database/UI.

Stop the stack:

```bash
docker compose down
```

---

## RouterOS requirements

Before deployment, verify that the router has:

- a RouterOS version and package set that supports containers,
- the `container` package installed,
- container and sniffer functionality enabled in device mode,
- SSH access from the build machine,
- enough storage for the image, container root directory, Suricata rules, logs, and SQLite database,
- preferably an external disk instead of small internal flash storage.

Useful RouterOS checks:

```routeros
/system/resource/print
/system/package/print
/system/device-mode/print
/disk/print
/container/print
```

---

## Automated RouterOS deployment

Create the deployment configuration:

```bash
cp deploy-routeros.env.example deploy-routeros.env
```

Upload a ready image first:

```bash
./scripts/upload-routeros-image.sh build/routeros-suricata-tzsp-arm64.tar
```

Then deploy by giving the **RouterOS-side TAR path** directly:

```bash
./scripts/deploy-routeros.sh routeros-suricata-tzsp-arm64.tar
```

The deployer no longer builds, detects image architecture, renames, or re-uploads the image. For project version `0.11.3` it creates:

```text
name=suricata_0.11.3
file=routeros-suricata-tzsp-arm64.tar
root-dir=/containers/suricata_0.11.3/root
```

The remaining deployment work is unchanged: private container bridge/VETH/NAT, environment, persistent mounts, optional RouterOS REST user/firewall integration, hybrid TZSP capture configuration, image extraction wait, container start, and final status. Existing containers are not removed. Re-running deployment for the same version stops with `Container suricata_0.11.3 already exists`.

For SSH key authentication set:

```dotenv
ROUTER_IDENTITY_FILE=/home/user/.ssh/id_ed25519
```

---

## Build RouterOS image without deployment

ARM64:

```bash
./scripts/build-routeros.sh arm64
```

AMD64/x86_64:

```bash
./scripts/build-routeros.sh amd64
```

ARMv7/armhf:

```bash
./scripts/build-routeros.sh arm
```

Generated files are written to `build/`, for example:

```text
build/routeros-suricata-tzsp-arm64.tar
build/routeros-suricata-tzsp-arm64.tar.sha256
```

---

## Manual SCP and RouterOS import

Example:

```bash
./scripts/build-routeros.sh arm64
scp build/routeros-suricata-tzsp-arm64.tar admin@192.168.88.1:/
```

RouterOS templates are located in `routeros/`:

```text
01-container-network.rsc
02-tzsp-hybrid.rsc
03-rest-and-firewall.rsc
04-container-import-amd64.rsc
04-container-import-arm64.rsc
04-container-import-arm.rsc
rollback.rsc
```

After extraction, inspect and start the container:

```routeros
/container/print detail
/container/start suricata-ids
/log/print where message~"suricata|TZSP|IDS"
```

---

## Persistent data on RouterOS

The deployment uses one persistent directory outside the image root:

```text
<disk>/containers/suricata-data -> /data
```

Inside it the application keeps:

```text
/data/ids.db                 SQLite, sessions, traffic archive and analytics snapshots
/data/redis/                 Redis runtime directory (persistence disabled by default)
/data/logs/suricata/         EVE/raw Suricata logs
/data/lib/suricata/          suricata-update feeds, cache and vendor rules
/data/suricata/              custom rules, thresholds and update filters
```

Replacing or restarting the application container does not remove any of these files.

---

## TZSP capture

The default RouterOS deployment uses a hybrid capture model:

```text
routed IPv4  -> /ip firewall mangle action=sniff-tzsp
non-IPv4 L2  -> /tool sniffer filter-mac-protocol=!ip
                     \_______________________________/
                              TZSP UDP/37008
```

Relevant `deploy-routeros.env` settings:

```dotenv
TZSP_PORT=37008
CONFIGURE_TZSP_CAPTURE=true
CONFIGURE_IPV4_MANGLE=true
CONFIGURE_L2_SNIFFER=true
START_L2_SNIFFER=true
TZSP_L2_INTERFACE=
TZSP_L2_MAC_PROTOCOL=!ip
```

No production bridge/VLAN name is assumed. Empty `TZSP_L2_INTERFACE` maps to RouterOS `filter-interface=all`. To intentionally constrain non-IPv4 capture, set an exact interface name, for example `TZSP_L2_INTERFACE=bridge-core`.

The IPv4 rule is added to `chain=forward` and is owned by the comment `MikroSuricata TZSP IPv4`. Only that project-owned rule is replaced; unrelated mangle rules are left intact. `sniff-tzsp` clones the packet and then continues to subsequent mangle rules.

The Packet Sniffer complement uses `filter-mac-protocol=!ip`. RouterOS defines `ip` as IPv4 EtherType `0x0800` and `ipv6` separately as `0x86DD`, so this stream includes ARP, IPv6, LLDP, PPPoE and other non-IPv4 Ethernet protocols. `filter-stream=yes` prevents the sniffer's own TZSP stream from being recaptured.

Current RouterOS builds may expose the target as `streaming-server=IP:PORT`; older builds/documentation may expose `streaming-server` plus `streaming-port`. Deployment tries the combined form first and falls back automatically.

To migrate an already running installation without changing the container image:

```bash
./scripts/configure-routeros-tzsp-hybrid.sh
```

Preview without modifying RouterOS:

```bash
DRY_RUN=true ./scripts/configure-routeros-tzsp-hybrid.sh
```

Packet Sniffer is global RouterOS state. If another administrative workflow owns `/tool/sniffer`, use `CONFIGURE_L2_SNIFFER=false` and manage the low-volume L2 complement separately. Hardware-offloaded bridge-only traffic may also be invisible to Packet Sniffer on some devices.

Full coverage details, caveats, migration behavior and verification commands are in [`docs/routeros-tzsp-capture.md`](docs/routeros-tzsp-capture.md).

---

## TAP and Suricata

Inside the container the application creates:

```text
suritap0
```

The pipeline is:

```text
TZSP datagram
  -> Rust batched UDP receiver
  -> Rust TZSP decoder
  -> Ethernet frame
  -> TAP suritap0
  -> Suricata
  -> eve.json
  -> Python control plane / EVE watcher
  -> SQLite / Web UI
```

The Rust receiver emits only a compact 1 Hz telemetry sample to Python. That sample drives the live throughput card and is persisted asynchronously for historical throughput charts.

Useful receive-path tuning:

```dotenv
TZSP_RCVBUF_BYTES=33554432
TZSP_BATCH_SIZE=256
TZSP_QUEUE_MB=64
TZSP_DATAGRAM_BYTES=12288
```

On RouterOS the kernel may cap the actual socket buffer around a few hundred KiB regardless of `TZSP_RCVBUF_BYTES`. The 0.11.0 receiver therefore does not rely on sysctl: a dedicated RX thread drains UDP into the preallocated userspace queue. The UI exposes both kernel drops and userspace queue drops, plus TZSP ingress versus traffic actually injected into Suricata.

If the RouterOS container cannot create the TAP interface, the application will fail early with an error related to `/dev/net/tun`, `TUNSETIFF`, or permissions. This is the main platform-specific capability to validate on the target router.

---

## RouterOS REST reaction

Automatic blocking is intentionally disabled by default:

```dotenv
AUTO_BLOCK=false
```

Observation mode should be used first. After validating alerts and false positives, the optional reaction engine can add selected addresses to a RouterOS firewall address list through REST.

Relevant settings:

```dotenv
AUTO_BLOCK=true
AUTO_BLOCK_MAX_SEVERITY=1
ROUTEROS_URL=https://172.31.255.1
ROUTEROS_USER=suricata-api
ROUTEROS_PASSWORD=CHANGE_ME
ROUTEROS_ADDRESS_LIST=IDS-BLOCK
BLOCK_TIMEOUT=1h
```

Do not enable automatic blocking until the monitored networks, exclusion list, REST credentials, firewall rule placement, and alert policy have been reviewed.

---

## Important files

```text
Dockerfile
Dockerfile / Debian slim runtime image

dev.sh
Local web-only development launcher

requirements.txt
Local Python development/test dependencies

app/main.py
Full RouterOS/container application entry point

app/dev_web.py
Local web-only entry point

rust/tzsp-receiver/
Production Rust TZSP receiver: batched UDP receive, decoder, counters and TAP injection

app/tzsp_rust.py
Rust process supervision plus 1 Hz telemetry bridge; no packet bytes enter Python

app/tzsp.py / app/tap.py
Legacy decoder/TAP helpers retained for focused tests; not used by the production packet path

app/eve.py
Suricata EVE JSON watcher

suricata/ids-output.yaml
Project-owned Suricata 8 EVE telemetry profile

app/policy.py
Alert/blocking policy

app/tuning.py
Second-stage alert severity/category/SID filter

app/rules.py
Validated custom rules, suppressions and vendor-rule updates

app/maintenance.py
Storage detection and safe maintenance helpers

app/routeros.py
RouterOS REST client

app/store.py
SQLite alert storage

app/webui.py
Dashboard and JSON API

scripts/build-routeros.sh
Build and save image to TAR

scripts/deploy-routeros.sh
Build, SCP, import, and start on RouterOS

routeros/
RouterOS configuration templates
```

---

## Prometheus `/metrics` ACL

The Prometheus endpoint is restricted independently from the dashboard. The source
IP/CIDR allowlist is always enforced and is parsed once at startup, so rejected
scrapes do not render metrics or trigger any application-side calculations.

```dotenv
# Default: local scrapes only. Exact IPs and CIDRs can be mixed, comma-separated.
METRICS_ALLOWED_IPS=127.0.0.1/32,::1/128

# Leave both empty for IP-only ACL. Set both for IP + Basic Auth.
METRICS_BASIC_AUTH_USERNAME=
METRICS_BASIC_AUTH_PASSWORD=
```

For example, to allow Prometheus at `192.168.88.50` and require Basic Auth:

```dotenv
METRICS_ALLOWED_IPS=192.168.88.50/32
METRICS_BASIC_AUTH_USERNAME=prometheus
METRICS_BASIC_AUTH_PASSWORD=<long-unique-password>
```

A client outside the allowlist receives HTTP `403`. A permitted IP with missing or
invalid Basic Auth receives HTTP `401`. Supplying only one Basic Auth variable is a
configuration error. An empty `METRICS_ALLOWED_IPS` denies all access. The ACL uses
the TCP peer address and deliberately ignores `X-Forwarded-For`; when using a reverse
proxy, allow the proxy address itself.

---

## Safety defaults

The default configuration is observation-oriented:

```dotenv
AUTO_BLOCK=false
ALERT_MAX_SEVERITY=2
UPDATE_RULES_ON_START=false
FORENSIC_PCAP_MODE=alerts
ROUTEROS_PASSWORD=CHANGE_ME
ADMIN_USERNAME=admin
ADMIN_PASSWORD=
SESSION_COOKIE_SECURE=false
```

Keep automatic firewall actions disabled until the capture path and alert quality are validated on the real network. Set a strong `ADMIN_PASSWORD` before enabling dashboard maintenance/rule-management actions, and restrict the Web UI to a trusted management network.

---

## Image-only upgrade on an already configured RouterOS

After the first deployment, when the VETH/private bridge/NAT, hybrid TZSP capture, `IDS_ENV` and `IDS_MOUNTS` already exist, do not run the full deploy just to change the image.

1. Upload the ready TAR only:

```bash
./scripts/upload-routeros-image.sh build/routeros-suricata-tzsp-arm64.tar
```

2. Swap the Suricata container only:

```bash
./scripts/upgrade-routeros-container.sh routeros-suricata-tzsp-arm64.tar
```

For version `0.11.3` the second command creates:

```text
name=suricata_0.11.3
file=routeros-suricata-tzsp-arm64.tar
root-dir=/containers/suricata_0.11.3/root
interface=veth-ids
envlist=IDS_ENV
mountlists=IDS_MOUNTS
```

The upgrade helper does not modify the bridge, IP addresses, NAT, VETH, hybrid TZSP capture, firewall, REST user or envlist definitions. It stops older `suricata_*` containers, normalizes `IDS_MOUNTS` to the single `<disk>/containers/suricata-data -> /data` mapping, creates the new versioned container, waits for image extraction and starts it. Older containers are kept stopped for rollback.

All new mutable state is written below `/data`, so subsequent image upgrades need only that one persistent mount.

### 0.9.3 HTTP disconnect hardening

- Treat browser/client disconnects during JSON responses and file downloads as normal connection termination.
- Suppress `BrokenPipeError`, `ConnectionResetError`, and equivalent socket disconnect tracebacks from the threaded HTTP server.
- No change to Redis-only traffic history semantics from 0.9.2.


### 0.9.4 Throughput visibility

- Traffic throughput now always renders the raw TZSP **Total** series, even when inbound/outbound classification is unavailable.
- The dashboard exposes unclassified throughput as `OTHER` instead of silently drawing an empty IN/OUT chart.
- Default `MONITORED_NETWORKS` now covers RFC1918 private LAN ranges (`192.168/16`, `10/8`, `172.16/12`) so common RouterOS LANs classify correctly without editing the image.
- Throughput remains persisted in Redis; no RAM history fallback is reintroduced.

### 0.9.5 Traffic accuracy and CPU reduction

- **Observed traffic** is calculated from raw TZSP packet-byte samples for the selected time range. EVE transaction records are no longer summed as traffic volume, avoiding repeated cumulative flow counters (for example impossible hundreds of GB in a 15-minute view).
- Traffic throughput has a lightweight `/api/traffic/throughput` path, so the speed graph can render without waiting for full EVE analytics.
- `failed`, `unknown`, `none` and similar Suricata application classifications are excluded from Top applications. Application counts are deduplicated by flow.
- `SURICATA IPv4 truncated packet` / IPv6 equivalents are hidden from dashboard history and analytics. Alerts rejected by the configured alert tuner are no longer persisted into dashboard Redis history; raw EVE remains on disk.
- Analytics snapshots are **demand-driven**: only time windows used by a browser/API are refreshed. The old unconditional full 24h Redis scan every minute was removed. Default refresh cadence is 60s (15m), 120s (1h), 300s (6h), 900s (24h), with current throughput read separately in constant time.
- The TZSP session tracker keeps packet throughput active but skips per-flow OrderedDict/hash work while Live Sessions streaming is off.
- LAN membership lookups are cached, reducing repeated `ipaddress` parsing for packet direction classification.
- Redis event persistence batches up to 128 EVE records into one multi-member `ZADD`, reducing Python socket and Redis command overhead while retaining every accepted dashboard event.
- Duplicate `fast.log` and standalone `stats.log` outputs are disabled because the application consumes EVE alerts/stats already.
- Analytics cache namespace is bumped to v3 so incorrect pre-0.9.5 traffic-volume snapshots are not reused after upgrade.