fix memory usage redis
This commit is contained in:
Mateusz Gruszczyński
@@ -1,6 +1,17 @@
|
||||
# MikroSuricata
|
||||
|
||||
Project version: `0.11.2`
|
||||
Project version: `0.11.3`
|
||||
|
||||
|
||||
## What changed in 0.11.3
|
||||
|
||||
- Reworked traffic retention so **Redis is only a short-lived ingestion buffer**. A dedicated archive worker commits normalized EVE events and throughput samples to SQLite and removes them from Redis only after the SQLite transaction succeeds.
|
||||
- Dashboard analytics and chart snapshots are now calculated from the **SQLite traffic archive** and persisted in SQLite. Multi-hour views no longer scan large Redis sorted sets.
|
||||
- Added an exact **5 hour (`18000` seconds)** dashboard range. Requested ranges are no longer rounded to the nearest predefined analytics window.
|
||||
- Added bounded-memory analytics for long windows: endpoint/application cardinality is aggregated inside SQLite and a separate snapshot worker processes one requested window at a time instead of retaining large Python flow/IP sets. Archiving continues while a long 24h snapshot is being calculated.
|
||||
- Managed Redis now defaults to a **128 MiB hard memory limit**, `noeviction`, no RDB snapshots and no AOF. The defaults prevent Redis from growing until the RouterOS container is killed by OOM; durable traffic history lives in `/data/ids.db`.
|
||||
- Added `TRAFFIC_ARCHIVE_INTERVAL_SECONDS`, `TRAFFIC_ARCHIVE_LAG_SECONDS` and `TRAFFIC_ARCHIVE_BATCH_SIZE` controls. Defaults move committed data out of Redis every 5 seconds with a 10 second live-buffer lag.
|
||||
- Upgrades with a large legacy Redis backlog drain it in bounded batches before publishing fresh SQLite snapshots, preventing partially migrated history from appearing in charts.
|
||||
|
||||
|
||||
## What changed in 0.11.2
|
||||
@@ -116,10 +127,15 @@ NDR_AUTO_BLOCK_RISK=92
|
||||
ROUTEROS_INVENTORY_INTERVAL_SECONDS=300
|
||||
NOTIFY_WEBHOOK_URL=
|
||||
NOTIFY_MIN_RISK=80
|
||||
REDIS_AOF=true
|
||||
REDIS_MAXMEMORY_MB=128
|
||||
REDIS_SNAPSHOT_SECONDS=0
|
||||
REDIS_AOF=false
|
||||
TRAFFIC_ARCHIVE_INTERVAL_SECONDS=5
|
||||
TRAFFIC_ARCHIVE_LAG_SECONDS=10
|
||||
TRAFFIC_ARCHIVE_BATCH_SIZE=1000
|
||||
```
|
||||
|
||||
All NDR state, IOC data, Redis persistence, Suricata logs/rules and forensic PCAP rotation remain below the single persistent `/data` mount.
|
||||
All durable NDR state, IOC data, traffic history/chart snapshots, Suricata logs/rules and forensic PCAP rotation remain below the single persistent `/data` mount. Redis is a bounded transient ingest buffer by default; SQLite holds the durable traffic archive.
|
||||
|
||||
## Architecture
|
||||
|
||||
@@ -134,10 +150,11 @@ single RouterOS container
|
||||
+ TAP suritap0
|
||||
+ Suricata IDS
|
||||
+ Python control plane / EVE JSON watcher
|
||||
+ SQLite alerts / assets / NDR incidents / sessions
|
||||
+ SQLite alerts / assets / NDR incidents / sessions / traffic archive
|
||||
+ MikroSuricata behavior + correlation engine
|
||||
+ local IOC datasets (IP/domain/SHA256/JA3/JA4/HASSH)
|
||||
+ Redis traffic history
|
||||
+ bounded Redis ingest buffer
|
||||
+ SQLite dashboard snapshots
|
||||
+ Rust -> Python 1 Hz Unix telemetry
|
||||
+ WebSocket live throughput / event stream
|
||||
+ Web UI :8080
|
||||
@@ -152,7 +169,7 @@ RouterOS TZSP UDP
|
||||
v
|
||||
Rust receiver -- recvmmsg() --> TZSP decode --> TAP write --> Suricata
|
||||
|
|
||||
+-- 1 Hz counters only --> Unix datagram --> Python --> Redis / WebSocket / Prometheus
|
||||
+-- 1 Hz counters only --> Unix datagram --> Python --> short Redis buffer / SQLite / WebSocket / Prometheus
|
||||
```
|
||||
|
||||
If Redis, the browser or an analytics request is slow, it cannot block UDP receive/TAP injection.
|
||||
@@ -424,7 +441,9 @@ Administrative actions require a dashboard session. Configure `ADMIN_USERNAME` a
|
||||
|
||||
Authenticated maintenance includes clearing incident/history data, SQLite `VACUUM`, runtime counter reset, RouterOS block-list actions, validated custom-rule/threshold edits, live Suricata rule reloads and managed signature-feed updates. Keep port `8080` on a trusted management network or place the dashboard behind HTTPS.
|
||||
|
||||
SQLite also stores the four rolling chart summaries (`900`, `3600`, `21600`, `86400` seconds). They refresh every `ANALYTICS_SNAPSHOT_INTERVAL_SECONDS` and are served immediately after UI entry/restart when newer live history is temporarily unavailable.
|
||||
SQLite also stores the normalized traffic archive and rolling dashboard summaries. Standard warm ranges are `900`, `3600`, `18000`, `21600` and `86400` seconds (15m / 1h / 5h / 6h / 24h), while arbitrary requested ranges are materialized on demand. The archive worker first commits old Redis entries to SQLite, then deletes those exact Redis members; chart calculations therefore do not require multi-hour Redis history.
|
||||
|
||||
Redis is deliberately transient in 0.11.3. New deployments use `REDIS_MAXMEMORY_MB=128`, `REDIS_SNAPSHOT_SECONDS=0` and `REDIS_AOF=false`. If an existing RouterOS `IDS_ENV` from an older release still explicitly sets `REDIS_AOF=true` or a non-zero `REDIS_SNAPSHOT_SECONDS`, change those values to the new defaults when you want the fully transient Redis model.
|
||||
|
||||
---
|
||||
|
||||
@@ -626,15 +645,15 @@ Then deploy by giving the **RouterOS-side TAR path** directly:
|
||||
./scripts/deploy-routeros.sh routeros-suricata-tzsp-arm64.tar
|
||||
```
|
||||
|
||||
The deployer no longer builds, detects image architecture, renames, or re-uploads the image. For project version `0.11.2` it creates:
|
||||
The deployer no longer builds, detects image architecture, renames, or re-uploads the image. For project version `0.11.3` it creates:
|
||||
|
||||
```text
|
||||
name=suricata_0.11.2
|
||||
name=suricata_0.11.3
|
||||
file=routeros-suricata-tzsp-arm64.tar
|
||||
root-dir=/containers/suricata_0.11.2/root
|
||||
root-dir=/containers/suricata_0.11.3/root
|
||||
```
|
||||
|
||||
The remaining deployment work is unchanged: private container bridge/VETH/NAT, environment, persistent mounts, optional RouterOS REST user/firewall integration, hybrid TZSP capture configuration, image extraction wait, container start, and final status. Existing containers are not removed. Re-running deployment for the same version stops with `Container suricata_0.11.2 already exists`.
|
||||
The remaining deployment work is unchanged: private container bridge/VETH/NAT, environment, persistent mounts, optional RouterOS REST user/firewall integration, hybrid TZSP capture configuration, image extraction wait, container start, and final status. Existing containers are not removed. Re-running deployment for the same version stops with `Container suricata_0.11.3 already exists`.
|
||||
|
||||
For SSH key authentication set:
|
||||
|
||||
@@ -715,8 +734,8 @@ The deployment uses one persistent directory outside the image root:
|
||||
Inside it the application keeps:
|
||||
|
||||
```text
|
||||
/data/ids.db SQLite, sessions and analytics snapshots
|
||||
/data/redis/ Redis persistence
|
||||
/data/ids.db SQLite, sessions, traffic archive and analytics snapshots
|
||||
/data/redis/ Redis runtime directory (persistence disabled by default)
|
||||
/data/logs/suricata/ EVE/raw Suricata logs
|
||||
/data/lib/suricata/ suricata-update feeds, cache and vendor rules
|
||||
/data/suricata/ custom rules, thresholds and update filters
|
||||
@@ -972,12 +991,12 @@ After the first deployment, when the VETH/private bridge/NAT, hybrid TZSP captur
|
||||
./scripts/upgrade-routeros-container.sh routeros-suricata-tzsp-arm64.tar
|
||||
```
|
||||
|
||||
For version `0.11.2` the second command creates:
|
||||
For version `0.11.3` the second command creates:
|
||||
|
||||
```text
|
||||
name=suricata_0.11.2
|
||||
name=suricata_0.11.3
|
||||
file=routeros-suricata-tzsp-arm64.tar
|
||||
root-dir=/containers/suricata_0.11.2/root
|
||||
root-dir=/containers/suricata_0.11.3/root
|
||||
interface=veth-ids
|
||||
envlist=IDS_ENV
|
||||
mountlists=IDS_MOUNTS
|
||||
|
||||
Reference in New Issue
Block a user