A Lightweight Sensor: Extending Capture to Branches and the Cloud (Part 13)
Previous: The Management Console · Next: Deployment, Versioning, Upgrade and Rollback
The first twelve parts of this series described one big box: 10G capture, two DPI engines, an event pipeline, intel matching, machine learning, a sandbox, multi-engine AV and a management console, all on one server (or a pair). Real networks aren’t that tidy. A branch office has one small server. A cloud VPC has no physical tap to plug into. An edge gateway counts its memory in hundreds of megabytes. Shipping the full platform to each of those places would be expensive and mostly wasted.
This post covers the outermost layer of the platform: the lightweight sensor. It has one job. It captures traffic at a remote site, pulls out metadata, and gets that metadata back to the central node reliably. Detection, correlation, storage and the UI all stay at the center.
Design goals: keep only what you need
The constraints were simple:
| Dimension | Target |
|---|---|
| Memory | Under 500 MB total |
| CPU | Low single-digit percent of one core per Gbps |
| Local dependencies | Suricata plus one Python agent. No local Elasticsearch or MySQL |
| Network direction | Outbound only. The sensor opens no inbound ports |
| Outage tolerance | Buffer locally when the center is unreachable, replay on recovery |
| Install | One package, a few minutes of configuration |
That meant dropping a lot. There’s no PF_RING ZC: the sensor uses the kernel’s built-in AF_PACKET, so it runs without special drivers. There’s no file extraction and no Zeek. The rule set is either a small essential subset or nothing at all.
Three layers
Remote host (sensor) Central node
┌──────────────────────────┐ ┌───────────────────────┐
│ (1) Capture │ │ Event pipeline │
│ Suricata / AF_PACKET │ │ (ZeroMQ source) │
│ EVE JSON output │ │ │ │
│ │ │ │ ▼ │
│ ▼ │ ZeroMQ / │ Correlation / intel │
│ (2) Event processor │ TLS syslog │ │ │
│ filter → enrich → buf │ ────────────▶ │ ▼ │
│ │ │ │ Elasticsearch / UI │
│ ▼ │ └───────────────────────┘
│ (3) Transport │
│ batch, compress, │
│ heartbeat, reconnect │
└──────────────────────────┘
(1) Capture. Suricata runs in AF_PACKET mode with 2–4 worker threads. AF_PACKET is Linux’s built-in fast capture interface. It needs no vendor driver, so it works on cloud instances and VMs. All output goes through EVE JSON, Suricata’s structured event log with one JSON object per line.
(2) Event processor. A Python daemon reads the EVE output continuously and does three things:
- Filter. It keeps only the event types we care about (alert, dns, http, tls, flow) and drops alerts below a severity threshold.
- Enrich. It stamps every event with sensor ID, hostname, location and group. Without these tags the center couldn’t tell which branch a DNS query came from.
- Buffer. Events go into a local SQLite queue first. A separate sender thread drains it.
(3) Transport. ZeroMQ is the primary channel and syslog over TLS (RFC 5424) is the fallback. Events are batched (100 events or 5 seconds, whichever comes first), compressed with zlib and sent. A separate thread sends a heartbeat every 60 seconds with the buffer backlog and uptime.
A trimmed-down config looks roughly like this (illustrative):
[sensor]
sensor_id = branch-a-01
location = branch-a
[capture]
interface = eth1
mode = af_packet
[processing]
event_types = alert,dns,http,tls,flow
min_severity = 2
batch_size = 100
batch_timeout_sec = 5
[buffer]
max_size_mb = 1024
[transport]
primary = zeromq
heartbeat_sec = 60
What the center has to add
A thin sensor means the center picks up the slack:
- A receiver. We added a dedicated agent to the event pipeline with one ZeroMQ source and one TLS syslog source. It fans out to Kafka for the correlation engine and to Elasticsearch for search. This is the same pipeline machinery from Part 5 with one extra entry point.
- A sensor registry. One table holds each sensor’s ID, location, last heartbeat, status (online / offline / error) and version. A second table stores the time series of heartbeat stats: events sent, backlog, CPU and memory.
- A console page. The management console lists every sensor and its health. The center decides when a sensor counts as offline, based on how long it’s been since the last heartbeat.
With those three pieces you can answer the two questions operators ask most: “How many sensors are up right now?” and “Why did data from that branch stop?”
Observations from a real trial
We ran a trial on an ordinary VM attached to the mirror port of a small office network, with metadata only and no detection rules loaded:
| Metric | Result |
|---|---|
| Events in the first 10 minutes | Tens of thousands, 100% forwarded |
| Average rate | ~90 events/s, peaks hit the rate limiter |
| Memory | ~75 MB Suricata + ~20 MB agent, under 100 MB total |
| CPU | A little over 2% combined |
| Egress | 10–20 Mbps |
The sensor hit its resource targets easily. Running without rules turned out to be the more interesting decision. The sensor ships DNS, HTTP, TLS and flow metadata, and detection happens at the center. Intel matching, DGA detection and correlation only need that metadata. Keeping rules off the sensor means there are no rule updates to push to the edge. The cost is a bigger event stream at the center.
Pitfalls
A smooth trial doesn’t prove the design is right. Looking back, these are the issues worth calling out.
1. ZeroMQ PUB/SUB is not reliable delivery
This was the big one. A PUB socket is fire-and-forget. If there’s no subscriber, if the subscriber hasn’t finished connecting, or if the high-water mark is full, messages are dropped silently and send() still reports success. Our sender deleted events from the buffer as soon as a send succeeded, so the local buffer gave no protection in exactly the case it was built for: the center being down.
There are two ways to fix this:
- Switch to PUSH/PULL or DEALER/ROUTER and add an application-level ack. The center confirms each batch, and the sensor deletes only after the ack arrives.
- Or use ZeroMQ only as a fast path and rely on an acknowledged protocol for reliability. TCP syslog with TLS at least notices when the connection drops.
# Illustrative: only clear the buffer after an ack
batch = buffer.peek(100)
sock.send_json({"seq": seq, "events": batch})
if sock.poll(timeout=3000) and sock.recv_json().get("ack") == seq:
buffer.delete(batch)
2. Event ID collisions
The buffer table used the event ID as its primary key and deduplicated with INSERT OR IGNORE. The ID was a hash of timestamp, flow ID and event type. One flow can produce several events of the same type at the same instant, such as multiple records in one DNS answer or several HTTP transactions on one connection. Those were treated as duplicates and thrown away. Hashing the full raw JSON line, or just using an auto-increment sequence, avoids this.
3. Who owns Suricata’s lifecycle?
In the original design the agent started and stopped Suricata itself. In deployment this clashed with an existing Suricata service, so we split them. The agent now only reads and ships events, and systemd manages Suricata. The lesson: one process, one job, and leave lifecycle management to the init system. Business logic shouldn’t double as a process supervisor.
4. Reading a file vs. reading a socket
The design assumed Suricata would write to a Unix socket. For simplicity the deployment wrote to a file instead, and the agent became a tail -f: open the file, seek to the end, read new lines. That causes two problems. Events written before the agent started are skipped. And after log rotation, the agent still holds the old file handle, so it stops seeing new data without raising an error. The fix is to persist the read offset and check the inode periodically, reopening the file when it changes.
5. Rate limiting and the “dropped” counter
The sensor caps events per second to protect the center. The trial stats showed zero dropped events, yet a footnote said events over the threshold were discarded. That means the drop path wasn’t counted. Rate-limited drops need their own counter, reported in the heartbeat. Otherwise operators will always see “100% forwarded.”
6. TLS can’t be “off for now”
TLS was disabled during the trial for convenience. But sensors sit in the least controlled networks you have (branches, public cloud), and the events contain internal IPs and visited domains. Sending them in cleartext hands a map of your internal network to anyone on the path. TLS, ideally with mutual certificates, should be on by default and require an explicit opt-out.
Trade-offs at a glance
| Choice | You get | You pay |
|---|---|---|
| AF_PACKET instead of PF_RING ZC | Runs on any host | Lower per-box ceiling. Full 10G needs more cores |
| Metadata only, detection at center | No rule maintenance on the edge | More events and bandwidth at the center |
| Local SQLite buffer | Replay after outages | Disk caps and an eviction policy (drop oldest 10% when full) |
| Heartbeat + registry | Center sees every sensor’s state | Extra state management and offline-detection logic |
A lightweight sensor splits capture from analysis. Once you split them, three things you used to get for free have to be designed explicitly: reliable delivery, identity, and remote observability.
Next time we go back to the central node and look at how all these components get installed on one machine, pinned with a version manifest, and upgraded and rolled back safely.
The Series
- The Big Picture
- 10G Packet Capture
- Suricata DPI and Rule Management
- Protocol Analysis with Zeek
- An Event Pipeline with Flume and Esper
- Threat Intel Matching and the Knowledge Base
- Rule-Based Anomaly Detection
- Machine Learning Detection
- Alert Correlation, Noise Reduction and Aggregation
- Dynamic Sandbox
- Multi-Engine AV Scanning
- The Management Console
- A Lightweight Sensor
- Deployment, Versioning, Upgrade and Rollback
- Ops Monitoring and Self-Healing
This series is a personal study of system architecture. Details are abstracted and anonymized; it does not describe any specific commercial product or any organization’s internal configuration or data.
Comments