Building an Event Pipeline with Flume and Esper (Part 5)
Previous: Protocol Analysis with Zeek · Next: Threat Intel Matching and the Knowledge Base
Every engine covered so far produces events. Suricata emits signature alerts and file-extraction records. Zeek emits protocol logs and notices. The intel matcher emits IOC hits, the sandbox emits sample verdicts, and the asset discovery module reports software and services it finds. They use different formats and run at different rates. The consumers are just as varied. Some events go to Elasticsearch for search, some to Redis queues for the next analysis step, some to a customer’s SIEM as syslog, and occasionally some to Kafka.
Wiring every producer straight to every consumer gives you N×M connections. We put an event pipeline in the middle instead. Apache Flume handles collection, enrichment and routing, and an embedded Esper engine handles time-window aggregation and multi-step correlation. This post walks through that pipeline.
Flume in three pieces
Flume is Apache’s log-collection framework, and its model is simple:
- Source receives events from outside.
- Channel is a buffer that decouples the receive rate from the send rate.
- Sink writes events to a destination.
Between the source and the channel you can attach a chain of interceptors, which do light per-event processing. One Flume process (an “agent”) can host several source/channel/sink groups. When the built-in components aren’t enough, you write your own in Java and drop the jar in. Most of our components were built that way.
producers (detection engines)
│ ZeroMQ (IPC)
▼
Source ──► interceptor chain ──► Channel ──► Sink ──► ES / Redis / Syslog / Kafka
│
└──► Esper: aggregate / correlate ──► new events
Input: ZeroMQ sources
The engines and the pipeline usually run on the same box, so we use ZeroMQ over IPC (Unix domain sockets) between processes. There’s no broker, latency is low and throughput is high. Each event family gets its own socket and topic: “IPS alert”, “IOC hit”, “file extracted”, “sandbox verdict”, “asset notice”, and so on.
One design choice is worth calling out. Live traffic and offline analysis (replaying a pcap a user uploaded) go through separate sources. An offline job that dumps a flood of events at once can’t crowd out the real-time alert path, and downstream consumers always know where an event came from.
A custom source looks roughly like this (illustrative):
agent.sources.alerts.type = com.example.flume.ZmqSubSource
agent.sources.alerts.endpoint = ipc:///run/pipeline/alerts.sock
agent.sources.alerts.topic = ips_alert
agent.sources.alerts.channels = c_alert
Enrichment: the interceptor chain
Before an event reaches a channel, it passes through a chain of interceptors:
| Interceptor | What it does |
|---|---|
| GeoIP | Resolves source/destination IPs to country, city and coordinates (MaxMind-format mmdb) |
| Filename | Normalizes file names and extensions and fixes encodings |
| Email MIME | Extracts sender, recipients, attachment names and other mail fields |
| Engine-specific | Fills in missing fields and normalizes names for one engine’s event format |
| Communication ID | Tags events with a session-level ID so events about one conversation can be linked across sources |
agent.sources.files.interceptors = norm fname geo mime commid
agent.sources.files.interceptors.geo.type = com.example.flume.GeoIpInterceptor$Builder
agent.sources.files.interceptors.geo.ipFields = src_ip dst_ip
One lesson: normalize fields at the entrance. Zeek calls a field id.orig_h (see the last post), Suricata calls it src_ip, and another engine might call it sip. We keep a protocol field-mapping table and rename everything in an interceptor. That way the aggregation and correlation rules downstream only have to be written once.
Buffering: choosing a channel type
We mostly use memory channels, one per event family. Alert channels get the biggest capacity (around a million events). Sandbox and file events are much lower volume, so their channels are an order of magnitude smaller.
| Type | Pros | Cons |
|---|---|---|
| Memory channel | Fast, almost no overhead | Lost on crash; capacity bounded by the JVM heap |
| File channel | Persisted to disk, survives restarts | Costs disk IO, lower throughput |
Protocol metadata is high-volume and can tolerate a little loss, so a memory channel is a reasonable fit. For high-severity alerts, where you don’t want to lose even one, a file channel is worth the cost, or the sink should write into a queue that persists to disk.
Aggregation: compressing events with Esper
Esper is a CEP (Complex Event Processing) engine. You write a SQL-like language (EPL) that says “within this time window, compute this over that event type”, and Esper evaluates it continuously in memory and emits results.
Why aggregate at all? Because there are far too many raw events. A client can hit the same site hundreds of times in 30 seconds, and one IPS signature can fire thousands of times between the same two hosts. Storing all of that as-is buries both the storage and the analyst UI. We wrote one aggregation statement per protocol and alert type. A typical pair looks like this:
-- HTTP: merge same peers, site, method and status within 30 s
select src_ip, dst_ip, dst_port, host, method, status_code,
count(*) as count, uri, user_agent, start_time
from http_event.win:time_batch(30 sec)
group by src_ip, dst_ip, dst_port, host, method, status_code
output first every 30 sec
-- IPS alerts: merge same IP pair and signature within 30 s
select src_ip, dest_ip, signature, severity, category,
count(*) as count, start_time
from ips_alert.win:time_batch(30 sec)
group by src_ip, dest_ip, signature
output first every 30 sec
The key parts:
win:time_batch(30 sec)is a tumbling 30-second batch window.group bydefines what counts as “the same thing”. It’s the knob that trades compression against lost detail.count(*)records how many events were merged, so that information isn’t simply thrown away.output firstemits the detail fields from the first event in each group.
The list of event types covered is long: web, DNS, mail (SMTP/IMAP/POP3), file transfer (FTP/SMB), authentication (NTLM/Kerberos/RADIUS), remote access (RDP/VNC/SSH), industrial protocols (Modbus/DNP3), VoIP, and every alert type. Under bursty traffic, aggregation often cuts event volume by one to two orders of magnitude.
Correlation: multi-step rules
Aggregation deals with volume. Correlation deals with meaning. One alert on its own often says little, but “A, then B, between the same two hosts, within one day” says a lot.
Our correlation rules are declarative. Each rule is made of steps, and each step defines:
- Condition: which events match (by alert category, rule ID range or field values).
- Keys: which fields must be equal across steps, e.g.
src_ip,dst_ip. - Window: anywhere from minutes to days.
- Counts: a minimum number of occurrences (mintimes) and a cap on how many to count (maxtimes).
- Reliability: how much this step adds to a “compromised” verdict.
- Attack result: reconnaissance, probing, or compromise.
Each rule as a whole also carries a severity, a threat category and a kill-chain phase. Here’s a made-up example:
rule: tls-fingerprint-then-malware-download
severity: high
killchain: command-and-control
steps:
- name: suspicious_tls
when: category == "malicious-tls-fingerprint"
keys: [src_ip, dst_ip]
window: 1d
reliability: 5
- name: malware_file
when: category == "malware-file"
keys: [src_ip, dst_ip]
window: 1d
min_times: 1
reliability: 8
result: compromise
Under the hood, each step is translated into an Esper pattern statement (along the lines of pattern [every a=... -> b=... where timer:within(...)]). A match produces a new “correlated alert”, which goes back into the pipeline and is routed like any other alert. Heavier cross-source correlation and noise reduction live in a separate correlation engine, covered in Part 9.
Output: routing by type
| Destination | Purpose |
|---|---|
| Elasticsearch | Long-term storage and search for alerts, metadata and sandbox verdicts |
| Redis list | Work queue, e.g. file-extraction events consumed one by one by downstream analyzers |
| Syslog | Forwarding to third-party SIEMs in an agreed format |
| Kafka (optional) | High-throughput bus for large deployments |
All sinks write in batches (say 1,000 events or 5 seconds, whichever comes first). The ES sink also has a connection pool and async worker threads.
Lessons learned
- “Channel is full.” When a destination slows down (ES merging segments, a full disk), the channel fills fast and the source starts rejecting events. Raising capacity buys time. The real fix is a sink that keeps up: bigger batches, more threads, or moving the slow destination to its own channel so one slow sink can’t stall everything else.
- Elasticsearch goes read-only. Once disk usage passes the flood-stage watermark, ES marks indices read-only and the sink errors nonstop. You need disk monitoring and index lifecycle management up front. Unlocking indices by hand after an outage isn’t a plan.
- JVM heap. Memory channels and Esper windows both live on the heap. When you raise capacities, raise the heap with them, and use G1 to keep GC pauses short.
- Window length is a trade-off. Longer windows compress more but delay alerts more. Users barely notice the difference between 15 and 30 seconds. Above 60 seconds they will.
- Correlation rules fail silently. If a rule names a field that doesn’t match the real events, you get no error. The rule just never fires. That’s why field normalization at the entrance matters so much. It also helps to have regression tests that run the rules against replayed data.
- ZeroMQ’s slow-joiner problem. With PUB/SUB, messages published before a subscriber finishes connecting are lost, and so are messages beyond the high-water mark when a subscriber falls behind. Be deliberate about startup order and HWM settings. Also get the permissions on IPC socket files right. Wrong permissions show up as “connected but receiving nothing”.
Wrap-up
This pipeline is the platform’s central nervous system. Events from every engine get uniform field names and added context here, are compressed by time window, are correlated by multi-step rules, and are then routed to storage, queues and external systems. Flume gives us a pluggable framework for moving events in and out, and Esper gives us declarative stream computation. Together they made the pipeline fairly cheap to build.
Next we look at one of the pipeline’s key producers: the threat intel matcher, along with the knowledge base that supplies the whole platform’s taxonomy, severities and rules.
The Series
- The Big Picture
- 10G Packet Capture
- Suricata DPI and Rule Management
- Protocol Analysis with Zeek
- An Event Pipeline with Flume and Esper
- Threat Intel Matching and the Knowledge Base
- Rule-Based Anomaly Detection
- Machine Learning Detection
- Alert Correlation, Noise Reduction and Aggregation
- Dynamic Sandbox
- Multi-Engine AV Scanning
- The Management Console
- A Lightweight Sensor
- Deployment, Versioning, Upgrade and Rollback
- Ops Monitoring and Self-Healing
This series is a personal study of system architecture. Details are abstracted and anonymized; it does not describe any specific commercial product or any organization’s internal configuration or data.
Comments