# Inside a Network Threat Detection Platform: The Big Picture (Part 1)

- URL: https://wuming.si/blog/4024/
- Author: Wuming
- Published: 2025-01-21
- Language: en
- Categories: Technology, English

> Next: [10G Packet Capture](https://wuming.si/blog/4025)

Picture a copy of every packet crossing a company's core switch, arriving at 10 Gbps and up, around the clock. Somewhere in that stream is a phishing attachment, a beacon to a command-and-control server, or an attacker quietly testing passwords on an internal file server. The job of an ATD (Advanced Threat Detection) platform is to find those few needles, tie them together into a story, and hand an analyst something they can act on, without drowning them in noise.

No single tool does all of that. A signature engine sees one packet stream at a time. A threat-intel lookup has no idea what happened five minutes ago. A sandbox is slow and can only look at files. So a real platform ends up as a layered system: fast, dumb-but-precise layers at the bottom, slower and smarter layers toward the top, and a pipeline in between that turns raw traffic into events, events into alerts, and alerts into incidents.

This series takes such a system apart layer by layer. This first post is the map: what the layers are, how data flows through them, what two real attack scenarios look like end to end, and how the whole thing is deployed. Every later post zooms in on one box in the diagram below.

## Design goals

Before the boxes, the constraints that shaped them:

| Goal | What it forces |
|---|---|
| Keep up with line rate (10G+) | Kernel-bypass capture, per-core workers, no per-packet work above the DPI layer |
| Catch multi-stage attacks | A correlation layer that remembers hosts and events across hours or days |
| Low false-positive rate | Several independent detectors that must agree, plus asset context and whitelists |
| Explain every alert | Each rule and intel hit maps to a category, severity, and kill-chain stage |
| Run on one box or fifty | Components talk over message buses, not in-process calls |
| Survive in unattended sites | Process supervision, self-healing, versioned upgrades with rollback |

The key idea is a **funnel**. Each layer reduces volume by one or two orders of magnitude and adds context. Packets (millions per second) become protocol records and alerts (thousands per second), which become correlated threats (a handful per hour).

## The layered architecture

Here is the whole platform on one page. The numbers on the right point to the post in this series that covers each part.

```
          SPAN port / network TAP / virtual switch mirror
                              │  raw packets, 10G+
┌─────────────────────────────▼────────────────────────────────┐
│ 1. CAPTURE   NIC → PF_RING ZC → flow-hash load balancer      │  Part 2
└──────────────┬──────────────────────────────┬────────────────┘
               │ per-core packet queues       │
┌──────────────▼───────────────┐  ┌───────────▼────────────────┐
│ 2a. DPI ENGINE (Suricata)    │  │ 2b. PROTOCOL ENGINE (Zeek) │  Part 3, 4
│  signatures · app-layer logs │  │  rich protocol events      │
│  file extraction             │  │  scriptable logic          │
└───┬───────────────┬──────────┘  └───────────┬────────────────┘
    │ files         │ alerts + metadata       │ events
    │               │ (ZeroMQ)                │ (Redis)
    │       ┌───────▼─────────────────────────▼──────────────┐
    │       │ 3. EVENT PIPELINE  Flume + Esper CEP → Kafka   │  Part 5
    │       └───────┬────────────────────────────────────────┘
    │               │ normalized event streams
    │       ┌───────▼────────────────────────────────────────┐
    │       │ 4. DETECTORS                                   │
    │       │   threat-intel matching   (Part 6)             │
    │       │   rule-based anomaly      (Part 7)             │
    │       │   machine learning        (Part 8)             │
    │       └───────┬────────────────────────────────────────┘
    │               │ candidate threats
    │       ┌───────▼────────────────────────────────────────┐
    │       │ 5. CORRELATION  assets · kill chain · scoring  │  Part 9
    │       │    false-positive reduction · aggregation      │
    │       └───────▲───────────────┬────────────────────────┘
┌───▼───────────────┴──┐            │ incidents
│ 6. FILE ANALYSIS     │ verdicts   │
│  sandbox (QEMU/KVM)  │            │                          Part 10, 11
│  multi-engine AV     │            │
└──────────────────────┘            │
┌────────────────────────────────────▼───────────────────────────┐
│ 7. STORAGE  Elasticsearch (events) · MySQL (config, knowledge) │
│             Redis (cache, queues, live counters)               │
└────────────────────────────────────┬───────────────────────────┘
┌────────────────────────────────────▼───────────────────────────┐
│ 8. CONSOLE   Django REST API · SPA front end · Kibana · Celery │  Part 12
└────────────────────────────────────────────────────────────────┘

 Cross-cutting:  knowledge base (rules, intel, taxonomy)    Part 3, 6
                 lightweight remote sensors                 Part 13
                 packaging, versioning, upgrade/rollback    Part 14
                 monitoring and self-healing                Part 15
```

Let's walk through it from the bottom up.

### 1. Capture

A standard Linux network stack copies each packet several times and takes an interrupt per batch. At 10 Gbps of small packets that is roughly 15 million packets per second, about 67 nanoseconds per packet, which is far less than the stack needs. The capture layer uses **PF_RING ZC** (a kernel-bypass library where the NIC writes packets by DMA straight into memory the application can read, "zero copy") and a balancer process that hashes each packet by its flow and drops it into one of N per-core queues. Every packet of a given connection lands on the same worker, which is what lets the next layer keep state.

### 2. Deep packet inspection and protocol analysis

Two engines read those queues.

- **Suricata** is the workhorse. It reassembles TCP streams, parses application protocols (HTTP, DNS, TLS, SMB, SMTP and more), runs large signature sets against them, and extracts files that cross the wire. It emits three things: **alerts** (a signature fired), **metadata** (one record per HTTP request, DNS query, TLS handshake, flow), and **files** written to disk.
- **Zeek** takes a different angle. It does not ship with a big signature set; it turns traffic into a high-level event log and lets you write detection logic as scripts. It is good at things like "this host has made 400 failed SSH logins".

Running both costs CPU, but they complement each other: Suricata is precise on known bad patterns, Zeek is flexible for behavioral logic.

### 3. Event pipeline

Everything above this line is C and runs per packet. Everything below is per event. The bridge is an **Apache Flume** agent with custom **ZeroMQ** sources (ZeroMQ is a lightweight brokerless messaging library). Flume parses the JSON events, normalizes field names and timestamps, runs them through an **Esper** CEP interceptor (CEP, complex event processing, means continuous queries over sliding time windows), and writes them to **Kafka** topics and Elasticsearch.

The Esper stage catches simple "count over time" patterns early and cheaply. An illustrative rule:

```sql
-- illustrative only: many distinct ports from one source in a minute
SELECT src_ip, COUNT(DISTINCT dest_port) AS ports
FROM AlertEvent.win:time(60 sec)
WHERE category = 'scan'
GROUP BY src_ip
HAVING COUNT(DISTINCT dest_port) > 100
```

### 4. Detectors

Three families of detectors consume the normalized streams. Each answers a different question.

| Detector | Question it answers | Typical input |
|---|---|---|
| Threat-intel matcher | "Have we seen this domain, IP, URL, hash or certificate on a known-bad list?" | DNS, HTTP, TLS, mail, file hashes |
| Rule-based anomaly engine | "Does this behavior match a known attack pattern?" (brute force, auth abuse, odd payloads) | protocol records, auth events |
| ML engine | "Does this look like something we have no rule for?" (DGA domains, phishing pages, malicious encrypted flows) | DNS names, URLs, TLS and flow features |

They are deliberately independent. When two of them agree about the same host, confidence goes up a lot.

### 5. Correlation

This is the "brain". It keeps an **asset inventory** built passively from traffic (which IPs are servers, which run which services, what is normal for them), and it remembers recent detections per host. Its job is to:

1. **Deduplicate**: the same file may trigger a signature, an intel hit and an AV verdict. That is one incident, not three.
2. **Chain**: map each event to a **kill-chain** stage (reconnaissance, delivery, exploitation, installation, command and control, actions on objectives) and link events on the same host or attacker over time.
3. **Score**: combine severity, confidence, asset importance and stage into a risk score.
4. **Suppress**: apply whitelists and known-benign patterns.
5. **Aggregate**: periodic jobs roll similar alerts into one alarm per time window.

A simplified scoring idea:

```python
# illustrative: not a real formula from any product
def risk(severity, confidence, asset_value, stage_weight):
    base = severity * confidence * asset_value      # each normalized 0..1
    return round(100 * min(1.0, base * stage_weight))
```

### 6. File analysis

Files extracted by the DPI layer go to two analyzers. A **dynamic sandbox** built on **QEMU/KVM** runs the sample in a disposable VM matched to its type (Windows for PE files and Office documents, Linux for ELF, an emulator for APKs), records process, file, registry and network behavior, and scores it. A **multi-engine AV scanner** runs the same file through several commercial engines plus open-source **ClamAV** and fuses the verdicts. AV is fast and catches known samples; the sandbox is slow but catches new ones.

### 7. Storage

Three stores, each used for what it is good at:

- **Elasticsearch**: all time-series data (alerts, protocol metadata, flows, correlated threats, sandbox reports), in daily indices with hot/warm/cold retention.
- **MySQL**: configuration, users and permissions, assets, and the **knowledge base**: the rule repository, threat taxonomy, kill-chain stages, CVE data, and intel.
- **Redis**: intel lookup cache, sessions, Celery broker and results, and live counters for dashboards.

### 8. Console

A **Django** REST backend with role-based access control, a single-page front end, **Kibana** for free-form hunting, and **Celery** workers for scheduled jobs (reports, risk recalculation, alarm e-mails, cache refresh). Analysts live here.

## How data flows

It helps to think of three planes:

```
 HOT PATH (per packet, C, per-core)
   NIC ─► PF_RING ─► Suricata / Zeek
                         │
 EVENT PLANE (per event, JVM + Python, message buses)
   ZeroMQ ─► Flume/Esper ─► Kafka ─► detectors ─► correlation
                                                       │
 ANALYST PLANE (per query, web + batch)
   Elasticsearch / MySQL / Redis ◄─ Django + Celery ◄─ browser
```

A few things about the joints are worth calling out:

- **ZeroMQ between the engines and the pipeline.** The engines must never block on a slow consumer. ZeroMQ PUB/SUB is fire-and-forget: if Flume falls behind, messages are dropped at the edge instead of stalling packet processing. Losing some metadata is acceptable; dropping packets is not.
- **Kafka inside the pipeline.** After normalization, durability matters. Kafka lets the detectors and the correlation engine replay, restart and scale independently.
- **One normalized event shape.** Every engine's output is mapped onto a common schema early, so downstream code never cares whether an event came from Suricata or Zeek:

```json
{
  "ts": "2026-01-14T09:15:02.113Z",
  "sensor": "sensor-a",
  "kind": "alert",
  "src_ip": "<redacted>", "dest_ip": "<redacted>", "dest_port": 443,
  "proto": "tls",
  "sig_id": 1000001,
  "category": "trojan-activity",
  "severity": 2,
  "attack_stage": "command-control",
  "flow_id": 8812734412
}
```

- **The knowledge base as the dictionary.** Engines emit terse identifiers (a signature ID, an intel list name). The knowledge base turns those into human meaning: category, severity, description, remediation advice, kill-chain stage. Keeping that mapping in one place is what makes alerts consistent across engines.

## Example 1: a malicious download

A user clicks a link in an e-mail and downloads an executable over HTTP.

```
T+0s    HTTP response carries an .exe
T+1s    Suricata: reassembles stream, extracts file, computes hashes,
        a "generic downloader" signature fires → alert + fileinfo via ZeroMQ
T+2s    Flume: normalizes, writes to Kafka and Elasticsearch
T+3s    Intel matcher: SHA-256 is on a known malware-family list → hit
        Multi-AV: most engines flag it → "malicious, medium-high confidence"
T+5s    Correlation: three signals, one file, one host → one incident
        stage = installation, risk = high → submit to sandbox
T+5min  Sandbox: drops a copy of itself in a temp dir, sets a Run key,
        injects into a system process, connects out over 443 → score critical
T+5min  DPI: a TLS connection from the same host to the address the
        sandbox saw → correlation links it, stage = command-control
T+6min  Incident raised to critical; e-mail + syslog to SIEM;
        optional response hook (block the address, quarantine the host)
```

Notice the order. The cheap, fast signals (signature, hash lookup, AV) come in seconds and are enough to raise an alert. The expensive one (sandbox) runs only on files that already look suspicious, and its output becomes a new **IOC** (indicator of compromise) that the DPI and intel layers can watch for on the network.

## Example 2: a slow, multi-stage intrusion

The harder case is an attacker who never trips a single high-severity rule.

| Time | What happens | Who sees it | Kill-chain stage | Risk |
|---|---|---|---|---|
| Day 1, 09:15 | External host scans common ports across an internal subnet | Suricata scan signatures; Esper counts distinct ports | Reconnaissance | Low |
| 14:32 | HR employee gets a spreadsheet with a macro, from a look-alike domain | Intel matcher (sender domain); DPI extracts the file; AV is unsure (obfuscated) | Delivery | Medium |
| 14:45 | Macro downloads a payload over HTTP | Suspicious-download signature; sandbox sees encoded PowerShell and a scheduled task | Exploitation / Installation | High |
| 15:00 | Workstation starts beaconing every 60 s over TLS | ML flags the encrypted flow; intel matches the certificate | Command and control | Critical |
| 16:30 | Workstation tries SMB logons to 15 internal hosts | Anomaly engine: many auth attempts fan-out | Actions on objectives | Critical |

Each row on its own is something a busy analyst might ignore. The correlation layer's value is in linking them: same external address in rows 1 and 2, same workstation in rows 3 to 5, and stages that move forward in the right order. The result is one incident with a timeline, not five unrelated alerts.

## Deployment shapes

The same components are packaged three ways.

```
 ALL-IN-ONE APPLIANCE              DISTRIBUTED
 ┌─────────────────────┐           ┌──────────┐ ┌──────────┐ ┌──────────┐
 │ capture + DPI       │           │ sensor A │ │ sensor B │ │ sensor C │
 │ pipeline + detectors│           │ capture  │ │ capture  │ │ light    │
 │ correlation         │           │ DPI      │ │ DPI      │ │ sensor   │
 │ sandbox + AV        │           │ pipeline │ │ pipeline │ │          │
 │ ES + MySQL + Redis  │           └────┬─────┘ └────┬─────┘ └────┬─────┘
 │ console             │                └──────┬─────┴────────────┘
 └─────────────────────┘                       ▼
  one 2-socket server,              central: ES cluster, correlation,
  one 10G link                      console, knowledge base
                                    analysis cluster: sandbox farm, AV pool
```

- **All-in-one**: one two-socket server with a 10G capture card. CPU cores are partitioned: some for the OS, a block for DPI workers, a block for Zeek, the rest for storage and web. Good for a single site.
- **Distributed**: capture and DPI sit at each site, close to the traffic; heavy analysis (sandbox farm, AV pool) and storage are centralized. Only events and files cross the WAN, never raw packets.
- **Lightweight sensor**: for branch offices or cloud VMs where a full box is overkill. It captures and extracts metadata with a small footprint and forwards events to the center (Part 13).

## Trade-offs and lessons

- **The funnel is everything.** Most performance problems come from doing work at the wrong layer. A regex on every packet belongs in Suricata with a fast-pattern prefilter, not in Python downstream. A rule about "the same host over three days" belongs in correlation, not in the DPI engine.
- **Prefer independent signals to one clever one.** A hash hit plus a signature plus an AV verdict is far more trustworthy than a single high-scoring model output. Design detectors so their agreement means something.
- **Drop at the edge, not in the middle.** It is better to lose metadata at the engine boundary under overload than to back-pressure the packet path. Then make every drop visible in monitoring.
- **Volume lives in metadata, not alerts.** Flow and protocol logs are 10 to 100 times larger than alerts. Retention, index design and disk sizing are driven by metadata.
- **Many components means version discipline.** Dozens of packages (engines, rules, models, web, schemas) must move together. A version manifest and a tested rollback path are not optional (Part 14).
- **Unattended boxes must heal themselves.** Process supervision, resource metrics and auto-restart turn a 3 a.m. crash into a log line (Part 15).

## What's next

Everything above depends on not losing packets at the bottom. In the next post we go down to the NIC and look at how PF_RING ZC, flow hashing and CPU pinning keep up with 10 Gbps.

## The Series

1. [The Big Picture](https://wuming.si/blog/4024)
2. [10G Packet Capture](https://wuming.si/blog/4025)
3. [Suricata DPI and Rule Management](https://wuming.si/blog/4026)
4. [Protocol Analysis with Zeek](https://wuming.si/blog/4027)
5. [An Event Pipeline with Flume and Esper](https://wuming.si/blog/4028)
6. [Threat Intel Matching and the Knowledge Base](https://wuming.si/blog/4029)
7. [Rule-Based Anomaly Detection](https://wuming.si/blog/4030)
8. [Machine Learning Detection](https://wuming.si/blog/4031)
9. [Alert Correlation, Noise Reduction and Aggregation](https://wuming.si/blog/4032)
10. [Dynamic Sandbox](https://wuming.si/blog/4033)
11. [Multi-Engine AV Scanning](https://wuming.si/blog/4034)
12. [The Management Console](https://wuming.si/blog/4035)
13. [A Lightweight Sensor](https://wuming.si/blog/4036)
14. [Deployment, Versioning, Upgrade and Rollback](https://wuming.si/blog/4037)
15. [Ops Monitoring and Self-Healing](https://wuming.si/blog/4038)

---
