# Alert Correlation, Noise Reduction and Aggregation (Part 9)

- URL: https://wuming.si/blog/4032/
- Author: Wuming
- Published: 2025-03-18
- Language: en
- Categories: Technology, English

> Previous: [Machine Learning Detection](https://wuming.si/blog/4031) · Next: [Dynamic Sandbox](https://wuming.si/blog/4033)

By now the platform runs a row of detection engines side by side: Suricata signatures, intel matching, rule-based behavior detection and ML detection. Each of them is "correct" on its own. Put together, though, they hand the analyst a wall of alerts. One SQL injection scan produces thousands of events, the same infected host reports DGA every minute, and an admin's ordinary file upload gets flagged as a webshell.

This post covers the stretch between the detection layer and the console, where **raw alerts become something a person can handle**. It has three parts:

1. **Asset context**: knowing which machines are ours and how much each one matters.
2. **A correlation and noise-reduction engine**: checks whether attacks succeeded, removes false positives, and links related alerts into attack chains.
3. **Periodic aggregation tasks**: Celery jobs that run every minute, fold the flood of events into alerts according to rules, and then roll alerts up into "risky hosts".

```
 detection engines ──► raw events (ES)
                          │
                          ├──► correlation/noise engine: success check,
                          │        reliability score, extra context
                          ▼              │
          periodic aggregation (Celery, every minute) ◄── rules & taxonomy (knowledge base)
                          │                           ◄── asset info
                          ▼
              alerts (merged by source / destination / rule)
                          │
                          ▼
              risky hosts (rolled up per host) ──► console
```

## 1. Asset context: know who you are first

The same "brute force" alert means very different things against a test box and against a domain controller. So the platform has a middleware service dedicated to **asset discovery**. It also acts as the relay that forwards Zeek logs to the detection engines.

**Passive discovery** does most of the work, because traffic reveals a lot:

| Source | What you learn |
|---|---|
| ARP / DHCP | IP–MAC bindings, address assignment |
| DNS | Hostnames |
| HTTP User-Agent | OS and device type |
| TLS SNI / certificates | Services a host offers |
| MAC OUI | NIC vendor |

**Active discovery** (Nmap port and service probing) is optional and off by default. On many customer networks, active scanning is itself a sensitive act.

Each asset record holds first and last seen times, open services, device type, group and an **asset value**. Later, the aggregation step uses this to:

- work out **traffic direction**: internal→internal, internal→external, external→internal, external→external;
- spot **internal penetration**: when both source and destination are known assets, that is usually more worrying than an outside scan;
- flag **unknown devices**: a MAC address that has never been seen before suddenly appears.

The same service subscribes to the per-log-type channels Zeek publishes. It routes HTTP logs to web detection and DNS logs to DGA detection, adding asset fields on the way so downstream engines don't each have to query the database.

## 2. Correlation and noise reduction: from "attempted" to "succeeded"

### Checking whether the attack worked

The most effective noise filter is answering one question: **did this attack succeed?** Detection engines usually look only at the request. The correlation engine looks at the request and the response together:

| Alert type | Signs of success | Signs of failure / false positive |
|---|---|---|
| SQL injection | DB error text in the response, length far from baseline, a clear delay from time-based injection | Normal response, same length |
| XSS | Payload reflected verbatim | Payload HTML-entity-encoded |
| Command injection | Typical command-output formats in the response | No output, response unchanged |
| Webshell | Accessed soon after upload, obfuscated, calls system-command functions | Never accessed after upload, normal admin upload |
| DGA | Newly registered domain, resolves to a suspicious address | Popular site, domain registered years ago |
| Weak password | Login succeeded + privileged account | Login failed, or the password is actually strong |
| Brute force | Short window, many usernames, source on an anonymizing network | A handful of failures (probably a forgotten password) |

An illustrative verdict function (my own sketch, just to show the idea):

```python
def verdict(indicators: set) -> str:
    strong = {"db_error", "cmd_output", "payload_reflected"}
    if indicators & strong:
        return "success"
    if len(indicators) >= 2:
        return "likely"
    if indicators:
        return "possible"
    return "attempt"        # just an attempt, downgrade it
```

The verdict is written back onto the original event, which gets a success/failure flag and an adjusted reliability score. Nothing is deleted. A false-positive check can itself be wrong, and keeping the original makes review possible.

### Correlation scores and attack chains

For each new alert, the engine looks for related earlier alerts inside a time window and adds up a score along a few dimensions:

```python
def corr_score(a, b):
    s = 0.0
    if a.src == b.src:               s += 0.3
    if a.dst == b.dst:               s += 0.3
    if abs(a.ts - b.ts) < 300:       s += 0.2
    if stage_follows(a.kind, b.kind): s += 0.2   # e.g. recon -> exploit
    return s
```

High-scoring pairs then get a relationship type: reconnaissance → exploitation, injection → webshell (initial access → persistence), credential theft → lateral movement, and so on. These stages line up with the kill-chain stages in the knowledge base, and the console draws them as a timeline.

### Enrichment and intel refresh

The correlation engine also adds protocol context to alerts (HTTP URI/UA/Referer, DNS query and answer, TLS certificate, connection 5-tuple), plus WHOIS and geolocation. On a schedule it refreshes intel from public sources, such as Tor exit node lists and open-source malicious IP feeds, and syncs them into Bloom filters so the detection engines can look them up quickly.

## 3. Periodic aggregation: Celery folds events into alerts

Even after noise reduction, there is still one event per request. What an analyst wants is a sentence like **"A launched 3,000 SQL injection attempts at B in the last hour, and 2 of them succeeded."** A set of periodic Celery tasks produces that.

### How the tasks are organized

- Celery beat fires every minute. Redis is both the broker and the result backend.
- Each task owns one **threat category** (APT, malware, exploitation, DDoS, suspicious activity, weaknesses, and so on) and only runs the rules in that category.
- Tasks are **registered from config**: the config gives the class name, the source, the destination and the category filter, and they are imported dynamically at startup.
- **The aggregation rules themselves live in the knowledge base** (see Part 6). Each rule has a query, grouping keys (e.g. `src_ip,dst_ip`), severity/reliability overrides, a run interval, a category and a kill-chain stage. Operators can change rules without a release.

```yaml
# illustrative config
tasks:
  malware:
    class: MalwareAggTask
    source: raw_events
    sink: alarms
    categories: [malware]
```

### What a single run does

```
1. Pick rules: category matches && this minute falls on the rule's interval
2. Read watermark: where each rule left off last time (first run looks back 7 days)
3. Rate check: has this rule hit today's alert cap? (Redis counter cache)
4. ES aggregation: terms agg on the grouping keys; max(severity), max(reliability),
                   first/last seen, category distribution, engine distribution
5. Add assets: batch-lookup assets, work out direction and internal penetration
6. Merge: combine with existing alerts that have the same key
7. Bulk write: write back with bulk, update the counter cache, log the run
```

A few details:

- **Step back one minute.** Elasticsearch has a refresh delay, so data from "now" may not be visible yet. Each run therefore ends at "current time minus one minute".
- **Incremental watermarks.** Each rule's last end time is read back from the run log, and the next run starts there. Nothing is processed twice and nothing is skipped.
- **Merge, don't append.** An alert's key is source–destination–rule–attack result. If an alert with that key already exists, the run widens its time range (earliest start, latest end), unions the tag, category and engine sets, and adds up the event counts. Only a new key creates a new alert. That way an ongoing attack between two hosts stays **one** alert that keeps getting updated.
- **Successes get their own alert.** For attack rules, a variant that counts only successful events is derived automatically. Success and failure become two separate alerts, and the success one gets higher priority.
- **A circuit breaker.** Each rule has a daily alert cap. Once the cap is reached, that rule's aggregation pauses, so one badly written rule can't flood the whole system.
- **Back off when buckets overflow.** Elasticsearch limits how many aggregation buckets a query can return. When a query fails on that limit, it is retried with a size ten times smaller.
- **Strict worker settings.** Concurrency 1, each child process recycled after a single task, prefetch 1. That gives up a little throughput in exchange for no memory leaks and no task pile-ups.

### Layer two: risky hosts

Alerts are rolled up one more level. For each host, the platform sums the alerts it was involved in as victim or attacker, takes the highest severity and the category mix, and produces a "risky hosts" list. This is the first thing on the console's home page. Analysts start with which machines are in trouble, drill down to alerts, and then down to raw events and PCAPs.

```
raw events (millions/day) → alerts (thousands) → risky hosts (dozens)
```

## Pitfalls and lessons

- **Reduce noise in layers, not in one cut**: dedupe inside the engines → success checks in correlation → merging in aggregation → circuit breakers and rate caps. Each layer does only what it is most sure about.
- **Asset data amplifies everything.** Without asset context, every alert looks equally important. With it, "internal penetration + high-value asset" floats straight to the top.
- **Watermarks and idempotency are what aggregation depends on.** Tasks will fail, rerun and fall behind. Writing with "key + merge" means a rerun never creates duplicate alerts.
- **Keep the evidence chain.** Every alert must trace back to raw events, and every false-positive verdict needs a stated reason. Once an analyst catches the noise filter swallowing one real alert, they stop trusting it.
- **Rules in a database cut both ways.** They're quick to change and just as quick to break. The circuit breaker is the safety net for that.

That completes the network-side detection chain. One thing still can't be judged from traffic alone: the **files** reassembled from it. Next time: putting them in a dynamic sandbox and actually running them.

## The Series

1. [The Big Picture](https://wuming.si/blog/4024)
2. [10G Packet Capture](https://wuming.si/blog/4025)
3. [Suricata DPI and Rule Management](https://wuming.si/blog/4026)
4. [Protocol Analysis with Zeek](https://wuming.si/blog/4027)
5. [An Event Pipeline with Flume and Esper](https://wuming.si/blog/4028)
6. [Threat Intel Matching and the Knowledge Base](https://wuming.si/blog/4029)
7. [Rule-Based Anomaly Detection](https://wuming.si/blog/4030)
8. [Machine Learning Detection](https://wuming.si/blog/4031)
9. [Alert Correlation, Noise Reduction and Aggregation](https://wuming.si/blog/4032)
10. [Dynamic Sandbox](https://wuming.si/blog/4033)
11. [Multi-Engine AV Scanning](https://wuming.si/blog/4034)
12. [The Management Console](https://wuming.si/blog/4035)
13. [A Lightweight Sensor](https://wuming.si/blog/4036)
14. [Deployment, Versioning, Upgrade and Rollback](https://wuming.si/blog/4037)
15. [Ops Monitoring and Self-Healing](https://wuming.si/blog/4038)

---

*This series is a personal study of system architecture. Details are abstracted and anonymized; it does not describe any specific commercial product or any organization's internal configuration or data.*
