# Rule-Based Anomaly Detection (Part 7)

- URL: https://wuming.si/blog/4030/
- Author: Wuming
- Published: 2025-03-04
- Language: en
- Categories: Technology, English

> Previous: [Threat Intel Matching and the Knowledge Base](https://wuming.si/blog/4029) · Next: [Machine Learning Detection](https://wuming.si/blog/4031)

In the earlier posts we turned raw packets into structured protocol events: Zeek emits connection, HTTP, DNS and authentication logs, Suricata adds signature hits and metadata, and the intel layer matches IOCs. But a whole class of threats is neither "a known byte pattern in a packet" nor "a connection to a blacklisted domain". The problem is the **behavior** itself: one source failing SSH logins two hundred times in a minute, an admin account logging in successfully with `123456`, FTP sending a password in clear text, an internal host speaking the Stratum protocol to a mining pool.

This post covers the layer that handles that: a **rule-driven behavior detection engine**. It sits after protocol analysis and before alert correlation, subscribes to the stream of protocol events, runs a set of pluggable "detectors" built on heuristics, and emits alerts with a severity and a reliability score.

## Why signatures are not enough

A signature is great at answering "does this packet or flow contain a known malicious pattern?" These questions, though, can't be answered from a single packet:

| Question | What you need |
|---|---|
| Is this brute force? | Failure counts across many connections, over a time window |
| Did the weak password actually **work**? | The password from the request plus the status from the response |
| Are credentials sent in clear text? | The protocol, and whether it ran over TLS |
| Is a scanner enumerating something? | The distribution of paths or usernames over a short period |

All of these need **state** and **context**. That makes them a good fit for a separate detection layer written in Python: quick to develop, easy to change rules, and scaled out with multiple processes.

## Overall structure

```
   Protocol events (Zeek / Suricata metadata)
                │  Redis pub/sub or ZeroMQ
                ▼
        ┌──────────────────────┐
        │ Dispatcher (main proc)│  subscribe, deserialize, route by channel
        └───────┬──────────────┘
      ┌─────────┼──────────┐
      ▼         ▼          ▼
 [weak pwd] [brute force] [web attack] ...   one or more processes per detector
      │         │          │
      └─────────┴──────────┘
                ▼
     Alert output (message bus / Elasticsearch / syslog)
```

The main process does three things: read config, set up subscriptions, and hand messages to detectors through multiprocessing queues. Each detector runs in its own process, so a crash in one doesn't take down the others, and CPU-heavy modules can get extra workers.

## The detector framework: three hooks

Every detector extends one abstract base class, and the flow is always the same three steps (the classic Template Method pattern):

- `intercept(msg)`: pre-filter. Drop messages with missing fields, the wrong protocol, or whitelisted values, and pull out the fields needed later.
- `detect(data)`: the actual decision. Pattern matching, threshold counting and intel lookups happen here.
- `notice(data, result)`: build the alert, with severity, reliability and context.

An illustrative detector (a simplified sketch I wrote, not real code) looks like this:

```python
class ClearTextCredDetector(BaseDetector):
    channels = ["auth_events"]

    def intercept(self, msg):
        if msg.get("proto") not in ("ftp", "telnet", "pop3", "imap", "http"):
            return None
        if msg.get("tls"):
            return None            # encrypted, so not clear text
        if not msg.get("password"):
            return None
        return {"src": msg["src_ip"], "dst": msg["dst_ip"],
                "proto": msg["proto"], "user": msg.get("user")}

    def detect(self, d):
        return True                # reaching here means a hit

    def notice(self, d, hit):
        return Alert(kind="cleartext_credential", severity=2,
                     reliability=8, src=d["src"], dst=d["dst"],
                     extra={"proto": d["proto"], "user": d["user"]})
```

The base class takes care of batching, counters (messages processed, hits, latency, errors), logging and shared false-positive filtering. Whoever writes a new detector only has to think about those three methods.

## Config-driven module loading

Detectors are plugged in through configuration: the config names the class path, the channels to subscribe to and a batch size, and the main process loads the class by reflection. Adding or retiring a detector doesn't touch the core program.

```yaml
modules:
  bruteforce:
    enabled: true
    class: detectors.bruteforce.Detector
    batch_size: 100
    channels: [auth_events]
  web_attack:
    enabled: true
    class: detectors.web.Detector
    batch_size: 50
    channels: [http_events]
```

You can also start only a subset of modules from the command line, which helps when debugging or when trimming the engine down for a resource-constrained sensor.

## Run modes: one codebase, three ways in

To fit different deployment shapes, the engine abstracts where messages come from:

- **Redis mode** (default): subscribe to Redis pub/sub channels. The simplest option, and enough for a single box.
- **ZeroMQ mode**: pull directly from upstream ZeroMQ sockets (TCP or IPC). One less hop of middleware and higher throughput.
- **Stream-processing mode**: run as an Apache Storm bolt for larger distributed deployments.

If the mode isn't given explicitly, startup picks one in this order: an environment variable, then a check for whether the ZeroMQ ports are already bound, then a fallback to Redis. Upstream messages can be JSON or a compact delimiter-joined text format whose field order comes from a shared field-definition file. At high volume, the text format saves a good deal of serialization cost.

## A tour of the detectors

| Area | Detector | Idea |
|---|---|---|
| Auth | Weak password | Successful login + password in a weak-password list or with a low strength score |
| Auth | Clear-text password | Credentials seen in a protocol with no TLS |
| Auth | Brute force | Failures in a time window, distinct usernames, many sources against one target |
| Web | Web attack | Patterns and heuristics for SQL injection, XSS, command injection, path traversal, file inclusion, XXE |
| Web | Unusual HTTP methods | TRACE, WebDAV-style methods and other rare verbs |
| Web | Scanners / crawlers | User-Agent, request rate, enumeration patterns (e.g. WordPress plugin probing) |
| Web | Out-of-band (OOB) | Signs that an injection triggered a DNS/HTTP callback to an outside host |
| Payload | Shellcode | NOP sleds, common encoders, instruction-sequence patterns |
| Payload | PowerShell attack frameworks | Markers of well-known open-source attack scripts, encoded commands |
| Host | Crypto-mining trojans | Pool domains, the Stratum protocol, wallet address formats |
| Network | VPN protocol identification | OpenVPN, IPSec, L2TP, PPTP, WireGuard |

## Two detectors up close

### Weak passwords: the hard part is "did it work?"

If you alert every time you see `123456`, failed attempts and scanners will bury you. What matters is a **successful login with a weak password**. So the weak-password detector does the following:

1. **Extracts credentials from 20+ protocols**: HTTP forms and Basic auth, FTP, SSH, Telnet, mail protocols, database protocols, RDP, SMB and more.
2. **Works out whether the login succeeded**: for HTTP from the status code and redirect, for other protocols from the auth reply. Failures are dropped.
3. **Scores the password**: first against a weak-password dictionary, then with the open-source zxcvbn estimator, while also catching keyboard walks and common patterns.
4. **Tells account types apart**: a weak password on a privileged account such as admin or root is raised to a higher severity.
5. **Deduplicates and filters out scanning**: an account is reported once per time window. A source that "auto-retries" many passwords in a short burst counts as scanning, not a real login. Background requests made by the login page itself are excluded too.

### Brute force: multi-dimensional thresholds

Brute force is about statistics. The detector keeps sliding windows keyed by source → target → service:

```python
def on_failed_login(ev, win):
    key = (ev.src, ev.dst, ev.service)
    win.add(key, ev.ts, ev.user)
    stats = win.stats(key, seconds=300)
    if stats.failures >= 30:
        return "bruteforce"
    if stats.distinct_users >= 10:
        return "user_enumeration"     # dictionary-style username guessing
    return None
```

You also have to handle *distributed* brute force: many sources hit one target, each staying under the threshold. You only see that pattern when you aggregate by target instead of by source. The thresholds are config values, because environments differ a lot.

## Small tools for fewer false positives

- **A Bloom filter of popular sites**: roughly the top million domains, stored in a Bloom filter (a compact set structure with O(1) lookups and a tiny false-match rate). Low-confidence hits involving those domains are let through.
- **Intel blacklist lookups**: events that involve known C&C servers or malicious IPs get a higher reliability score.
- **Batching**: messages are fetched and checked in batches, which spreads the per-message overhead.

## Operational details

The engine has a built-in supervisor thread. It checks CPU, memory and IO on a schedule, cleans up zombie child processes, and uses a PID file so the engine can't be started twice. Each detector regularly reports its throughput, hit count, latency and error count, so the console can show which module is slowing down.

## Trade-offs and lessons

- **The strength of a rule engine is explainability.** Every alert can say exactly why, for example "47 failures in 5 minutes". For security operations that matters a lot.
- **The ceiling is just as clear.** It only catches known patterns. It can do nothing about obfuscated variants, encrypted traffic or unknown malware families. Covering those is the job of the ML engine in the next post.
- **"Did it succeed?" is worth more than "was it attempted?"** Most alert noise comes from failed scans. The earlier you separate the two, the better, and the correlation layer checks again later.
- **Don't hard-code thresholds.** Normal baselines vary hugely between networks, so thresholds have to be configurable, ideally per asset group.
- **Process isolation is worth the memory.** Detection logic written in Python will throw exceptions sooner or later. One module crashing and restarting is far better than the whole engine going down.

Next, we look at the other leg on the same framework: machine learning for DGA domains, malicious communication inside encrypted traffic, and obfuscated web attacks.

## The Series

1. [The Big Picture](https://wuming.si/blog/4024)
2. [10G Packet Capture](https://wuming.si/blog/4025)
3. [Suricata DPI and Rule Management](https://wuming.si/blog/4026)
4. [Protocol Analysis with Zeek](https://wuming.si/blog/4027)
5. [An Event Pipeline with Flume and Esper](https://wuming.si/blog/4028)
6. [Threat Intel Matching and the Knowledge Base](https://wuming.si/blog/4029)
7. [Rule-Based Anomaly Detection](https://wuming.si/blog/4030)
8. [Machine Learning Detection](https://wuming.si/blog/4031)
9. [Alert Correlation, Noise Reduction and Aggregation](https://wuming.si/blog/4032)
10. [Dynamic Sandbox](https://wuming.si/blog/4033)
11. [Multi-Engine AV Scanning](https://wuming.si/blog/4034)
12. [The Management Console](https://wuming.si/blog/4035)
13. [A Lightweight Sensor](https://wuming.si/blog/4036)
14. [Deployment, Versioning, Upgrade and Rollback](https://wuming.si/blog/4037)
15. [Ops Monitoring and Self-Healing](https://wuming.si/blog/4038)

---

*This series is a personal study of system architecture. Details are abstracted and anonymized; it does not describe any specific commercial product or any organization's internal configuration or data.*
