Wuming 無名

Blog

Rule-Based Anomaly Detection (Part 7)

· Technology · English

Previous: Threat Intel Matching and the Knowledge Base · Next: Machine Learning Detection

In the earlier posts we turned raw packets into structured protocol events: Zeek emits connection, HTTP, DNS and authentication logs, Suricata adds signature hits and metadata, and the intel layer matches IOCs. But a whole class of threats is neither “a known byte pattern in a packet” nor “a connection to a blacklisted domain”. The problem is the behavior itself: one source failing SSH logins two hundred times in a minute, an admin account logging in successfully with 123456, FTP sending a password in clear text, an internal host speaking the Stratum protocol to a mining pool.

This post covers the layer that handles that: a rule-driven behavior detection engine. It sits after protocol analysis and before alert correlation, subscribes to the stream of protocol events, runs a set of pluggable “detectors” built on heuristics, and emits alerts with a severity and a reliability score.

Why signatures are not enough

A signature is great at answering “does this packet or flow contain a known malicious pattern?” These questions, though, can’t be answered from a single packet:

Question What you need
Is this brute force? Failure counts across many connections, over a time window
Did the weak password actually work? The password from the request plus the status from the response
Are credentials sent in clear text? The protocol, and whether it ran over TLS
Is a scanner enumerating something? The distribution of paths or usernames over a short period

All of these need state and context. That makes them a good fit for a separate detection layer written in Python: quick to develop, easy to change rules, and scaled out with multiple processes.

Overall structure

   Protocol events (Zeek / Suricata metadata)
                │  Redis pub/sub or ZeroMQ
                ▼
        ┌──────────────────────┐
        │ Dispatcher (main proc)│  subscribe, deserialize, route by channel
        └───────┬──────────────┘
      ┌─────────┼──────────┐
      ▼         ▼          ▼
 [weak pwd] [brute force] [web attack] ...   one or more processes per detector
      │         │          │
      └─────────┴──────────┘
                ▼
     Alert output (message bus / Elasticsearch / syslog)

The main process does three things: read config, set up subscriptions, and hand messages to detectors through multiprocessing queues. Each detector runs in its own process, so a crash in one doesn’t take down the others, and CPU-heavy modules can get extra workers.

The detector framework: three hooks

Every detector extends one abstract base class, and the flow is always the same three steps (the classic Template Method pattern):

  • intercept(msg): pre-filter. Drop messages with missing fields, the wrong protocol, or whitelisted values, and pull out the fields needed later.
  • detect(data): the actual decision. Pattern matching, threshold counting and intel lookups happen here.
  • notice(data, result): build the alert, with severity, reliability and context.

An illustrative detector (a simplified sketch I wrote, not real code) looks like this:

class ClearTextCredDetector(BaseDetector):
    channels = ["auth_events"]

    def intercept(self, msg):
        if msg.get("proto") not in ("ftp", "telnet", "pop3", "imap", "http"):
            return None
        if msg.get("tls"):
            return None            # encrypted, so not clear text
        if not msg.get("password"):
            return None
        return {"src": msg["src_ip"], "dst": msg["dst_ip"],
                "proto": msg["proto"], "user": msg.get("user")}

    def detect(self, d):
        return True                # reaching here means a hit

    def notice(self, d, hit):
        return Alert(kind="cleartext_credential", severity=2,
                     reliability=8, src=d["src"], dst=d["dst"],
                     extra={"proto": d["proto"], "user": d["user"]})

The base class takes care of batching, counters (messages processed, hits, latency, errors), logging and shared false-positive filtering. Whoever writes a new detector only has to think about those three methods.

Config-driven module loading

Detectors are plugged in through configuration: the config names the class path, the channels to subscribe to and a batch size, and the main process loads the class by reflection. Adding or retiring a detector doesn’t touch the core program.

modules:
  bruteforce:
    enabled: true
    class: detectors.bruteforce.Detector
    batch_size: 100
    channels: [auth_events]
  web_attack:
    enabled: true
    class: detectors.web.Detector
    batch_size: 50
    channels: [http_events]

You can also start only a subset of modules from the command line, which helps when debugging or when trimming the engine down for a resource-constrained sensor.

Run modes: one codebase, three ways in

To fit different deployment shapes, the engine abstracts where messages come from:

  • Redis mode (default): subscribe to Redis pub/sub channels. The simplest option, and enough for a single box.
  • ZeroMQ mode: pull directly from upstream ZeroMQ sockets (TCP or IPC). One less hop of middleware and higher throughput.
  • Stream-processing mode: run as an Apache Storm bolt for larger distributed deployments.

If the mode isn’t given explicitly, startup picks one in this order: an environment variable, then a check for whether the ZeroMQ ports are already bound, then a fallback to Redis. Upstream messages can be JSON or a compact delimiter-joined text format whose field order comes from a shared field-definition file. At high volume, the text format saves a good deal of serialization cost.

A tour of the detectors

Area Detector Idea
Auth Weak password Successful login + password in a weak-password list or with a low strength score
Auth Clear-text password Credentials seen in a protocol with no TLS
Auth Brute force Failures in a time window, distinct usernames, many sources against one target
Web Web attack Patterns and heuristics for SQL injection, XSS, command injection, path traversal, file inclusion, XXE
Web Unusual HTTP methods TRACE, WebDAV-style methods and other rare verbs
Web Scanners / crawlers User-Agent, request rate, enumeration patterns (e.g. WordPress plugin probing)
Web Out-of-band (OOB) Signs that an injection triggered a DNS/HTTP callback to an outside host
Payload Shellcode NOP sleds, common encoders, instruction-sequence patterns
Payload PowerShell attack frameworks Markers of well-known open-source attack scripts, encoded commands
Host Crypto-mining trojans Pool domains, the Stratum protocol, wallet address formats
Network VPN protocol identification OpenVPN, IPSec, L2TP, PPTP, WireGuard

Two detectors up close

Weak passwords: the hard part is “did it work?”

If you alert every time you see 123456, failed attempts and scanners will bury you. What matters is a successful login with a weak password. So the weak-password detector does the following:

  1. Extracts credentials from 20+ protocols: HTTP forms and Basic auth, FTP, SSH, Telnet, mail protocols, database protocols, RDP, SMB and more.
  2. Works out whether the login succeeded: for HTTP from the status code and redirect, for other protocols from the auth reply. Failures are dropped.
  3. Scores the password: first against a weak-password dictionary, then with the open-source zxcvbn estimator, while also catching keyboard walks and common patterns.
  4. Tells account types apart: a weak password on a privileged account such as admin or root is raised to a higher severity.
  5. Deduplicates and filters out scanning: an account is reported once per time window. A source that “auto-retries” many passwords in a short burst counts as scanning, not a real login. Background requests made by the login page itself are excluded too.

Brute force: multi-dimensional thresholds

Brute force is about statistics. The detector keeps sliding windows keyed by source → target → service:

def on_failed_login(ev, win):
    key = (ev.src, ev.dst, ev.service)
    win.add(key, ev.ts, ev.user)
    stats = win.stats(key, seconds=300)
    if stats.failures >= 30:
        return "bruteforce"
    if stats.distinct_users >= 10:
        return "user_enumeration"     # dictionary-style username guessing
    return None

You also have to handle distributed brute force: many sources hit one target, each staying under the threshold. You only see that pattern when you aggregate by target instead of by source. The thresholds are config values, because environments differ a lot.

Small tools for fewer false positives

  • A Bloom filter of popular sites: roughly the top million domains, stored in a Bloom filter (a compact set structure with O(1) lookups and a tiny false-match rate). Low-confidence hits involving those domains are let through.
  • Intel blacklist lookups: events that involve known C&C servers or malicious IPs get a higher reliability score.
  • Batching: messages are fetched and checked in batches, which spreads the per-message overhead.

Operational details

The engine has a built-in supervisor thread. It checks CPU, memory and IO on a schedule, cleans up zombie child processes, and uses a PID file so the engine can’t be started twice. Each detector regularly reports its throughput, hit count, latency and error count, so the console can show which module is slowing down.

Trade-offs and lessons

  • The strength of a rule engine is explainability. Every alert can say exactly why, for example “47 failures in 5 minutes”. For security operations that matters a lot.
  • The ceiling is just as clear. It only catches known patterns. It can do nothing about obfuscated variants, encrypted traffic or unknown malware families. Covering those is the job of the ML engine in the next post.
  • “Did it succeed?” is worth more than “was it attempted?” Most alert noise comes from failed scans. The earlier you separate the two, the better, and the correlation layer checks again later.
  • Don’t hard-code thresholds. Normal baselines vary hugely between networks, so thresholds have to be configurable, ideally per asset group.
  • Process isolation is worth the memory. Detection logic written in Python will throw exceptions sooner or later. One module crashing and restarting is far better than the whole engine going down.

Next, we look at the other leg on the same framework: machine learning for DGA domains, malicious communication inside encrypted traffic, and obfuscated web attacks.

The Series

  1. The Big Picture
  2. 10G Packet Capture
  3. Suricata DPI and Rule Management
  4. Protocol Analysis with Zeek
  5. An Event Pipeline with Flume and Esper
  6. Threat Intel Matching and the Knowledge Base
  7. Rule-Based Anomaly Detection
  8. Machine Learning Detection
  9. Alert Correlation, Noise Reduction and Aggregation
  10. Dynamic Sandbox
  11. Multi-Engine AV Scanning
  12. The Management Console
  13. A Lightweight Sensor
  14. Deployment, Versioning, Upgrade and Rollback
  15. Ops Monitoring and Self-Healing

This series is a personal study of system architecture. Details are abstracted and anonymized; it does not describe any specific commercial product or any organization’s internal configuration or data.

Comments

  1. Loading comments…

Leave a comment

Plain text; links are fine. Comments with many links wait for review.