Wuming 無名

Blog

Threat Intel Matching and the Knowledge Base (Part 6)

· Technology · English

Previous: An Event Pipeline with Flume and Esper · Next: Rule-Based Anomaly Detection

Attackers reuse infrastructure all the time: the same C2 servers, the same phishing domains, the same malware download URLs. Collecting those known-bad things gives you the most basic layer of threat intelligence, IOCs (Indicators of Compromise: malicious IPs, domains, URLs, file hashes). Checking network traffic against IOCs is cheap and precise, so it’s the best-value detection you can build.

Matching alone isn’t enough, though. When an alert fires, users want to know what kind of threat it is, how severe it is, which stage of an attack it represents and what to do about it. Answering those questions takes a shared vocabulary across the platform. So this post covers two things. The online intel matcher consumes the protocol metadata from Zeek and Suricata and sends IOC alerts into the event pipeline from the last post. The offline knowledge base gives every engine the same taxonomy, severities, kill-chain mapping and rule definitions.

Part one: the intel matcher

Structure

The matcher is a Python service with a small task manager inside. Each task runs as a thread. A task checks its environment before it starts, watches for config changes while it runs, and is restarted when its config changes. The registered tasks are:

  • A management API task, used to query status, add or remove whitelist entries, and hot-reload intel and rules.
  • One detection task per protocol: DNS, HTTP, SMTP, POP3, IMAP, TLS, TCP connections, plus an optional raw-packet task that re-checks IPS alerts at the packet level.

Input comes in two modes. In ZeroMQ mode, each protocol task subscribes directly to its metadata channel. It’s faster and it’s the default. In Redis mode, events are popped (with a blocking pop) from one shared list and then dispatched by protocol. That was the earlier design. When load is too high, you can turn off protocol channels you don’t need in config. Connection logs are usually the first to go because they have the most volume.

 protocol metadata (ZeroMQ / Redis)
        │
        ▼
 ┌──────────────────────────────────────────┐
 │ task manager                             │
 │  DNS │ HTTP │ mail │ TLS │ conn │ ...    │ ← one thread per protocol
 └──────────────────────────────────────────┘
        │  each event runs the detection chain
        ▼
   alert (rule ID, category, severity attached)
        │
        ▼
   event pipeline / queue / ES

The per-event detection chain

Every event goes through the checks in a fixed order, and the first hit produces an alert and stops the chain:

event ─► whitelisted? ──yes──► drop
            │no
            ▼
         behavior rule hit? ──yes──► alert
            │no
            ▼
         domain in intel? ──yes──► alert (query to malicious domain)
            │no
            ▼
         dst IP in intel? ──yes──► alert (connection TO malicious IP)
            │no
            ▼
         src IP in intel? ──yes──► alert (connection FROM malicious IP)
            │no
            ▼
          pass

Here’s DNS as an example (illustrative):

def handle_dns(ev):
    if ev.get("qtype_name") == "PTR":      # reverse lookups aren't useful here
        return
    q = ev.get("query", "")
    if not is_valid_domain(q):
        return
    if whitelist.hit(domain=q):
        return
    if behavior_rules["dns"].match(ev):
        return emit(ev, "dns-abnormal")
    if domain_intel.hit(q):
        return emit(ev, "dns-query-malicious-domain")
    if ip_intel.hit(ev["dst_ip"]):
        return emit(ev, "conn-to-malicious-ip")
    if ip_intel.hit(ev["src_ip"]):
        return emit(ev, "conn-from-malicious-ip")

“Connection to a malicious IP” and “connection from a malicious IP” are deliberately separate alerts. The first usually means an internal host may be compromised. The second is most likely an outside scan or attack. Because the direction differs, so do the severity and the remediation advice.

What each protocol checks

Protocol Main things matched
DNS Queried domain, resolved IPs
HTTP Host, full URL, User-Agent, download location
SMTP/POP3/IMAP Sender/recipient domains, mail server IPs, attachment info
TLS SNI, certificate fields, client fingerprint (JA3)
Connections Destination/source IP, suspicious ports

Matching techniques

Intel sets typically run from hundreds of thousands to millions of entries, and the matcher has to check tens of thousands of events per second, so every lookup needs to be O(1) or close to it:

  • Exact match. IPs, full domains and URLs go into hash sets.
  • Suffix / wildcard match. An entry like *.evil.example has to match any subdomain. We walk the domain’s labels from right to left and look each suffix up in a table, rather than running a regex per wildcard entry.
  • CIDR match. Network-range intel and whitelists use a prefix tree, or buckets keyed by mask length.
  • Bloom filter. Every lookup goes through a Bloom filter before the real set. Most clean events are ruled out at that step, which saves both memory and time.
def domain_hit(name, exact, wildcard):
    if name in exact:
        return True
    labels = name.split(".")
    for i in range(1, len(labels) - 1):          # strip leftmost label each time
        if ".".join(labels[i:]) in wildcard:
            return True
    return False

Feeds and hot reload

Intel comes from three places: public intel curated by the open-source Maltrail project (C2s, malware infrastructure, phishing, scanners and more), blocklists we maintain ourselves, and custom intel that users add in the management console. Every entry has a category, and each category maps to a threat type in the knowledge base. Alerts therefore arrive with a classification and a recommendation already attached.

Updating intel doesn’t need a restart. The management API has “reload intel” and “reload rules” endpoints. New data loads in the background and then replaces the old reference in one swap.

Whitelists

False positives are the biggest problem with intel matching. Most come from shared CDN and cloud IPs, sinkhole domains that security vendors have taken over, and stale entries nobody cleaned up. Whitelists accept exact values, wildcard domains, suffixes and CIDR ranges, and you can add or remove entries through the API at any time. The whitelist check is the first step of the chain, so nothing on it can trigger an alert from any intel source.

Behavior rules

Besides IOCs, the matcher also runs a light class of “behavior rules”. Each one names a protocol, a field and a pattern, optionally with source/destination address scopes and an active time window, plus extra filter conditions in JSON. Two examples: “the HTTP User-Agent contains the marker of a known pentest tool” and “a DNS query’s subdomain has unusual length and entropy”. Heavier statistical anomaly detection belongs to the framework in the next post.

Part two: the knowledge base

What problem it solves

The platform runs several detection engines, and each has its own rules and alert types. If every engine decided for itself what a threat is and how severe it is, the console would show a mess of inconsistent labels. The knowledge base is the single source of truth. Every rule, category, kill-chain stage, malware family and vulnerability record is defined there and synced everywhere else.

Data model ideas

Here’s an abstracted entity diagram (names are illustrative):

  attack_stage            threat_class
  (kill-chain stage)      (top-level class)
       ▲                      ▲
       │                      │ 1:N
       │                 threat_type
       │                 (severity, attacker/victim direction,
       │                  description, remediation, references)
       │                      ▲
       └──────────┬───────────┘
                  │
            detection_rule ──────► malware_family (with ATT&CK TTPs)
  (rule ID, owning engine, rule body, severity, reliability,
   enabled/deprecated, user-edited flag, revision)
                  │
                tag ◄──────── custom_intel

  behavior_rule   (protocol, field, pattern, address scope, active window, extra filter)
  vuln_entry      (CVE ID, description, references)

Some design points worth borrowing:

  • Kill-chain stages follow the classic seven-stage model: reconnaissance, weaponization, delivery, exploitation, installation, command and control, actions on objectives. Every rule maps to one stage. The console can then order a host’s alerts by attack progress and show at a glance how far an attack has gone.
  • Two-level classification (class → type). Severity and remediation advice live on the type, and rules just reference a type, so nobody repeats the same advice on thousands of rules.
  • Attacker/victim direction lives on the type. For some alert types the source IP is the attacker; for others it’s the destination. In “internal host talks to C2”, for example, the victim is the source. That field decides who gets highlighted in red in the UI and whose asset risk score goes up.
  • Rule ID ranges per engine. Signatures, behavior rules, protocol-analysis scripts and ML models each get their own range. You can tell which engine fired an alert from its ID alone, and correlation rules can refer to a whole family of alerts by range.
  • Rules carry a reliability score. Severity answers “how bad is this if it’s true”. Reliability answers “how likely is it to be true”. Keeping them separate gives the correlation engine and the scoring something real to work with.
  • Malware families link to ATT&CK techniques, and the vulnerability table holds public CVE entries, so the alert detail page can show background context.
  • Bilingual columns. Names and descriptions have both Chinese and English columns, so switching the UI language is a column choice.

Sync and upgrade

The knowledge base is a MySQL database. It ships with each rule package, and an upgrade script runs daily or on demand. The script’s steps:

  1. Create tables, all with IF NOT EXISTS so the step can be re-run safely.
  2. Delete expired or deprecated data.
  3. Import new categories, rules, intel and vulnerability data.
  4. Apply schema changes.
  5. Refresh the intel whitelists.

After that, the data is pushed in three directions: the management console’s operational database (for display and user edits), the detection engines’ rule stores, and the IDS rule files.

Lessons learned

  1. User edits got overwritten. A user disables or tunes a rule in the UI, and the next upgrade puts the vendor version straight back. The fix was a “user-edited” flag on each rule. Sync skips flagged rows and only updates rows nobody has touched.
  2. Big imports time out. Loading a SQL file with hundreds of thousands of lines, like the vulnerability data, in one go can hit MySQL’s packet-size and timeout limits. Either raise max_allowed_packet and the timeouts, or split the file into chunks and import them in batches.
  3. Primary-key collisions. Re-running an upgrade made plain INSERTs fail on duplicate keys. Every import should be idempotent (INSERT ... ON DUPLICATE KEY UPDATE, or delete-then-insert).
  4. Credentials hard-coded in scripts. Early upgrade scripts passed the database user and password on the command line. That’s insecure, and it turns a password change into a project. Read credentials from a protected config file or environment variables, and use a least-privilege account.
  5. The cost of “first hit wins”. The chain is simple and fast, but when one event matches both a bad domain and a bad IP, only one alert survives. The fact that two independent signals agreed is lost. If the correlation engine needs that evidence, keep checking after the first hit and attach every match as extra fields.
  6. Intel goes stale. IP intel ages fast, and last month’s C2 address may belong to a legitimate service today. Store first-seen and last-seen times on each entry, and demote or drop entries automatically once they expire.

Wrap-up

The intel matcher’s job is to recognize known-bad things quickly. The knowledge base’s job is to tell everyone what a threat is, how serious it is and what to do about it. The matcher runs online and is built for throughput. The knowledge base runs offline and is built for consistency. Between them, every alert on the platform gets the same semantic labels.

Next up is a different kind of detection. It doesn’t rely on known IOCs. It uses rules to describe abnormal behavior such as brute force, authentication attacks and suspicious payloads.

The Series

  1. The Big Picture
  2. 10G Packet Capture
  3. Suricata DPI and Rule Management
  4. Protocol Analysis with Zeek
  5. An Event Pipeline with Flume and Esper
  6. Threat Intel Matching and the Knowledge Base
  7. Rule-Based Anomaly Detection
  8. Machine Learning Detection
  9. Alert Correlation, Noise Reduction and Aggregation
  10. Dynamic Sandbox
  11. Multi-Engine AV Scanning
  12. The Management Console
  13. A Lightweight Sensor
  14. Deployment, Versioning, Upgrade and Rollback
  15. Ops Monitoring and Self-Healing

This series is a personal study of system architecture. Details are abstracted and anonymized; it does not describe any specific commercial product or any organization’s internal configuration or data.

Comments

  1. Loading comments…

Leave a comment

Plain text; links are fine. Comments with many links wait for review.