# Machine Learning Detection for Encrypted Traffic and DGA (Part 8)

- URL: https://wuming.si/blog/4031/
- Author: Wuming
- Published: 2025-03-11
- Language: en
- Categories: Technology, English

> Previous: [Rule-Based Anomaly Detection](https://wuming.si/blog/4030) · Next: [Alert Correlation, Noise Reduction and Aggregation](https://wuming.si/blog/4032)

The rule engine from the last post is good at catching "known bad". There are three kinds of problem where it is close to useless:

- **DGA domains.** A DGA (Domain Generation Algorithm) lets malware compute hundreds or thousands of random-looking domains each day and try them until one reaches its C&C server. Blacklists can never keep up.
- **Encrypted traffic.** Once TLS is involved the payload is invisible and signatures stop working. All you have left is side information: packet sizes, timing, handshake parameters.
- **Obfuscated web attacks.** After encoding, inline comments and case tricks, regexes are easy to get around.

This post covers the platform's machine learning detection engine. It **shares the same framework** as the rule engine: it subscribes to the same protocol events, uses the same detector base class and emits the same alert format. The difference is that inside `detect()` the rules are replaced by feature engineering plus model inference.

## The pipeline: a few extra steps inside detect

```
message → intercept → feature extraction → inference → post-processing → alert
            │              │                   │              │
       pre-filter    vectorize/normalize   batch predict   threshold / dedupe
       whitelist     sequence encoding     probabilities   group by host, family
```

Two points stand out:

1. **Pre-filtering matters more than the model.** Inference costs milliseconds and rules cost microseconds. Filtering out popular domains, junk outside valid TLDs and known-benign traffic up front saves most of the compute.
2. **Batch inference.** A neural network takes about as long to score 256 inputs as it does to score one, so detectors collect messages into batches before calling the model.

## DGA detection

### Features

DGA domains "don't look like names a person would pick", and you can put numbers on that:

| Feature | Intuition |
|---|---|
| Length | DGA names tend to be long |
| Character entropy | More random means higher entropy |
| Vowel and digit ratios | Human-chosen names have regular vowel patterns |
| n-gram frequency | Letter pairs/triples common in real domains |
| Meaningful-word probability | Can the label be split into dictionary words? |

A tiny Shannon entropy example:

```python
import math
from collections import Counter

def entropy(label: str) -> float:
    n = len(label)
    return -sum(c / n * math.log2(c / n) for c in Counter(label).values())

entropy("google")        # ~1.92
entropy("xkq3vzj8pwr")   # ~3.46
```

### Model

Hand-built features suit tree-based models. Alongside those we used a character-level LSTM (Long Short-Term Memory, a recurrent neural network that handles sequences well). It reads the domain's characters directly, so nobody has to write features by hand:

```
domain chars → char embedding → bi-directional LSTM → dropout → dense → softmax
                                               output: [benign, family A, family B, ...]
```

The output isn't just "DGA or not". It also names the most likely **malware family**, which analysts find very useful when it's attached to the alert.

### Post-processing

A single suspicious domain is weak evidence. The real signal is **one host resolving a burst of DGA-looking domains in a short time, most of them returning NXDOMAIN**. So post-processing groups results by host, looks at timing, and uses whichever domain finally resolved to point at the live C&C.

## Encrypted traffic analytics (ETA)

ETA doesn't decrypt anything. It only looks at the "shape" of a flow:

- **Packet-length sequence**: size and direction of the first N packets
- **Inter-arrival times**: the timing rhythm. Beaconing C&C traffic is often very regular.
- **TLS handshake**: version, cipher suites and extension list from the ClientHello, which together form a client fingerprint
- **Certificate features**: self-signed certificates, odd validity periods, strange issuer or subject fields
- **Flow statistics**: bytes, packets, duration

```
packet sizes [120, 450, 89, ...] → normalize → 1D CNN → pooling → LSTM → dense → class
```

The 1D CNN (one-dimensional convolution) picks up local patterns, such as the small-big-small rhythm of a handshake. The LSTM captures longer-range dependencies over time. A separate random forest works on the flow statistics, and the two complement each other.

The same approach also covers:

- **Tor identification**: fixed-size cells and a distinctive TLS fingerprint, combined with the public list of exit nodes.
- **Tunnel detection**: DNS tunnels (very long subdomains, high entropy, TXT/NULL records, high query rates), ICMP tunnels (unusual payload sizes and entropy), and HTTP tunnels.

## ML for web attacks

The rule engine already has pattern checks for SQL injection and XSS. The ML versions are there to **resist obfuscation**:

- **SQL injection**: character-level CNN + LSTM + an attention layer (attention lets the model focus on the suspicious part of the input). Input is a URI or a parameter. Output is the probability of injection.
- **XSS**: mixed word- and character-level tokens, word2vec embeddings and a bi-directional LSTM.
- **Command injection / code execution / XXE / framework exploits**: each modeled separately on its own syntax.
- **Webshells**: besides content classification, a fuzzy hash such as TLSH (similar files produce similar hashes) catches variants of known webshells.

The open-source libinjection serves as a strong baseline and as a source of features. TF-IDF with a classic classifier is kept as a lightweight model to compare against.

The engine also includes phishing-site detection (URL structure, domain age, certificate, forms and outbound links on the page, brand impersonation, look-alike domains) and sensitive-data leak detection (Luhn checks for card numbers, formats of keys and tokens). The leak detector is really rule-based, but it lives in the same engine so it can be scheduled the same way.

## Getting models into production

### Three run stages

The engine can start in one of three stages:

- **detect**: production detection.
- **train**: feature extraction only. Samples are archived with positive/negative labels for offline training.
- **detect + train**: detect while collecting samples for ongoing iteration.

```bash
# illustrative: collect positive-sample features for one module
engine --stage train --label positive --module dga
```

### Offline training, online inference

Training happens offline: load the archived features, split them into train/validation/test sets, train with Adam, cross-entropy and early stopping, and save the model file (HDF5). Shipping a model means replacing the file, updating config and restarting the matching detector workers. There's no fancy online learning, but the process is stable and easy to roll back.

### An inference service

As well as consuming the message stream, the engine can run a small HTTP inference endpoint. You send a module name and some data, and it returns a verdict, a family and a confidence score. The console's "check this domain" feature and analysts' submissions of suspicious samples both go through it.

### A self-learning whitelist

One small but very practical design choice: **a Bloom filter that remembers samples confirmed as false positives**. On a schedule, samples judged benign are added to the filter and saved to disk. Next time the same input arrives, it is skipped before inference, which cuts false positives and saves compute at the same time.

## Rule engine vs. ML engine

| Dimension | Rule engine | ML engine |
|---|---|---|
| Good at | Known attacks, compliance checks | Unknown variants, DGA, encrypted traffic, tunnels |
| Latency | Microseconds | Milliseconds |
| Missed detections | High for unknown attacks | Better generalization |
| False positives | Low and controllable | Depends on model and data |
| Explainability | High | Low |
| Maintenance | Edit rules | Collect, label, train, release |

The sensible deployment runs both: rules on the fast path, ML for deeper analysis.

## Pitfalls and lessons

- **The base-rate fallacy.** In production, malicious samples might be one in a million. A model with "99.5% accuracy" can still produce thousands of false positives a day. Set thresholds from the alert volume you can handle, not from lab metrics.
- **Class imbalance and drift.** The DGA families in your training set aren't the ones in the wild, and benign domains keep changing too (CDNs, URL shorteners, random cloud subdomains). Keep feeding false positives back and retraining.
- **Pin your dependencies.** Upgrading a deep learning framework can easily stop old models from loading. In production, pin the framework version and version model files together with the code.
- **Add explanations yourself.** Attach the key features to each alert (entropy, length, matched n-grams, closest known family). Without them, analysts won't trust ML alerts.
- **Isolate resources.** Models take a lot of resident memory. Run them separately from the rule engine, or cap the worker count, so they don't starve capture and protocol analysis of CPU.

Between the two detection engines, Suricata and intel matching, the platform can produce a flood of raw alerts every minute. Next time: how to correlate, de-noise and aggregate them into incidents an analyst can actually get through.

## The Series

1. [The Big Picture](https://wuming.si/blog/4024)
2. [10G Packet Capture](https://wuming.si/blog/4025)
3. [Suricata DPI and Rule Management](https://wuming.si/blog/4026)
4. [Protocol Analysis with Zeek](https://wuming.si/blog/4027)
5. [An Event Pipeline with Flume and Esper](https://wuming.si/blog/4028)
6. [Threat Intel Matching and the Knowledge Base](https://wuming.si/blog/4029)
7. [Rule-Based Anomaly Detection](https://wuming.si/blog/4030)
8. [Machine Learning Detection](https://wuming.si/blog/4031)
9. [Alert Correlation, Noise Reduction and Aggregation](https://wuming.si/blog/4032)
10. [Dynamic Sandbox](https://wuming.si/blog/4033)
11. [Multi-Engine AV Scanning](https://wuming.si/blog/4034)
12. [The Management Console](https://wuming.si/blog/4035)
13. [A Lightweight Sensor](https://wuming.si/blog/4036)
14. [Deployment, Versioning, Upgrade and Rollback](https://wuming.si/blog/4037)
15. [Ops Monitoring and Self-Healing](https://wuming.si/blog/4038)

---

*This series is a personal study of system architecture. Details are abstracted and anonymized; it does not describe any specific commercial product or any organization's internal configuration or data.*
