# Dynamic Sandbox Analysis (Part 10)

- URL: https://wuming.si/blog/4033/
- Author: Wuming
- Published: 2025-03-25
- Language: en
- Categories: Technology, English

> Previous: [Alert Correlation, Noise Reduction and Aggregation](https://wuming.si/blog/4032) · Next: [Multi-Engine AV Scanning](https://wuming.si/blog/4034)

Everything so far in this series has been about watching traffic. The capture layer grabs packets, Suricata and Zeek parse protocols and match signatures, and intel, rules and models hunt for anomalies in the metadata. One kind of threat stays blurry from that angle, though: **files**. An email attachment, an executable pulled over HTTP or a zip archive is just a run of bytes on the wire. You only find out what it does by running it.

That's the job of the dynamic sandbox. It comes right after the DPI engine's file extraction step. The engine carves a file out of the traffic and drops it on a queue. The sandbox picks it up, runs it inside an isolated virtual machine and records everything it does. The output is a behavior report plus a verdict, and both go back into the event pipeline so they can be correlated with everything else.

> **Sandbox**: an execution environment that is isolated from real systems and can be reset at will, used to run suspicious programs safely and watch what they do.

## The Big Flow

```
 DPI file extraction / manual upload / API submit
               │
               ▼
        ┌──────────────┐
        │  Task queue   │  ordered by priority
        └──────┬───────┘
               ▼
        ┌──────────────┐
        │  Scheduler    │  VM allocation, concurrency limits
        └──────┬───────┘
     ┌─────────┼──────────┐
     ▼         ▼          ▼
  Static     Dynamic     Multi-AV
  (type/hash (QEMU/KVM   (next post)
   /YARA)    + guest agent)
     └─────────┼──────────┘
               ▼
   Behavior signatures → threat score → reports
               ▼
   Elasticsearch / event pipeline / file storage
```

Each sample goes through roughly six stages:

| Stage | What happens |
|---|---|
| Intake | Pull a task from a Redis queue, ZeroMQ or a REST endpoint |
| Pre-analysis | Hash the file, detect its real type, run YARA and the AV engines |
| VM allocation | Pick a free VM for the target platform (Windows / Linux / Android) |
| Execution | Revert snapshot, drop the sample, start capture, run with a timer |
| Post-processing | Collect logs, match behavior signatures, score, extract IOCs |
| Publishing | Index into Elasticsearch, push to the event pipeline, store the report |

## Pre-analysis: Do the Cheap Things First

Dynamic execution is expensive, because one VM spends several minutes on one sample. So before it boots anything, the scheduler runs a static pass:

- **Hashes.** MD5 and SHA-256 identify the sample and also handle deduplication. The same file can show up in traffic hundreds of times, and there's no point detonating it on every sighting. A Bloom filter gives a fast "seen it before?" check.
  > **Bloom filter**: a memory-efficient set structure that answers "definitely not seen" or "probably seen" very quickly.
- **File type.** Use libmagic-style detection and ignore the extension. The real type decides which VM to use and which "launch package" opens the file. An exe, a DLL, an Office document and an APK each need a different launcher.
- **YARA.** Byte-pattern rules for known malware families.
- **Multiple AV engines.** Several engines scan the file and their results feed into the score. The next post covers this in detail.

If the hash already matches known-bad intel, or the AV engines agree the file is malicious, the task can jump the queue or even get a verdict right away.

## Dynamic Execution: VMs Plus a Guest Agent

The execution layer runs on QEMU/KVM, managed through libvirt. Each platform has a prepared image with a "clean" snapshot.

```python
# Illustrative: the core of one analysis run
def run_task(task):
    vm = pool.acquire(task.platform)      # take an idle VM from the pool
    try:
        vm.revert("clean")                # back to the clean snapshot
        vm.upload(task.sample, task.target_path)
        sniffer.start(vm.iface)           # capture this VM's traffic on the host
        vm.agent.run(task.package, timeout=task.timeout)
        return vm.agent.collect()         # behavior log, dropped files, memory dumps
    finally:
        sniffer.stop()
        vm.poweroff()
        pool.release(vm)
```

A **guest analyzer** (an agent) runs inside the VM. Its jobs:

- launch the sample, opening it the right way for its file type;
- hook key APIs (file I/O, process creation, registry writes, network connections) and record the call sequence;
- watch for changes to the file system, the registry and the process tree;
- dump memory where needed to catch injected code;
- ship all logs back to the host when the timer runs out.

Meanwhile the host captures traffic on the VM's virtual NIC and ends up with a full PCAP. That PCAP is valuable in its own right. You can feed it back through the DPI engine and intel matching to see which domains the sample contacted and whether it reached any known C2.

### Network Isolation

The VMs sit on their own virtual bridge, and there are two modes:

- **Offline**: host-only networking, so the sample can't reach the internet. This is safe, but a lot of malware just goes quiet when it can't phone home, so you miss behavior.
- **Online**: NAT out to the internet. You see real callbacks, but you accept the risk that the sample does real damage outside.

You have to make this trade-off at deployment time. The system includes a script that switches modes with one command.

## Behavior Signatures and Scoring

A raw behavior log can easily contain tens of thousands of API calls, which is far more than anyone can read. A layer of **behavior signatures** turns those low-level events into conclusions that carry meaning:

| Signature | What it suggests |
|---|---|
| Drops an executable | Classic downloader or dropper |
| Installs a service / autorun entry | Persistence |
| Injects into another process | Hiding, evasion |
| Mass-encrypts files | Ransomware |
| Deletes shadow copies | Blocking recovery, common with ransomware |
| Disables the firewall | Weakening defenses |
| Takes screenshots, reads browser credentials | Info stealing |
| Probes for VM / debugger | Anti-analysis |

Those signatures then combine with the other evidence into a single score. Here's an illustrative way to do it:

```python
# Illustrative: weighted evidence, each dimension capped
score  = min(av_hits * W_AV, CAP_AV)
score += min(sum(weight(s) for s in signatures), CAP_BEHAVIOR)
score += network_score(conns, c2_hit)
score += file_activity_score(dropped_files)
verdict = "malicious" if score >= HIGH else "suspicious" if score >= LOW else "clean"
```

The key design choice is **capping each dimension**. Without caps, an installer that writes a lot of files could get flagged as malicious on "file activity" alone. High-risk behaviors such as encrypting files, deleting shadow copies or stealing credentials carry more weight.

## Report Output

Each report is produced in several forms:

- **JSON**: complete and detailed, for machines;
- **HTML**: a readable version for analysts;
- **Elasticsearch document**: searchable, ready for dashboards;
- **ZeroMQ event**: streamed back into the event pipeline in real time, where it joins other alerts in the correlation engine.

The last one matters most. Once the sandbox calls a file malicious, the correlation engine can link that verdict to the original download flow, the host that downloaded the file and any callback alerts that follow. Together they make one attack chain.

## Pitfalls and Trade-offs

- **Resources are the hard limit.** A Windows VM needs several GB of RAM, so concurrency is low. We set a VM count per platform and a global concurrency cap. When samples pile up, priority and dedup absorb the peak. Adding machines without limit was never the plan.
- **Choosing the timeout.** A short timeout misses delayed payloads, and a long one kills throughput. In practice the default is a few minutes and can be changed per task. For samples that sleep before acting, a common technique is to have the guest agent shorten sleep calls.
- **Always revert the snapshot.** Every run has to end back at the clean snapshot, or leftovers from one sample contaminate the next result. There's also a "rebuild every VM" script for when an image gets corrupted.
- **Anti-sandbox tricks.** More and more samples check for signs of virtualization, mouse movement or uptime. Images need hardening to look less like VMs, plus some simulated user activity. Even then, "no malicious behavior observed" is not the same as "clean". That's why the score always combines static and AV results with the dynamic findings.
- **Host safety.** Samples never run on the host, sample files are kept separately, and every VM has CPU and memory limits so one sample can't take down the box.
- **Image maintenance.** Several Windows versions, Linux and an Android emulator, each with its own patches, Office version and runtimes. Keeping them current is a long-term staffing cost.

## Wrap-up

The dynamic sandbox closes the gap that traffic analysis leaves open: you can't learn what a file intends from packets alone. It runs extracted files in resettable VMs and gets the full behavioral picture from API hooks, file/registry/process monitoring and packet capture. Behavior signatures and multi-factor scoring then produce a verdict, and the result goes back into the event pipeline for correlation.

The static stage mentioned "several AV engines scanning together". The next post looks at how that works: orchestrating multiple engines, fusing their results and keeping their signature databases up to date.

## The Series

1. [The Big Picture](https://wuming.si/blog/4024)
2. [10G Packet Capture](https://wuming.si/blog/4025)
3. [Suricata DPI and Rule Management](https://wuming.si/blog/4026)
4. [Protocol Analysis with Zeek](https://wuming.si/blog/4027)
5. [An Event Pipeline with Flume and Esper](https://wuming.si/blog/4028)
6. [Threat Intel Matching and the Knowledge Base](https://wuming.si/blog/4029)
7. [Rule-Based Anomaly Detection](https://wuming.si/blog/4030)
8. [Machine Learning Detection](https://wuming.si/blog/4031)
9. [Alert Correlation, Noise Reduction and Aggregation](https://wuming.si/blog/4032)
10. [Dynamic Sandbox](https://wuming.si/blog/4033)
11. [Multi-Engine AV Scanning](https://wuming.si/blog/4034)
12. [The Management Console](https://wuming.si/blog/4035)
13. [A Lightweight Sensor](https://wuming.si/blog/4036)
14. [Deployment, Versioning, Upgrade and Rollback](https://wuming.si/blog/4037)
15. [Ops Monitoring and Self-Healing](https://wuming.si/blog/4038)

---

*This series is a personal study of system architecture. Details are abstracted and anonymized; it does not describe any specific commercial product or any organization's internal configuration or data.*
