Multi-Engine AV Scanning (Part 11)
Previous: Dynamic Sandbox · Next: The Management Console
In the last post, the sandbox ran every file through antivirus engines during its pre-analysis stage. This post covers that layer on its own: why use several engines, how to organize them, how to merge their results, and how to keep their signature databases fresh.
In the pipeline, multi-engine AV is a service that other components call. Files that the DPI engine pulls out of traffic and samples handed to the sandbox both get scanned here. The service never looks at traffic. It answers one question: what does each engine think of this file?
Why One Engine Isn’t Enough
Every AV engine has blind spots. Each vendor builds its signatures from different sources, so when a new variant appears, some engines cover it already and others haven’t caught up. Their heuristics also vary in aggressiveness: some miss more, others flag more false positives.
Running several engines side by side gets you two things right away:
- Fewer misses. If one engine doesn’t recognize a file, another may.
- A confidence signal. “4 of 5 engines flagged it” and “1 of 5 flagged it” mean very different things. The second case usually deserves a trip through the sandbox to confirm.
The costs are just as clear. You use several times the resources and run several times as many components, and each engine has its own install procedure, update mechanism and ways of failing. Most of the design work in this layer is about managing those costs.
Our mix is several commercial engines plus open-source ClamAV.
Architecture: One Orchestrator, Independent Engines
file to scan
│
▼
┌──────────────────┐
│ Scan orchestrator │ (called by sandbox / file extraction)
└────────┬─────────┘
┌────────┬──┴─────┬────────┐
▼ ▼ ▼ ▼
Engine A Engine B ClamAV Engine N each with its own daemon + DB
│ │ │ │
└────────┴───┬────┴────────┘
▼
┌──────────────────┐
│ Normalization │ → {engine, detected, name}
└────────┬─────────┘
▼
┌──────────────────┐
│ Result fusion │ count / vote / any-hit
└──────────────────┘
The core idea is to treat each engine as a black box. Some run as resident daemons and take scan requests over a local socket. ClamAV’s clamd, driven by the clamdscan client, is much faster than cold-starting clamscan for every file. Others are command-line scanners that you invoke once per scan. The orchestrator doesn’t care which. It just has a thin adapter for each engine:
# Illustrative: one adapter per engine, uniform output
class EngineAdapter:
name = "base"
def scan(self, path) -> dict:
raise NotImplementedError
class ClamAVAdapter(EngineAdapter):
name = "clamav"
def scan(self, path):
out = run(["clamdscan", "--no-summary", path], timeout=30)
# output looks like "/path: Eicar-Signature FOUND"
hit = out.strip().endswith("FOUND")
sig = out.split(": ", 1)[1].rsplit(" ", 1)[0] if hit else None
return {"engine": self.name, "detected": hit, "name": sig}
Each adapter handles three things: invoking the engine, enforcing a timeout and parsing the output. Every engine’s result ends up in the same shape, so the fusion logic never needs to know which engine it’s looking at. Adding or removing an engine means adding or removing an adapter.
Fusing the Results
After normalization, fusion is a matter of policy. The common choices:
| Policy | Rule | Good for |
|---|---|---|
| Any hit | One engine flags it → malicious | Block-first settings, e.g. mail gateways |
| Threshold | Malicious if hits ≥ N | Balancing false positives and misses |
| Majority | More than half flag it | Conservative, fewest false positives |
| Weighted | Engines carry different weights | When you have historical accuracy data |
Our approach: always expose the raw count and each engine’s detection name, and make the verdict policy configurable. The default is a threshold. At or above the threshold, the file is malicious. Below it but above zero, it’s marked “suspicious” and handed to the sandbox and the correlation engine for a closer look.
# Illustrative: fused result
def fuse(results, threshold=2):
hits = [r for r in results if r["detected"]]
return {
"detect_count": len(hits),
"total": len(results),
"names": sorted({r["name"] for r in hits}),
"verdict": "malicious" if len(hits) >= threshold
else "suspicious" if hits else "clean",
}
Two more details:
- Detection names don’t line up. Every vendor names the same trojan differently. We don’t try to map names exactly. We keep the original list and use simple keywords (Trojan, Ransom, Backdoor and so on) for a rough category to show in the UI and in stats.
- A timeout or failure must never count as “clean”. If an engine times out or crashes, its result is “unknown” and the denominator shrinks to match. Otherwise “0/5” might really mean “0/3, plus 2 that never ran”.
Signature Updates: Central Mirror, Local Swap
Updates are the most painful part of running several engines. Each one has its own database format, directory layout and loading behavior, and many deployments aren’t allowed to reach the internet directly.
Our answer is a central mirror. An update server on the internal network keeps the latest signature databases in one directory per engine. A scheduled job on each appliance runs during off-peak hours (e.g. overnight), pulls from the mirror with rsync and swaps in the new databases one engine at a time.
Update mirror
└── one directory per engine
│ rsync (scheduled, off-peak)
▼
┌───────────────────────────────────┐
│ For each engine: │
│ 1. download to a temp dir │
│ 2. back up the current database │
│ 3. swap in new DB, fix ownership │
│ 4. reload or restart as needed │
└───────────────────────────────────┘
Design points:
- Check connectivity first. If the mirror is unreachable, exit early and leave the current databases alone.
- Back up before replacing. If a new database turns out to be bad, you can roll back.
- Each engine succeeds or fails on its own. A failed update for one engine gets logged and doesn’t block the others.
- Reload behavior differs by engine. Some pick up new files automatically. Others need a daemon restart. ClamAV’s
clamdcan reload its database. The update script hides all these differences. - File ownership. Many engines run as a dedicated low-privilege user, so replaced files have to be chowned back. If they aren’t, the daemon can’t read its own database. We hit this one more times than any other bug.
# Illustrative: updating a single engine
update_engine() {
local name=$1 dst=$2 owner=$3
tmp=$(mktemp -d)
rsync -a "$MIRROR/$name/" "$tmp/" || { log "$name: fetch failed"; return 1; }
mv "$dst" "$dst.bak.$(date +%F)"
mv "$tmp" "$dst" && chown -R "$owner" "$dst"
reload_or_restart "$name"
}
Health Checks and Self-Healing
Long-running daemons crash sooner or later. Monit runs a check script for each engine, and the script asks two questions:
- Does the service manager (systemd) think the engine is running?
- Is the process actually there? Service state and real process state disagree more often than you’d expect.
If either check fails, the script restarts the engine and exits non-zero so Monit raises an alert.
# Illustrative: health check for one engine
systemctl is-active --quiet "$SVC" || { systemctl restart "$SVC"; exit 1; }
pgrep -f "$PROC_PATTERN" >/dev/null || { systemctl restart "$SVC"; exit 2; }
exit 0
There’s more on Monit in Part 15.
Pitfalls and Trade-offs
- CPU and memory. Several resident engines use a lot of memory, and scanning large files pushes CPU up. We capped ClamAV’s thread count, maximum file size and recursion depth. Oversized files are skipped and flagged so they don’t stall the pipeline.
- SELinux. On systems with SELinux enabled, AV processes may not be allowed to read arbitrary files by default, so you need to turn on the matching policy boolean.
- Allowlists and exclusions. Every engine configures exclusions in its own way. To keep false positives down, known-good files have to be excluded in each engine separately, which is another place where operations work multiplies by N.
- Commercial engine lifecycles. Each commercial engine has its own release cadence and installer quirks, so upgrades need dedicated pre- and post-checks. This kind of chore is easy to underestimate.
- “All clean” doesn’t mean safe. AV engines match known signatures and can’t do much against brand-new samples. Multi-AV results are one input to the score. The final verdict also weighs sandbox behavior and intel hits.
Wrap-up
Multi-engine AV comes down to four pieces. Adapters give very different engines one output format. A configurable fusion policy turns several opinions into one verdict. A central mirror plus per-engine swaps handles updates on offline networks. Monit keeps the engines running. Scanning itself is the easy part. The hard part is keeping N different engines healthy year after year.
That wraps up the detection layers. The next post moves to the user side: how the management console is built on Django, Celery and a Java service, and how Chinese/English internationalization works.
The Series
- The Big Picture
- 10G Packet Capture
- Suricata DPI and Rule Management
- Protocol Analysis with Zeek
- An Event Pipeline with Flume and Esper
- Threat Intel Matching and the Knowledge Base
- Rule-Based Anomaly Detection
- Machine Learning Detection
- Alert Correlation, Noise Reduction and Aggregation
- Dynamic Sandbox
- Multi-Engine AV Scanning
- The Management Console
- A Lightweight Sensor
- Deployment, Versioning, Upgrade and Rollback
- Ops Monitoring and Self-Healing
This series is a personal study of system architecture. Details are abstracted and anonymized; it does not describe any specific commercial product or any organization’s internal configuration or data.
Comments