Wuming 無名

Blog

10G Packet Capture with Zero Copy (Part 2)

· Technology · English

Previous: The Big Picture · Next: Suricata DPI and Rule Management

Every detection in a network security platform starts with one boring promise: we saw the packet. If the capture layer drops 2% of traffic under load, it is not dropping a random 2%. It drops during bursts, which is exactly when an attack or a large file transfer is happening. A missed packet in the middle of a TCP stream can make the DPI engine give up on reassembly, and the file or the exploit inside it simply never gets inspected.

This post covers the bottom box of the architecture from Part 1: how traffic gets from a mirror port into per-core queues that Suricata and Zeek can read at 10 Gbps and above, and what we have to tune on the NIC, the kernel and the CPUs to make that work.

The problem: 67 nanoseconds per packet

Ten gigabits per second sounds like a bandwidth problem. It is really a packets-per-second problem.

Packet size Packets per second at 10 Gbps Time budget per packet
64 bytes (minimum) ~14.88 million ~67 ns
512 bytes ~2.35 million ~425 ns
1500 bytes ~0.82 million ~1.2 µs

Real traffic is a mix, but bursts of small packets (DNS, TCP ACKs, scans) are common. A cache miss to main memory costs around 100 ns. A system call costs more. The standard Linux path, where the driver builds a socket buffer, the kernel walks the protocol stack, and libpcap copies the packet into user space, simply does not fit in that budget on one core.

So the capture layer has three jobs:

  1. Don’t copy. Get packet bytes from the NIC to the analyzer without moving them around.
  2. Spread the load. Use many cores, but keep each connection on a single core.
  3. Stay out of the way. Don’t let interrupts, other processes or memory layout steal cycles from the workers.

The design

   SPAN / TAP (10G)
         │
┌────────▼─────────────────────────────────────────────┐
│ NIC                                                  │
│  multi-queue RX, hardware RSS, offloads disabled     │
└────────┬─────────────────────────────────────────────┘
         │ DMA into pre-mapped buffers (no copy)
┌────────▼─────────────────────────────────────────────┐
│ PF_RING ZC driver + kernel module                    │
│  rings in huge pages, mapped into user space         │
└────────┬─────────────────────────────────────────────┘
         │
┌────────▼─────────────────────────────────────────────┐
│ Balancer process (one pinned core)                   │
│  symmetric 5-tuple hash → queue index                │
│  can fan out to several consumer groups              │
└───┬──────────┬──────────┬──────────┬─────────────────┘
    │ q0       │ q1       │ ...      │ qN-1   (shared memory, lock-free)
┌───▼───┐  ┌───▼───┐  ┌───▼───┐  ┌───▼───┐
│ DPI   │  │ DPI   │  │ DPI   │  │ DPI   │   one worker per isolated core
│ wkr 0 │  │ wkr 1 │  │  ...  │  │ wkr N │
└───────┘  └───────┘  └───────┘  └───────┘

Zero copy with PF_RING ZC

PF_RING is an open-source packet capture framework from the ntop project. Its ZC (zero copy) mode uses modified NIC drivers so that the card DMA-writes packets directly into memory buffers that are also mapped into the user-space process. The application reads the packet where the NIC put it. No socket buffer, no protocol stack, no recvfrom() copy.

The user-space side polls those rings rather than waiting for interrupts. Polling burns a core even when traffic is light, but at high rates it is far cheaper than an interrupt per batch, and latency becomes predictable.

The balancer

The NIC can already spread packets across its hardware queues with RSS (Receive Side Scaling, where the card hashes packet headers and picks a queue). Why add a software balancer on top?

  • Hash control. Many NICs use an asymmetric RSS hash by default, so the client-to-server and server-to-client halves of one TCP connection can land on different queues. A DPI engine that sees only half a conversation cannot reassemble streams. A software balancer lets us pick a symmetric hash.
  • Decoupled worker count. The number of hardware queues is fixed by the NIC; the number of analysis workers depends on how many cores we can spare. The balancer maps one to the other.
  • Fan-out. The same packet can be delivered to several consumer groups, for example 8 Suricata workers and 4 Zeek workers, without copying it twice. Each group gets its own set of queues pointing at the same buffers.
  • Visibility. One place to count packets in, packets per queue and drops.

PF_RING ships a small zero-copy balancer tool for exactly this. The delivery to workers goes through shared-memory, lock-free queues: each worker attaches to its queue by cluster ID and queue index.

Choosing a distribution mode

Mode How it works Good for Problem
Round-robin Next packet → next queue Stateless counting Breaks every stateful analyzer
Per-flow (5-tuple) Hash of src/dst IP, src/dst port, protocol DPI, IDS, flow logs Elephant flows can overload one worker
Per-IP pair (2-tuple) Hash of src/dst IP only Fragments, protocols with dynamic ports (FTP data, SIP) Coarser balance
Custom Your own function Tunnels, special encapsulations You maintain it

For a DPI platform the answer is almost always a symmetric per-flow hash. “Symmetric” means swapping source and destination gives the same result. The usual trick is to sort or combine the two endpoints before hashing:

/* illustrative symmetric flow hash */
static inline uint32_t flow_hash(uint32_t sip, uint32_t dip,
                                 uint16_t sp,  uint16_t dp, uint8_t proto)
{
    uint32_t a = sip ^ dip;           /* XOR is order-independent */
    uint32_t b = (uint32_t)(sp ^ dp);
    uint32_t h = a * 0x9E3779B1u ^ (b << 16 | proto);
    return h ^ (h >> 15);
}
/* queue = flow_hash(...) % num_workers; */

One thing to watch: IP fragments after the first one carry no ports. If the hash includes ports, fragments of one datagram can scatter. Either reassemble before balancing or fall back to a 2-tuple hash for fragments.

Making the box behave

Zero copy removes the biggest cost. The rest of the budget is lost to the machine itself. These are the knobs that mattered most.

1. Partition the CPUs

A typical 16-core layout:

core  0-1   OS, NIC interrupts, housekeeping
core  2     balancer
core  3-9   Suricata workers
core 10-13  Zeek workers
core 14-15  local services (Redis, pipeline agent)

Then tell the kernel to leave the worker cores alone with boot parameters such as isolcpus, nohz_full and rcu_nocbs, and pin each worker with CPU affinity. An isolated core runs one thread, with no scheduler ticks and no other tasks to evict its cache.

2. Keep interrupts away from workers

Route NIC and disk IRQs to the housekeeping cores by writing CPU masks into the IRQ affinity settings, and disable irqbalance (it will helpfully move them back).

3. Respect NUMA

On a two-socket server, each NIC is attached to one CPU socket’s PCIe lanes. If the balancer and workers run on the other socket, every packet crosses the inter-socket link. Rule of thumb: NIC, its memory buffers, the balancer and the workers that read it should all live on the same NUMA node. Use numactl or the capture library’s own binding options, and check which node the card is on before choosing cores.

4. Use huge pages

Packet rings are large. With 4 KB pages they need thousands of TLB entries; with 2 MB huge pages they need a handful. Pre-allocate huge pages on the right NUMA node at boot so they are available and physically contiguous.

5. Turn off NIC offloads

Offloads like GRO/LRO (merging small packets into big ones) and TSO/GSO are great for a server and bad for a sensor. They rewrite what was on the wire: the IDS sees one 64 KB “packet” that never existed, and evasion tricks hidden in segment boundaries disappear. Disable them on capture interfaces and size the NIC’s RX ring generously:

# illustrative: prepare a capture interface
ethtool -K cap0 gro off lro off tso off gso off rx-vlan-offload off
ethtool -G cap0 rx 4096
ethtool -L cap0 combined 8

6. Filter early

If some traffic is known to be uninteresting (backup jobs, storage replication), drop it with a BPF filter at capture time, before the balancer. Every packet not delivered is CPU saved. Be conservative here; anything filtered out is invisible to every detector.

Watching for drops

You cannot fix what you don’t measure. Drops happen in three places and each needs its own counter:

Where What it means Typical cause
NIC (rx_missed / rx_no_buffer) Card had nowhere to DMA the packet RX ring too small, balancer too slow
Balancer → queue A worker’s queue was full That worker is overloaded
Inside the analyzer Engine skipped or truncated processing Memory caps hit, expensive rules

A small script run every minute by cron or a process monitor can check that the balancer process is alive, read the per-queue counters, and log a warning when drops exceed a threshold (for example 0.1%). Feeding those numbers into the metrics stack (Part 15) turns “we think we’re fine” into a graph.

Unbalanced queues deserve a separate check. If one worker receives far more than its share, it is usually not the hash. It is the traffic: one huge backup flow, or most users talking to one proxy. Symmetric per-flow hashing cannot split a single flow. Options are to filter that flow, give it a dedicated worker, or accept the drop and alert on it.

Pitfalls and trade-offs

  • Driver lock-in. Zero-copy needs specific NIC families and matching driver versions. Kernel upgrades can break the driver build. Pin kernel and driver versions together and treat them as one unit in the release manifest.
  • Always keep a fallback. Not every deployment has a supported card (virtual machines, cloud instances, odd hardware). The capture layer should fall back to standard PF_RING, AF_PACKET or libpcap with a config switch, accepting lower throughput.
  • Polling costs power and cores. A busy-polling balancer pins a core at 100% even at night. That is the price of predictable latency; budget for it.
  • Tunnels and VLANs. Mirrored traffic often arrives inside VLAN tags, GRE or VXLAN. If the hash only sees the outer headers, all tunnelled traffic goes to one worker. Hash on inner headers when tunnels are present.
  • Time sync. Detection and correlation compare timestamps across sensors. Use hardware or reliable NTP time on capture hosts, or multi-sensor timelines will lie.
  • Test with real traffic shapes. A replay of large file transfers will show line rate with zero drops. A replay of small-packet DNS floods will show the real limits. Benchmark with both.

What’s next

We now have packets arriving on isolated cores with every connection intact. In the next post we look at what reads them: Suricata as a DPI engine, how it turns streams into alerts, metadata and extracted files, and how a large rule set is organized, classified and kept in sync.

The Series

  1. The Big Picture
  2. 10G Packet Capture
  3. Suricata DPI and Rule Management
  4. Protocol Analysis with Zeek
  5. An Event Pipeline with Flume and Esper
  6. Threat Intel Matching and the Knowledge Base
  7. Rule-Based Anomaly Detection
  8. Machine Learning Detection
  9. Alert Correlation, Noise Reduction and Aggregation
  10. Dynamic Sandbox
  11. Multi-Engine AV Scanning
  12. The Management Console
  13. A Lightweight Sensor
  14. Deployment, Versioning, Upgrade and Rollback
  15. Ops Monitoring and Self-Healing

This series is a personal study of system architecture. Details are abstracted and anonymized; it does not describe any specific commercial product or any organization’s internal configuration or data.

Comments

  1. Loading comments…

Leave a comment

Plain text; links are fine. Comments with many links wait for review.