Wuming 無名

Blog

Deployment, Version Manifests, Upgrade and Rollback: Moving Thirty Components Together (Part 14)

· Technology · English

Previous: A Lightweight Sensor · Next: Ops Monitoring and Self-Healing

Count up the components from the last thirteen posts: capture drivers and PF_RING, two DPI engines, Flume and its extensions, Kafka, Redis, MySQL, MongoDB, Elasticsearch and Kibana, the intel knowledge base, several detection engines, the sandbox and its guest images for each platform, multiple AV engines, a web backend, a frontend, and a Python virtualenv for most of them. That’s over thirty separately packaged parts.

This post covers the platform’s delivery layer. It answers three questions. How does a bare server with a fresh OS become a working detection appliance? How do you record exactly which version of each component is on a given box? And how do you back out when an upgrade fails? None of this is in the data path, but it decides whether everything else keeps running at a customer site.

1. First-boot initialization

The basic delivery flow:

Minimal OS → copy deploy scripts → set hardware-tier variable → enable init service → reboot
                                                                         │
                               first boot: init service runs  ◀──────────┘
   load tier profile → install base packages → tune OS → set up DBs / queues
   → deploy engines → create sandbox VMs → start all services → self-check

At the center is a systemd oneshot service, a unit type that runs once and exits. Two guards keep it from running twice:

# Illustrative
LOCK=/var/lock/platform-init
[ -f "$LOCK" ] && exit 0
systemctl disable platform-init   # disable ourselves first
touch "$LOCK"
# ... initialization steps follow

The script disables itself before it writes the lock file. That way, if power fails halfway through, the next boot won’t re-initialize the databases. The downside is that a mid-run failure needs a human, because the script can’t resume from step N. A better pattern writes a stage marker after each step and skips completed stages on rerun.

A few OS-level settings in the init are worth calling out:

  • swappiness lowered to 10. Elasticsearch and the engines are latency-sensitive. Failing fast is better than thrashing in swap.
  • NTP enabled. Events from multiple engines get correlated on a timeline, and clock skew breaks correlation outright.
  • Two Redis instances. One is general purpose and one is reserved as the engines’ message queue. They’re isolated and get separate memory limits.
  • Two NICs with separate roles. The management NIC has an IP and serves the web UI and SSH. The mirror NIC has no IP and runs in promiscuous mode, so it only receives. Promiscuous mode is set by a systemd template unit, promisc@<nic>.service, with one instance per capture interface.

2. Parameterizing hardware tiers

The same scripts run on several hardware tiers, from entry-level to high-end. The differences are collapsed into a small per-tier file that holds nothing but environment variables:

# Illustrative: a mid-range tier
export PLAT_CPU_CORES=16
export PLAT_DPI_WORKERS=6
export PLAT_PROTO_WORKERS=6
export PLAT_ES_HEAP=16g
export PLAT_REDIS_MAXMEM=4g

The init script appends the right tier file to the global profile. Every service’s start script then reads worker counts and memory limits from the environment. The upside is that tier differences live in exactly one place. The downside is that environment variables are an implicit dependency. If a service forgets to source the profile, it quietly starts with defaults, and that’s painful to track down.

Two related tools: CPU isolation (isolcpus reserves cores for capture threads, with IRQ affinity adjusted to match) and cgroup memory caps (systemd’s MemoryLimit), which keep Elasticsearch from eating the whole machine.

3. Validate at delivery with known traffic

How do you know the box actually works after deployment? “All processes are up” isn’t enough. The deploy bundle includes a few test pcaps:

Capture Contents Expected result
Intel hit Traffic to known-bad IPs / domains Intel alert
C&C Lookups of algorithmically generated domains DGA alert
Office baseline Normal HTTP / HTTPS / mail No alerts
Combined Mixed protocols Output from several detectors

Replay them into the capture port with tcpreplay, wait thirty seconds, and check that the alerts match. “The office baseline stays quiet” matters as much as “malicious traffic fires.” It’s the regression test for false positives. The same check suite also verifies Redis read/write and pub/sub, MongoDB read/write, Flume status, sandbox configuration, and version compatibility across components.

4. The version manifest: a snapshot you can source

With thirty-plus components, each with its own build number, a single product version can’t tell you what’s on a box. The approach: each release gets a directory containing one version file made of bash variables:

# Illustrative manifest for release X.Y.Z
VER_DEPLOY="X.Y.Z-101"
VER_WEB="X.Y.Z-2040"
VER_FRONT="X.Y.Z-3170"
VER_DPI_ENGINE="X.Y.W-5120"     # unchanged this release, older build carried forward
VER_PFRING="A.B.C-900"
VER_PFRING_DKMS="A.B.C-900"
VER_ES="X.Y.W-310"
VER_KIBANA="X.Y.W-305"
# ... thirty-odd entries

Key design points:

  • A full snapshot, not a delta. Even unchanged components are pinned to an exact build. Any single manifest can reproduce a whole appliance on its own.
  • Components can stay on old versions. The DPI engine kept the same build across several releases because it had stabilized, and the manifest still lists it. A component’s version prefix tells you which release last changed it.
  • Sourceable and diffable. The upgrade script sources the target manifest to get its versions. To see what changed between two releases, diff the two files.
  • Compatibility constraints are explicit. Some components have to move in lockstep: PF_RING and its kernel module, ES and Kibana, NIC drivers and PF_RING. They sit next to each other in the manifest, and a validation script checks that they match.

The format is major.minor.patch-build. Build numbers come from CI and only go up, so it’s easy to tell which is newer.

5. Orchestrating upgrade and rollback

The upgrade service is itself a small Django + Celery app. Django serves status and history endpoints, and a Celery worker runs the long steps in the background. You can trigger it from the CLI, a REST endpoint, or an “Upgrade” button in the console, and all three run the same flow.

precheck ─▶ backup ─▶ stop services ─▶ install packages ─▶ DB migration ─▶ config migration
                                                                                 │
       ◀── rollback on failure ◀── post-checks ◀── start services ◀──────────────┘
Stage What happens
Precheck Free disk space, all packages present, target version valid
Backup MySQL / MongoDB dumps, ES snapshot, configs archived under a timestamped dir
Stop Detection engines, event pipeline, web tier, in order
Install RPMs upgraded in dependency order
Migrate SQL migrations, ES mapping updates, reindex if needed
Config Merge new templates while keeping user customizations
Start + verify DBs, backend, engines, web, in order. Check processes, APIs, logs

Rollback runs the other way: stop services, restore databases, restore configs, downgrade RPMs, start, verify.

There’s also a lightweight path that updates only the knowledge base. Intel, rules and taxonomy mappings change far more often than the code, so they shouldn’t need a full upgrade every time. One config switch selects a full upgrade or a knowledge-only sync.

Each upgrade opens a task record in a history table with the source and target versions, and writes back the result and any error at the end. A global “upgrade in progress” flag tells the console to lock out other operations while it runs.

Pitfalls

Don’t compare versions as strings. One “is this gap too big?” check was written as [[ "$current" < "X.Y.20" ]]. Bash’s < compares lexicographically, so "2.3.9" > "2.3.12". Use sort -V or compare each segment as an integer.

version_lt() { [ "$1" != "$2" ] && [ "$(printf '%s\n' "$1" "$2" | sort -V | head -1)" = "$1" ]; }

Check the return code of every install. Early code ran os.system("rpm -Uvh ...") and ignored the result. When one package failed, the flow kept going and reported success, but component versions no longer matched the manifest. That’s why comparing the manifest against what’s actually installed after an upgrade is mandatory.

The “in progress” flag must recover on its own. It’s set to true at the start and false at the end. If the process crashes in between, the flag stays true forever and the console stays locked. Use try/finally, or give the flag a timeout or heartbeat.

Downgrading an RPM isn’t -U in reverse. RPM refuses to install an older version by default. Rollback needs --oldpackage, and the old packages must have been kept somewhere. So the backup stage should cover the currently installed packages, not just data and configs.

Database migrations are usually one-way. Rollback restores the pre-upgrade backup. It doesn’t run migrations backwards. Any data written during the upgrade window is lost on rollback, and users should be told that up front.

Upgrade across big gaps in steps. Jumping over intermediate releases can skip their migration scripts. Either walk through each intermediate release, or make migrations numbered and cumulative and record which ones have run (the same idea as Django migrations).

The orchestrator must not stop itself. The upgrade service is also a platform component. Exclude it from the “stop everything” list and update it last, or hand that step to a separate one-shot systemd unit.

Wrap-up

The delivery layer comes down to three rules:

  • Initialization should be idempotent and resumable.
  • Describe versions as full snapshots you can diff.
  • Upgrade only after a backup, check every step’s return code, and verify the final state against the manifest.

Once the appliance is installed and upgraded, it has to keep running. The final post covers how the platform monitors itself, restarts processes that die, and lets operators reach it remotely.

The Series

  1. The Big Picture
  2. 10G Packet Capture
  3. Suricata DPI and Rule Management
  4. Protocol Analysis with Zeek
  5. An Event Pipeline with Flume and Esper
  6. Threat Intel Matching and the Knowledge Base
  7. Rule-Based Anomaly Detection
  8. Machine Learning Detection
  9. Alert Correlation, Noise Reduction and Aggregation
  10. Dynamic Sandbox
  11. Multi-Engine AV Scanning
  12. The Management Console
  13. A Lightweight Sensor
  14. Deployment, Versioning, Upgrade and Rollback
  15. Ops Monitoring and Self-Healing

This series is a personal study of system architecture. Details are abstracted and anonymized; it does not describe any specific commercial product or any organization’s internal configuration or data.

Comments

  1. Loading comments…

Leave a comment

Plain text; links are fine. Comments with many links wait for review.