Ops Monitoring and Self-Healing: Teaching the Platform to Look After Itself (Part 15)
A single detection appliance runs more than twenty long-lived services: two Elasticsearch nodes, two Redis instances, MySQL, MongoDB, three Flume agents, the intel matcher, the correlation engine, the periodic aggregation jobs, the sandbox scheduler, Celery, uWSGI, Daphne, Nginx and Kibana, plus open-source ClamAV and several commercial AV engines. The box usually sits in a customer’s data center with nobody watching it around the clock. If any one process quietly exits, the detection chain can break at that point, and the alert page will just show “all quiet.”
So the last post in this series covers the platform’s self-care layer: noticing problems (monitoring), fixing them automatically (self-healing), and letting a human in when automation isn’t enough (remote terminal). It’s built from four small open-source tools: Monit, Glances, net-snmp and ttyd.
Who does what
┌────────────────── on the appliance ──────────────────┐
process/port/ │ Monit: sweep every 30s → alert / auto-restart │
disk │ │
resource │ Glances: sample every few s → export to ES → Kibana │
metrics │ │
external │ net-snmp + collector script → customer NMS pulls │
interface │ │
human access │ ttyd: a terminal in the browser (behind a proxy) │
└───────────────────────────────────────────────────────┘
| Tool | Question it answers | Action |
|---|---|---|
| Monit | Is the service alive? Are its resources out of control? | Restart, alert, run a script |
| Glances | How did CPU / memory / disk / network trend this week? | Records only, never intervenes |
| SNMP scripts | The customer’s NMS wants CPU, memory and disk usage | Exposes numbers to third parties |
| ttyd | Automation can’t fix it and a human needs to look | Provides a shell |
1. Monit: the process watchdog
Monit is a veteran tool: a small C daemon that checks “services” on a schedule and takes an action when a condition is met. Each managed service gets its own config file under monit.d/, more than twenty in total.
Identifying a process
Monit can identify a process in two ways:
# Option 1: PID file (preferred)
check process redis with pidfile /var/run/redis/redis.pid
start program = "/usr/bin/systemctl start redis"
stop program = "/usr/bin/systemctl stop redis"
# Option 2: command-line match
check process intel_matcher with matching "intel_matcher.py"
start program = "/usr/bin/systemctl start intel-matcher"
stop program = "/usr/bin/systemctl stop intel-matcher"
PID files are precise and cheap to check. Command-line matching is for Python scripts that don’t write a PID file, and the match string has to be unique. This box runs two ES instances, a hot node and a data node, and their command lines are nearly identical. Only their config directory paths tell them apart. Write matching "elasticsearch" and Monit treats the two as one.
Start and stop are both delegated to systemctl. Monit only makes the judgment call. systemd still owns the processes, so dependency ordering, environment and logging stay consistent.
More than “is it alive”
check process flume_meta with matching "flume-meta.conf"
start program = "/usr/bin/systemctl start flume-meta"
stop program = "/usr/bin/systemctl stop flume-meta"
if cpu > 90% for 10 cycles then restart # stuck CPU
if totalmem > 8 GB for 5 cycles then restart # memory leak
if 3 restarts within 5 cycles then timeout # stop flapping
check filesystem data with path /data
if space usage > 85% then alert
if space usage > 95% then exec "/usr/local/bin/purge-old-logs.sh"
check host es_http with address localhost
if failed port 9200 protocol http then alert
The key points:
for N cycles. Act only after N consecutive breaches, so a momentary spike doesn’t cause a restart.timeoutagainst flapping. If a service crashes as soon as it starts, restarting it forever just floods the logs and burns resources. After three failed restarts, Monit gives up and hands it to a human.- Disk thresholds that trigger cleanup. This matters most for the ES and Flume spool directories. On a detection box, the usual cause of a total outage isn’t a crash. It’s a full disk.
- Port probes. A running process isn’t necessarily a working service. Only an HTTP probe catches “alive but not answering.”
Start delay
The global config sets start delay 240, so Monit waits four minutes after boot before its first sweep. JVM services like ES, Kafka and Flume take minutes to cold-start. Checking too early mistakes “not up yet” for “dead,” and then Monit and systemd both try to start the same service.
2. Glances: resource metrics into Elasticsearch
Monit is good at “right now” and bad at trends. Trends go to Glances, a psutil-based Python monitor. Its plugins collect CPU, memory, load, disk I/O, filesystems and NIC traffic, and it can export to a dozen-plus backends. Here it exports to the local Elasticsearch with daily indices, and the charts live in Kibana or Grafana.
glances --export elasticsearch -C glances.conf \
--disable-process --disable-sensors
The configuration makes a few trade-offs:
- Process-list and sensor plugins are off. Per-process collection is the most CPU-hungry part of Glances, and Monit already handles process health.
- Noisy interfaces are hidden. The sandbox creates a lot of
vnet*,virbr*andtap*virtual NICs. Without a filter, they take over every chart. - Three threshold levels. careful / warning / critical (for example 50 / 70 / 90%), configurable per mount point so the data disk can turn yellow earlier.
[network]
hide = lo.*, virbr.*, vnet.*, tap.*, docker.*
[fs]
/data_careful = 40
/data_warning = 70
/data_critical = 90
Pitfall: monitoring data lives in the same ES as detection data. It saves running a second store. But when ES has trouble, you lose the data that would show the trouble. And metrics arriving every few seconds compete with detection events for write throughput. At minimum, give monitoring its own index prefix and its own retention (say, delete after 30 days), and have Monit, not Glances, decide whether ES itself is alive.
Glances can die too, so a cron job checks its systemd state every five minutes. Add systemd’s Restart=always and Monit, and one service can end up with three watchdogs. That leads to the most important lesson of this post, covered below.
3. SNMP: a standard way out for the customer’s NMS
Many customers already run their own monitoring platform (Zabbix, Nagios and the like) and want just three numbers: CPU, memory and disk usage. For them the box runs net-snmp’s snmpd plus a collector script:
cron every 5 min ─▶ collector ─▶ snmpwalk HOST-RESOURCES-MIB
│
compute percentages, write three small files
│
customer NMS ─▶ reader script (cpu / mem / disk) ─▶ returns "45.50%"
The calculations have a few details worth knowing:
- CPU is the average of
hrProcessorLoadacross all cores. - Memory is RAM used minus buffers and cache. Linux fills free memory with cache, so the raw “used” figure always looks close to 100%.
- Disk sums only storage entries whose type is “fixed disk,” which leaves out RAM, swap and virtual storage.
Some things to improve. The collector calls snmpwalk several times per storage entry, one after another, which gets slow as entries pile up. Walk the whole table once and parse it locally instead. The output files hold a bare percentage with no timestamp, so if cron stops, the external system can’t tell the number is stale. Write the collection time next to the value. And SNMPv3 is set to authentication without encryption (authNoPriv). If remote queries are needed, move to authPriv and restrict source addresses.
4. ttyd: a terminal in the browser
The last piece is for humans. ttyd is a small statically linked binary that exposes a command, usually a shell, as a web terminal over WebSocket. Customer sites often allow HTTPS but not SSH, and then ttyd is the operator’s only way in.
browser ──HTTPS──▶ Nginx (TLS, auth) ──WebSocket──▶ ttyd (localhost only) ──PTY──▶ shell
It’s also the riskiest component on the box, because one web page gets you a shell. Nearly every deployment rule is about security:
- Bind ttyd to localhost and put Nginx in front for TLS termination. The proxy must forward the
Upgrade/Connectionheaders and use a long read timeout, or WebSocket sessions get cut off periodically. - Handle authentication at the Nginx layer, where it can plug into single sign-on, rather than putting a fixed username and password in ttyd’s launch flags.
- Run as a non-root user where possible (
-u/-g) and use sudo when elevation is needed. - Use
-Rfor read-only demo sessions. - Record sessions, or at least command history, for audit.
- Let the console turn it on and off with one click, and keep it off when it isn’t needed.
The biggest lesson: watchdogs need a hierarchy
Looking back at this layer, the most trouble didn’t come from any one tool. It came from stacking several self-healing mechanisms on top of each other:
| Mechanism | Granularity | Best used for |
|---|---|---|
systemd Restart= |
One process exiting | Bringing a crashed process back within seconds |
| Monit | Process + resources + ports + disk | Catching “alive but unhealthy” and restarting with debounce |
| cron check scripts | Anything | A last-resort safety net, ideally alert-only |
If all three can restart the same service, you get this: systemd brings it up, Monit decides it isn’t ready and runs stop/start, then cron does it again. The service gets yanked back and forth, and in the logs it looks like the service itself is unstable. The convention we recommend: systemd handles crashes, Monit handles health judgments, and cron only alerts and never acts. Monit’s start and stop commands should also always go through systemctl and never launch binaries directly.
Self-healing also hides problems. A service that restarts ten times a day stays green on the dashboard. Restart count should be a metric you monitor and alert on.
Wrapping up the series
Over fifteen posts we started from a single mirror cable and took a network threat detection platform apart from the bottom up:
remote sensors ─┐
▼
10G capture (PF_RING ZC) ─▶ Suricata / Zeek engines ─▶ Flume + Esper pipeline
│
┌───────────────┬────────────────┬──────────────────────┤
▼ ▼ ▼ ▼
intel matching rule-based anomaly ML detection sandbox + multi-AV
└───────────────┴───────┬────────┴──────────────────────┘
▼
alert correlation, noise reduction, aggregation ─▶ management console
│
deployment / versioning / upgrade-rollback + ops monitoring & self-healing (the floor)
If I could keep only a few takeaways, they’d be these:
- Capture sets the ceiling for everything. Packets lost at the bottom can’t be recovered by anything smarter higher up (Parts 2 and 13).
- Multiple engines complement each other, but they need one exit. Suricata is strong on signatures, Zeek on protocol semantics, and ML on things nobody has seen before. All of them have to feed one event pipeline and speak one schema (Parts 3–8).
- An alert is only worth something if alerts are few and accurate. Correlation, allowlists, asset baselines and periodic aggregation decide whether an analyst sees 50 alerts a day or 50,000 (Part 9).
- You only know what a file does once you run it. The sandbox and multi-engine AV cover what the traffic layer can’t see (Parts 10 and 11).
- Engineering decides whether it ships. Version snapshots, upgrades you can roll back, and self-healing with debounce aren’t part of the detection path. They decide whether the system can run for years in a data center nobody is watching (Parts 12, 14 and 15).
Thanks for reading this far. I hope this breakdown is useful if you’re designing or running a system like this.
The Series
- The Big Picture
- 10G Packet Capture
- Suricata DPI and Rule Management
- Protocol Analysis with Zeek
- An Event Pipeline with Flume and Esper
- Threat Intel Matching and the Knowledge Base
- Rule-Based Anomaly Detection
- Machine Learning Detection
- Alert Correlation, Noise Reduction and Aggregation
- Dynamic Sandbox
- Multi-Engine AV Scanning
- The Management Console
- A Lightweight Sensor
- Deployment, Versioning, Upgrade and Rollback
- Ops Monitoring and Self-Healing
This series is a personal study of system architecture. Details are abstracted and anonymized; it does not describe any specific commercial product or any organization’s internal configuration or data.
Comments