# The Management Console: Django, Celery and i18n (Part 12)

- URL: https://wuming.si/blog/4035/
- Author: Wuming
- Published: 2025-04-08
- Language: en
- Categories: Technology, English

> Previous: [Multi-Engine AV Scanning](https://wuming.si/blog/4034) · Next: [A Lightweight Sensor](https://wuming.si/blog/4036)

The first eleven posts were all about detection: packet capture and DPI, intel and models, correlation, then sandboxing and AV. The alerts, assets and reports those components produce all end up in one place, the **management console**. Analysts use it to review alerts, trace incidents and tune policies. Admins use it to manage users, run upgrades and check system health.

The console is the top layer of the system. It sits outside the main data path but talks to almost every component. Its job is to **wrap a pile of different backends in one consistent interface with access control**.

This post has three parts: the web backend (Django), background jobs (Celery) and a supporting Java service. Then we look at how the Chinese/English internationalization works.

## Overall Structure

```
        Browser (single-page frontend)
                │ HTTPS
                ▼
        ┌──────────────┐
        │    Nginx      │  static files, TLS termination, reverse proxy
        └──┬────────┬──┘
           │        └──────────────────────┐
           ▼                                ▼
   ┌──────────────┐                ┌──────────────┐
   │ uWSGI+Django  │  main API      │ Spring Boot   │  heavy queries / big files / imports
   └──┬───┬───┬───┘                └──────┬───────┘
      │   │   │                            │
      ▼   ▼   ▼                            ▼
   MySQL Redis Elasticsearch  ←──── same shared stores
      ▲
      │
   ┌──────────────┐
   │ Celery worker │ ← scheduled by Celery beat
   └──────────────┘
```

Each store has a clear job:

| Store | What it holds |
|---|---|
| MySQL | The business database (users, roles, policies, assets, report metadata), plus a separate knowledge base (threat taxonomy, rule metadata, kill-chain mapping and so on; see Part 6) |
| Redis | Sessions, caches, and the Celery broker and result backend |
| Elasticsearch | Events and alerts from every detection engine: high volume, and you need to search and aggregate them |

## The Django Backend: One App per Domain

The Django project is split into a dozen or so apps by business domain, roughly:

- **Assets**: active and passive discovery, the asset inventory;
- **Alerts and incident tracing**: alert lists, handling status, tracing multi-stage attacks, tracing web attacks;
- **Risky hosts**: risk scores rolled up per host;
- **Reports**: daily, weekly, monthly and yearly reports rendered to PDF;
- **Overview and dashboards**: the landing-page dashboard;
- **Allowlists**: global IP, domain and hash allowlists;
- **System settings**: user management, access restrictions, email alerts, upgrades and detection policies (turning rules on and off, sandbox policy, reputation filtering, detection mode, PCAP replay, HTTPS inspection and more).

All URLs live under one versioned API prefix, and the frontend talks only to that REST API. Putting the version in the path means an old frontend and a new one can coexist briefly during an upgrade without stepping on each other.

### Reverse Proxy: One Front Door for Every Backend

The console also contains a fair amount of other people's UI: Kibana for raw events, direct Elasticsearch queries, a web terminal (ttyd), the ML service's API and the Java service's API. The browser never reaches any of them directly. Everything **goes through Django proxy views**:

```python
# Illustrative: an authenticated reverse proxy
@login_required
@permission_required("view_raw_events")
def kibana_proxy(request, path):
    upstream = f"{KIBANA_UPSTREAM}/{path}"
    resp = forward(request, upstream)     # pass through method, params, body
    return strip_hop_by_hop(resp)
```

The payoff is that **authentication and authorization happen in exactly one place**. Every backend listens only on localhost, and Nginx is the only thing exposed, so the attack surface is much smaller.

### Authentication, Authorization and Auditing

The admin UI of a security product is a high-value target in its own right, so this part is fairly heavy:

- **Authentication**: session and token backends side by side. Tokens serve the frontend and scripts.
- **RBAC**: user → role → permission, with permissions down to individual actions.
  > **RBAC (role-based access control)**: instead of granting permissions to users directly, you grant them to roles and assign roles to users.
- **Password policy**: minimum length, complexity, expiry, lockout after repeated failures, optional two-factor auth.
- **Session timeout**: configurable, and a background job actively expires stale sessions.
- **Access restrictions**: limit logins by source IP and time window.
- **Audit**: a middleware logs every user action, so you can always see who changed which policy and when.

These cross-cutting concerns are Django middleware, chained in order on each request:

```
request → session check → permission check → audit log → view → response
```

## Celery: Moving Slow and Scheduled Work Out of the Request

Web requests have to be fast, but the console has plenty of work that's either slow or needs to run on a schedule. Celery handles all of it.

> **Celery**: a distributed task queue for Python. The web process drops tasks on a broker (Redis here), worker processes run them asynchronously, and a companion "beat" process enqueues tasks on a schedule.

The periodic jobs look roughly like this:

| Job | Frequency | What it does |
|---|---|---|
| Report generation | Early morning, daily / weekly / monthly / yearly | Summarize alert stats, render PDF |
| Overview cache | Every few minutes | Precompute dashboard aggregations into Redis |
| Risky-host cache | Every few minutes | Same idea, for the risky-host list |
| Status push | Every few seconds | Push system status to the browser over WebSocket |
| Session check | Every few seconds | Expire timed-out sessions |
| Email alerts | Hourly | Email a digest of new alerts |
| Stale asset cleanup | Twice a day | Flag or remove assets not seen in a long time |
| Upgrade check | Polling | Look for pending upgrade jobs (Part 14) |

There are also plenty of **on-demand** tasks: active asset probing, PCAP replay, system reset, rule sync, ad-hoc reports.

```python
# Illustrative: a cache precompute task
@app.task(time_limit=600)
def refresh_overview_cache():
    data = {
        "top_threats": es_agg_top_threats(hours=24),
        "trend":       es_agg_trend(hours=24, interval="1h"),
        "risk_hosts":  top_risk_hosts(limit=10),
    }
    redis.set("overview:latest", json.dumps(data), ex=900)
```

**Why precompute?** The landing dashboard runs a batch of Elasticsearch aggregations, and on a busy system that can take several seconds. If every user who opens the page triggers them live, Elasticsearch gets hammered. Once a background job computes them every few minutes and the frontend reads only the cache, the page opens almost instantly. The cost is a few minutes of staleness, which is fine for a dashboard.

Some lessons from tuning the workers:

- **Recycle child processes after N tasks.** Jobs like report generation tend to leak memory, and periodically recycling workers is the least painful fix.
- **Prefetch of 1.** Task durations vary a lot, and you don't want one worker hoarding long jobs while the others sit idle.
- **Hard time limits.** Every task has a ceiling, so a hung job can't occupy a worker forever.
- **Schedule stored in the database.** The beat schedule can be adjusted from the UI without code changes or restarts.

## The Java Service: Heavy Queries and Big Files

Alongside Django there's a Spring Boot service. It uses MyBatis against the same MySQL and queries Elasticsearch directly. It handles a few kinds of work:

- **Situational awareness and a search center**: unified search across event types, plus heavy aggregation queries for trends and geographic spread, with caching.
- **Large uploads**: PCAPs and samples that can run to gigabytes.
- **Bulk import/export**: Excel templates for loading knowledge base entries, asset data and so on.
- **Auxiliary data** such as domain ownership.

The frontend reaches it through the Django proxy too, so the browser only ever sees one backend.

Why two backends at all? The split looks like something that grew as the system evolved, not something planned up front, and it has real costs. Two codebases touch the same tables, so every schema change has to land in both. There are also two sets of deployment scripts and two sets of health checks. Starting over, we'd lean toward one primary backend, with heavy queries exposed as a focused service rather than a second full web backend.

## Internationalization: One Source, Many Consumers

The console defaults to Chinese and also needs an English UI. The trouble is that text needing translation is scattered everywhere:

- buttons, labels and messages in the web UI;
- threat categories, subcategories and kill-chain stages, which come from the knowledge base;
- protocol names, device types, operating systems and geographic locations;
- **alert names from detection rules**, tens of thousands of them;
- Kibana dashboard titles and panel names.

All translations live in a dedicated translations repo, split by domain into twenty-odd JSON files:

```
translations/
├── zh/
│   ├── web-ui.json
│   ├── alarm-message.json
│   └── kibana/dashboards/…
└── en/
    └── … (only entries whose English display text differs from the key)
```

The format is plain key-value, and **the key is the English source text**:

```json
{
  "Remote Control": "远程控制",
  "alarm_name": "告警名称",
  "Request parameter error": "请求参数错误"
}
```

### The Merge Script

A merge script reads every file and writes out a few final files, one each for the web frontend, Django and Kibana:

```python
# Illustrative: merging translations
def load_tree(root):
    merged = {}
    for path in sorted(glob(f"{root}/**/*.json", recursive=True)):
        for k, v in json.load(open(path, encoding="utf-8")).items():
            if k in merged and merged[k] != v:
                log.warning("override %r: %s -> %s (%s)", k, merged[k], v, path)
            merged[k] = v
    return merged

zh = load_tree("zh")
en = {k: k for k in zh}          # English defaults to the key itself
en.update(load_tree("en"))       # the few English overrides go on top
write_all({"zh": zh, "en": en})  # outputs for frontend, Django, Kibana
```

A few key points in this design:

- **English falls back to the key.** Because the key is already English text, the English UI can show the key directly in most cases. The `en/` directory only needs the handful of entries whose key isn't fit for display, such as variable-style keys like `alarm_name`. That keeps English maintenance very cheap.
- **Warn on duplicate keys.** With twenty-odd files merged together, the same key will sometimes be translated two different ways. The script logs an override warning and a human decides which one wins.
- **The knowledge base is the source for some translations.** Threat categories and similar data already live in the knowledge base with Chinese and English columns. An export tool dumps them to JSON so they go through the same merge, and nobody ends up editing two copies.

### Release

The translations repo ships as its own system package. Its version number comes from the commit count, so it always goes up. A post-install hook runs the merge script and restarts the affected services. In other words, **changing a translation doesn't require re-releasing the whole platform**.

## Pitfalls and Trade-offs

- **The translation file is big.** Rule alert names alone run to tens of thousands of entries, and the merged file is several MB, which is heavy for the frontend to load in one go. Better options: split files by domain and load them on demand, and fetch things like rule names through an API instead of bundling them.
- **Caching.** After a translation update, the browser cache and Kibana's frontend build cache can both hide the new text. The release process has to bust those caches, or users will assume nothing changed.
- **English is a second-class citizen.** Chinese coverage is nearly complete, while English coverage is much lower and leans on "the key is the English". Where key styles are inconsistent, a raw variable name leaks into the UI. This needs ongoing investment.
- **Translation values are input too.** Translated text is rendered into pages, so it has to be escaped and never injected as raw HTML.
- **Keeping two backends in sync**, covered above, is the biggest piece of architectural debt in this layer.

## Wrap-up

Four ideas carry the management console:

- Django is the single front door. It handles authentication, authorization and auditing in one place and reverse-proxies every other backend.
- Celery runs all slow and scheduled work, and its precomputed caches protect Elasticsearch.
- A Java service handles heavy aggregation queries and large files.
- Internationalization uses one key-value translation source, a merge script and an independent release, and serves the frontend, the backend and Kibana from the same files.

So far we've described the full platform as deployed in a central data center. Often, though, all you need at a branch office is a small sensor. The next post covers the design and production deployment of a lightweight sensor.

## The Series

1. [The Big Picture](https://wuming.si/blog/4024)
2. [10G Packet Capture](https://wuming.si/blog/4025)
3. [Suricata DPI and Rule Management](https://wuming.si/blog/4026)
4. [Protocol Analysis with Zeek](https://wuming.si/blog/4027)
5. [An Event Pipeline with Flume and Esper](https://wuming.si/blog/4028)
6. [Threat Intel Matching and the Knowledge Base](https://wuming.si/blog/4029)
7. [Rule-Based Anomaly Detection](https://wuming.si/blog/4030)
8. [Machine Learning Detection](https://wuming.si/blog/4031)
9. [Alert Correlation, Noise Reduction and Aggregation](https://wuming.si/blog/4032)
10. [Dynamic Sandbox](https://wuming.si/blog/4033)
11. [Multi-Engine AV Scanning](https://wuming.si/blog/4034)
12. [The Management Console](https://wuming.si/blog/4035)
13. [A Lightweight Sensor](https://wuming.si/blog/4036)
14. [Deployment, Versioning, Upgrade and Rollback](https://wuming.si/blog/4037)
15. [Ops Monitoring and Self-Healing](https://wuming.si/blog/4038)

---

*This series is a personal study of system architecture. Details are abstracted and anonymized; it does not describe any specific commercial product or any organization's internal configuration or data.*
