It's easy to describe XCloak as "an agent sends logs and you get alerts," but that sentence hides about six distinct steps, each of which had to be a deliberate decision. This post walks through what actually happens between an agent seeing something and an analyst seeing an alert on their screen.
Step 1: Collection
The agent gathers telemetry — process events, network connections, auth logs, file hashes — either by polling on an interval or, on Linux, through the real-time eBPF connection collector described in an earlier post. Either way, the output is the same shape: a structured event, not raw bytes.
Step 2: Ingest and normalization
Events (and externally-sourced logs, for agentless sources) arrive at /api/logs/ingest or /api/ingest/http/:id. The backend supports five source formats — CEF, LEEF, JSON, NDJSON, and plain syslog — and normalizes all of them into a common field set: src_ip, dst_ip, user, auth_result, event_id, bytes_sent/bytes_recv, JA3 hash, command_line, parent_image, and a handful of others depending on source type. Everything downstream operates on this normalized shape, not on the original format — the Sigma engine doesn't know or care whether a given log started life as syslog or CEF.
Step 3: Sigma evaluation, on the hot path
This is the part that surprises people: rule evaluation happens synchronously, on the ingest request itself, not in a background queue. For each tenant, the ruleset is cached in memory, pre-bucketed by log format — so a rule that only applies to windows/process_creation logs never even gets considered for a Linux auth log. That pre-filtering is what makes synchronous evaluation viable at volume; without it, every log would need to be checked against every rule regardless of relevance.
A match creates an alert row immediately. There's no "wait for the next batch job" — the alert exists in the same request that ingested the log.
Step 4: Correlation
Once an alert exists, a separate correlation pass checks whether it relates to other recent alerts — same source, same technique, same time window — and groups related alerts into an incident if the pattern crosses that threshold. This is intentionally decoupled from the alert-creation step itself: correlation logic changes more often than detection logic does, and keeping them separate means tuning one doesn't risk regressing the other.
Step 5: Fan-out
The new alert gets published to a Kafka topic (xcloak.alerts) for anything consuming events externally, and — separately — broadcast to connected dashboard sessions over WebSocket via Redis pub/sub. That second path matters architecturally: it means the alert reaches every open dashboard regardless of which backend replica happened to handle the original ingest request. Without the Redis layer, a multi-replica deployment would only notify whichever replica's WebSocket hub happened to have that particular browser connected.
Step 6: Async work that doesn't block any of the above
Two things are deliberately kept off this path entirely:
- IOC matching — checking the log against threat-intel indicators — runs through a dedicated Kafka consumer, not inline. IOC lists can be large, and a slow lookup shouldn't add latency to every single log write.
- Sigma hit bookkeeping (recording which rule fired, for analytics) is written asynchronously in a goroutine, so the actual alert-creation response isn't waiting on a write that's purely for later reporting.
Why this design, and not a queue-everything approach
The obvious alternative is to queue every log and process detection asynchronously across the board — simpler to reason about, easier to scale independently. We didn't do that for the primary Sigma path specifically because alert latency is the metric that matters most for a SOC tool: the gap between "something happened" and "an analyst can see it" is the thing XCloak exists to shrink. Everything that doesn't affect that gap (IOC matching, hit analytics) is queued; everything that does, isn't.
The tradeoff is real: synchronous evaluation means the ingest endpoint's latency includes rule-matching time, and that has to stay fast as rule counts grow. The per-tenant, per-format caching is what keeps that bounded — but it's a constraint we chose deliberately, not a limitation we're unaware of.