A lot of the interesting architecture decisions in a self-hosted product aren't about the happy path — they're about what happens when a piece of optional infrastructure isn't there. XCloak uses Kafka for event fan-out across seven topics (alerts, incidents, agent tasks, audit, FIM alerts, YARA matches, async IOC-match jobs), but a fresh docker compose up evaluation, or a small production deployment, shouldn't have to stand up and operate a Kafka cluster just to get core detection working. So Kafka had to be genuinely optional, not "optional" in the way that quietly breaks half the product if you skip it.
What "optional" has to actually mean
The easy way to build an optional dependency is to wrap every call site in an if enabled check and call it done. The harder part — the part that actually matters — is deciding what happens to the behavior on the other side of that check, not just whether the code path executes.
KAFKA_ENABLED=false, or an unreachable broker, degrades functionality in specific, bounded ways:
- No external event fan-out. If you're piping XCloak alerts into an external SIEM or SOAR tool via Kafka, that stops. This is an expected, visible tradeoff for not running Kafka — you lose an integration point, not core functionality.
- IOC matching falls back to a synchronous path. Normally, IOC matching against a log is deliberately asynchronous — dispatched to a Kafka topic and handled by a dedicated consumer, specifically so a large indicator list doesn't add latency to every ingest request (this is described in more detail in the post on the detection pipeline). Without Kafka, that decoupling isn't available in the same form.
What does not degrade: Sigma rule evaluation, the 24 scheduled behavioral detectors, alert creation, the WebSocket real-time dashboard feed (which runs on Redis pub/sub, not Kafka), and SOAR playbook execution. None of that has a hard Kafka dependency, because none of it should — detection and response are the core product; Kafka is the plumbing that helps it scale and integrate outward.
Why not just require it
We could have made Kafka mandatory and simplified a chunk of conditional logic in the process. We didn't, for a reason that's more about who's actually running this software than about the code itself: a homelab user or a five-person security team evaluating XCloak for the first time is not well served by "step one, stand up a three-broker Kafka cluster with ZooKeeper." The BUSL free tier is explicitly aimed at self-hosted deployments up to 10 agents — exactly the size of deployment where requiring Kafka would be disproportionate operational weight relative to what's being protected.
The docker-compose path still includes Kafka by default, because for evaluation purposes it's easy enough to bring up alongside everything else in one command. The point isn't "don't use Kafka" — it's "don't require it," which is a different design constraint and a harder one to build correctly, because it means every Kafka-touching code path needs an honest answer to "what happens if this isn't here," not just a null check that silently swallows the feature.
The actual lesson
Optional infrastructure dependencies are easy to get wrong in a way that isn't visible until someone actually runs without them — the null check compiles, the tests pass because CI always has Kafka available, and the gap only shows up when a real user disables it and finds a feature silently broken instead of cleanly degraded. Treating "Kafka unavailable" as a first-class state to design for, rather than an edge case to guard against, is what made the difference here.