XCloak / blog
Detection · Firewall ·

Enterprise Firewall + Deep Packet Inspection

Two major upgrades landed together: the firewall went from a basic rule list to a proper enterprise policy engine, and we added four new L7 detection services that inspect traffic at the protocol level. Here's how they work and why we built them the way we did.

The firewall problem

The original firewall implementation was functional but flat: a list of IP/port rules that the agent applied with iptables. There was no concept of rule direction (in vs. out), no port ranges, no expiry, no per-tenant default policy, no bulk operations. Every enterprise deployment has a different threat model and a different egress policy — a flat list with no metadata doesn't serve that.

The upgrade added nine new fields to each rule:

  • direction — in, out, or both. The agent now generates separate iptables rules for INPUT vs. OUTPUT (or both). On Windows, it creates separate netsh rules with -in/-out suffixes to avoid naming collisions.
  • port_range — accepts 8000-9000, 80,443,8080, or 8080. The Linux agent uses -m multiport --dports for comma-separated ranges and a simple --dport for single ports.
  • tags — a TEXT[] column, surfaced as a tag distribution in the stats API and filter chips in the UI.
  • expires_at — rules auto-prune via a goroutine reaper that runs every hour.
  • log_enabled / log_prefix — when set, the agent inserts a LOG rule before the ACCEPT/DROP target, writing matched packets to the kernel log with the prefix.
  • created_by / updated_by / updated_at — full audit trail for compliance.

Atomic application on Linux

The previous approach applied rules incrementally — one iptables -A call per rule. That means if the sync is interrupted halfway through, the firewall ends up in an inconsistent state. The new primary path builds a complete iptables-restore script in memory and pipes it with --noflush. Either the entire new policy is installed atomically, or nothing changes. An incremental path is retained as a fallback for hosts without iptables-restore.

Enterprise policy

Each tenant now has a firewall_policy record with a default action (allow/deny) and an enforcement mode (enforcing/audit/disabled). The dashboard exposes a one-click toggle and a per-rule template picker with 12 built-in starting points: SSH allowlist, HTTPS egress, DNS allow, SMB block, and others. Rules can be bulk-enabled, bulk-disabled, or bulk-deleted in one API call.

Conflict detection

The conflict detector was rewritten to understand the new fields. Two rules conflict when their source/destination IP ranges overlap (using net.IPNet.Contains in both directions), their port ranges intersect (interval overlap on the parsed [][2]int pairs), their directions are compatible, and they have opposite actions. The endpoint returns the full conflict pair so the UI can highlight both rules.

Deep Packet Inspection

Network detection in XCloak previously worked at the connection level: source IP, destination port, connection count, JA3 hash. We had no visibility into what was actually in the payload — the protocol layer. Four new scheduled detectors fix that.

DGA domain scoring

The old DNS detector used a single Shannon entropy threshold. That works well for obviously random domains (konficker-era pure-random DGAs), but it has poor precision on modern families that use word-based or pseudo-dictionary generators, and poor recall on short labels that don't accumulate enough entropy.

The new multi-factor scorer runs five independent checks and sums them:

  1. Shannon entropy (up to 40 points): captures truly random-looking labels.
  2. English bigram deviation (up to 30 points): measures how far the observed bigram distribution deviates from the top-80 English bigrams. Human-readable domains like "google" or "cloudflare" score near zero; algorithmically generated ones score high.
  3. Digit ratio (up to 20 points): DGA families like Necurs mix in many digits; legitimate hostnames rarely exceed 15% digit content.
  4. Consonant cluster length (up to 15 points): run of 4–5 consecutive consonants is a reliable DGA signal.
  5. Label length (up to 15 points): labels ≥18 characters are suspicious; ≥25 characters very suspicious.

On top of the label score, the tenant-level scorer adds: suspicious TLD bonuses (25 TLDs: .tk, .gq, .ml, .cf, .ga, .top, .xyz, .pw, .cc, .ws…), IOC database cross-check, DGA family pattern matching (Conficker: 8–16 chars, <10% digits; Necurs: 12–22 chars; Mirai-variant: 8–12 chars), and NXDOMAIN storm detection (≥8 events to the same SLD in 5 minutes). A score ≥60 fires an alert with MITRE T1568.002.

The DNS log pipeline (AnalyzeDNSLogEntry) now delegates to ScoreDomainDGA() instead of the old inline entropy check, so real-time DNS events and the 30-minute background sweep use the same model.

TLS anomaly detection

Five checks run every 15 minutes against the network_connect_events DPI fields and endpoint log parsed_fields:

  • Weak ciphers: 12 patterns covering NULL, EXPORT, RC4, DES, DES-CBC3, ADH, AECDH, ANON, MD5. NULL and EXPORT ciphers → critical; the rest → high or medium.
  • Deprecated versions: SSLv3 → critical/90 score; TLS 1.0 → high/75 per RFC 8996; TLS 1.1 → medium/60.
  • Self-signed certificates: detected via "self-signed"/"selfsigned" string in log messages, or by regex extraction of cert_issuer and cert_subject fields — a match between them indicates self-signed (MITRE T1553.004).
  • TLS on non-standard ports: TLS traffic outside the known-good set (443, 8443, 993, 995, 465, 636, 5061, 8883). Connections to port 80, 25, 53, or 110 with TLS → high severity — these are typical C2-over-TLS evasion techniques (T1571).
  • SNI/Host mismatch: when the TLS ClientHello SNI field differs from the HTTP Host header on the same connection (excluding the www./non-www variant), it's domain fronting — routing C2 traffic through a CDN using a legitimate domain as the SNI while the actual C2 hostname appears only in the encrypted HTTP layer (MITRE T1090.004).

HTTP inspection

HTTP is still the most common C2 transport. The HTTP inspection service scans three data sources: the http_user_agent DPI field in NCE, parsed_fields in endpoint logs, and raw log message regex extraction. Detection categories:

  • Malicious User-Agents: 35+ signatures. Cobalt Strike and Metasploit beacons have recognizable default UA strings. So do most commodity RATs (njRAT, AsyncRAT, Remcos, Warzone, DCRat). Scanner tools (Nikto, sqlmap, Gobuster, Feroxbuster, Nuclei) use identifiable UA strings that are almost never legitimate in production traffic.
  • Webshell paths: 20+ known webshell filenames (c99.php, r57.php, b374k.php, alfa.php, etc.) plus common admin panel paths that shouldn't exist on endpoints.
  • Path traversal: ../, ..\, and %2e%2e sequences in URL paths.
  • Null byte injection: %00 in paths — used to bypass extension checks in older PHP applications.
  • Suspicious HTTP methods: PROPFIND, TRACK, TRACE, DEBUG outside WebDAV environments. TRACE and TRACK can be used for cross-site tracing; DEBUG is used for IIS remote debugging.
  • High-entropy User-Agents: some C2 frameworks generate random-looking UA strings for each beacon. These are caught by the entropy threshold on the entropy_score NCE field.

Protocol anomaly detection

The final piece is protocol-layer anomaly detection — catching protocols running on the wrong port or being abused for tunneling. Highlights:

  • Protocol-on-wrong-port: we maintain a map of 9 protocols (SSH, FTP, SMTP, RDP, SMB, DNS, LDAP, NFS, VNC) to their expected ports. SSH on 443 or 80 — the most common C2 evasion — fires high severity. SMB on any port other than 445/139 does the same.
  • DNS tunneling: two signals — subdomain labels longer than 40 characters (DNS tunnel tools encode data in label content), and more than 200 unique SNI/hostname contacts in 10 minutes (indicating a DGA rotation or exfil campaign using DNS lookups as the channel).
  • ICMP tunneling: ICMP payloads larger than 1 KB are suspicious; larger than 4 KB is high severity. Standard ping payloads are 32–64 bytes.
  • HTTP CONNECT abuse: CONNECT requests to RFC 1918 addresses via a proxy — a technique for lateral movement that hides the actual destination behind the proxy.
  • DNS-over-TCP flood: more than 30 TCP connections to port 53 in 10 minutes. DNS-over-TCP is normal for large responses but high-volume TCP DNS is a reliable tunneling indicator.
  • SMTP on non-standard ports: SMTP traffic outside 25/465/587. This covers both misconfigured relay abuse and exfil via custom SMTP-like channels.

Passive DPI on the agent

The new DPI fields on network_connect_events are only useful if the agent actually populates them. On Linux, a new passive_dpi_linux.go collector extracts L7 metadata after each eBPF connect event, without a pcap library or kernel module:

  1. Walk /proc/<pid>/fd/ to find the file descriptor connected to the target address (matched via the socket inode in /proc/net/tcp6 and /proc/net/tcp).
  2. Open the socket and peek at the first 512 bytes without consuming them (the data stays in the kernel buffer for the application to read normally).
  3. If the first byte is 0x16 (TLS content type handshake) and byte 5 is 0x01 (ClientHello), parse the SNI from the SNI extension (type 0x0000) and the TLS version from the supported_versions extension (type 0x002b for TLS 1.3).
  4. If the buffer starts with an HTTP verb, extract method, path, Host header, and User-Agent.
  5. Compute a Shannon entropy score on the first 256 bytes and store it as entropy_score.

The entire operation runs in a goroutine with an 80ms deadline. If /proc access is unavailable (non-root, container with restrictive seccomp), the goroutine times out and the event is sent without DPI fields — the rest of the pipeline continues normally.

Non-Linux platforms get a no-op stub (passive_dpi_other.go). Full passive DPI on macOS and Windows requires different mechanisms (network extension / WFP driver) which are out of scope for now.

The dashboard

A new Deep Inspection page under the Network group surfaces all DPI findings with four summary stat cards, a breakdown pill row by finding type and severity, a full filterable table (by type, severity, alert-fired status, or free-text search), score bars (0–100), and an expand-to-details row that shows the raw JSON context for each finding. MITRE technique codes link directly to att.ck.mitre.org.

The firewall page received a parallel upgrade: direction badge (in/out/both with color coding), bulk select with a floating toolbar, a template picker modal with live search across 12 built-in templates, a default policy toggle, JSON import/export, per-rule expiry countdown, and tags display.

What's next

The DPI field set on network_connect_events is designed to be extended. The natural next step is JA3S (server-side fingerprint from the ServerHello) correlation against the existing JA3 detector — once we have both sides of the TLS handshake, we can flag mismatches between the client fingerprint and the server certificate characteristics. That, combined with the SNI/Host mismatch detector, gives much stronger domain fronting coverage.

GitHub · Docs · Live Demo