Behaviour-Based Threat Detection and Security Decisions

A log-processing system that reads behaviour, records a decision and shares it with enforcement tools.

This threat detection case study covers a service that reads log activity, judges behaviour and records a decision. Connected enforcement tools then act on that decision. A firewall, proxy or application applies the response where traffic is controlled.

We kept detection and enforcement apart. One analysis pipeline can take in several log sources. Each protected service uses its own method to act on the decisions that come out.

The challenge

A detection rule needs context across events. The timing of an attempt can change what one log entry means, and so can other related attempts or a trusted address. Some enrichment also takes longer than parsing the event, as a DNS lookup does.

Operators want to study a rule before enabling it. Enforcement clients need a dependable way to follow each decision as it is added or removed.

What we built

Parsers turn incoming logs into structured events. Enrichment then adds detail such as the date, a DNS result or a location, and it can leave out trusted addresses. Scenarios group the events that remain and apply their rules. An alert reaches a local API. There, allowlists and policy profiles set the response that gets recorded.

Historical replay reads the logs' own timestamps, so past activity runs through the same rules. Simulation is the other test path. It records a scenario's alerts but holds back the usual response, so operators can check a rule before it enforces anything.

Technical approach

Go routines and channels separate the stages of work. Acquisition, parsing and scenario processing each run on their own, and output runs the same way. Stateful buckets hold the events that relate to one another. A leaky bucket drains at a set rate and raises an alert when it overflows. Event maps are copied before they enter a bucket, so two scenarios never share data that can change under them.

Later enrichment can cancel an alert once it spots trusted activity. Output workers do that work behind a bounded queue. If the queue fills, extra overflows may be dropped, which makes queue capacity something to watch in operation. Notification plugins run as separate processes.

Enforcement clients follow a stream of decisions that keeps their place, paging through additions and removals by identifier. The first synchronisation and the later updates begin from different positions. A decision written during a read still reaches the client afterward.

Each component authenticates to the API, yet a single API is still one shared trust domain. Keeping tenants apart therefore needs a separate deployment boundary.

Project success

The client can connect logs to stateful detection rules, replay past activity and test a response before switching it on. Enforcement clients can follow every change through a recorded cursor. Those are the controls we delivered. The service judges behaviour and publishes decisions, while blocking stays with the connected firewall, proxy or application.

Technology stack

Processing engine

Go, goroutines, channels

Rule configuration

YAML, Grok, expression filters

Decision storage

SQLite, MySQL or PostgreSQL

Request inspection

Coraza

Plugin processes

go-plugin

Component authentication

JWT, API keys, TLS certificates