An event-processing service that connects product activity to the metrics and subscriptions used for billing.
This usage-based billing case study covers a service that gets product activity ready for billing. It takes in usage events, works out what each one is worth, and ties it to a subscription before handing it on.
Our scope was this processing pipeline and its connectors. Invoice generation and the wider billing calculations sit outside it. The output is an enriched event that the billing system can read, and it stops before any finished charge or invoice.
The challenge
Arrival time can be misleading on its own. A usage event may reach the processor after the subscription has already changed. Reading only the customer's current state can then attach the wrong context.
A batch can also fail partway through. The pipeline has to separate work that is worth retrying from input that should be set aside for a closer look. At intake, an acknowledgement should mean the event reached the broker, instead of merely signalling that the HTTP request arrived.
What we built
Each event gets a source label and an arrival timestamp before it joins the queue. The HTTP connector waits for its Kafka write to finish, and authentication belongs at the gateway in front of it.
The processor looks up the right metric using the organisation and the metric code. A count metric is given a value of one. Other metrics read a named event property, or evaluate a custom expression. Metric definitions can handle sums and maximums. They also support unique counts, weighted sums and latest values.
Subscription lookup uses the time the event actually happened, and a recurring metric can fall back to a currently active subscription when that lookup finds nothing. When the context is still missing, the event moves on without the association, and downstream handling takes it from there.
Enriched events go to a separate topic. For billing in advance, the service flags the subscription that needs a refresh, and that flag asks for follow-up work kept separate from the enrichment itself.
Give failed events a defined path
The Go service reads the batches and hands each partition's work to a worker of its own. The results mark the retryable failures and the ones to capture.
A retryable failure inside the configured age window stays uncommitted. Other failures follow their policy into a dead-letter topic, which keeps the original event and an error code. Operators can then inspect the failed input, rather than rebuilding it from a status message.
Keep reference data current
An optional memory cache holds the lookup data used during enrichment. Change-stream consumers compare timestamps against the cached rows, dropping any update that is older or repeated, while a delete applies only when its identifier matches.
With caching on, every instance receives the full change stream and keeps its own copy. That saves repeated database lookups for cached references. It does need enough memory for a copy on each instance. The timestamp checks matter because the stream can deliver data older than the row already held.
Group subscription refresh flags
Redis sorted sets gather refresh flags into time buckets. Several events for one subscription can then share a single refresh request, instead of each sending its own downstream.
A connector build includes only the component groups its configuration calls for. The intake connector and the processing workers keep separate jobs. One accepts and queues events, while the other adds billing context and picks the failure path.
Project success
Downstream billing can receive the metric value and the subscription context with each event that enriches cleanly. Where the association fails, the missing context stays visible, and events that need a closer look keep their original input and error details in the dead-letter path.
The pipeline also sets out when work is accepted and when it is still unfinished. HTTP intake waits for the broker write. Retries follow an age-based policy, and repeated refresh flags can be grouped before any downstream handling.
Technology stack
Processing service
Go, franz-go
Database access
PostgreSQL, GORM, pgx
Event transport
Kafka, Redpanda Connect, Debezium
Cache and coordination
Redis, go-redis, BadgerDB
Expression evaluation
Rust library through CGO
Other engagements
Recognize the pattern?
If one of these looks like your situation, the discovery call is where we find out how close.
