Product Analytics and User Behaviour Tracking

Event collection and analysis for understanding how people use a digital product.

We built a product analytics workspace that takes a team from a product question to the events behind the answer. A team can compare funnels, retention and user paths, then save what it finds in dashboards or notebooks. To look closer, it can open event details and replay a recorded session.

Feature flags and experiments sit next to those views. They give a controlled product change the context the team needs to read its results.

The challenge

Looking back over old data is costly, and fresh events keep arriving while that work runs. We had to split the two jobs. Collection could then keep taking new activity while a query over the history was still running.

Identity adds a second problem to solve. Someone can browse while signed out and then sign in. Those early events need a processing order and a link to that person's record.

What we built

A capture service checks each incoming event before it reaches Kafka. It reads the event, its project token and any restrictions, then passes the accepted data on. Ingestion works out who the person is and applies the changes the team set up. The events that result are stored in ClickHouse.

The interface offers trends, funnels, retention, paths, lifecycle and repeat-usage views. Dashboards gather the questions a team asks again and again, while alerts help with follow-up. A long query shows its status, so a user can check it or cancel it instead of waiting with no sign of progress.

Keep collection and analysis apart

The capture path deliberately stays off PostgreSQL. If the queue is unhealthy, a storage fallback is on hand. Collection still needs a working destination, and it still answers to the rejection policies.

Processing groups events by project and by person. For one key, ordered streams limit how much work runs at the same time. An offset ledger moves forward only past work that finished in order. Delivery is at least once, so a replay is possible. Dead-letter topics and ingestion warnings surface any processing that failed.

Carry identity close to the event

Event rows hold person and group properties, which cuts the joins back to current identity records. When a later identity change makes a stored snapshot wrong, override data puts it right.

The typed query compiler adds project boundaries to the tables it supports. Raw or external database paths need their own separate controls. Query runners can return cached results, refresh them in the background, or run a query without making the user wait. Redis holds the status and the cancel signal for each one. Interactive traffic can also run on its own workload, apart from API and batch queries.

Project success

Teams can trace an aggregate result back to the recorded activity, and switch to a different analytical view when the question changes. Operators hold separate controls for collection and reporting. Ingestion warnings and query state give them a way to look into work that did not finish. Each chart arrives with evidence about the data and the processing that produced it.

Technology stack

Interface

React, TypeScript, Kea

Application services

Python, Django, Node.js

Intake and systems services

Rust

Storage and messaging

ClickHouse, PostgreSQL, Kafka, Redis

Background orchestration

Celery, Temporal, Dagster