Reusable connectors for extracting source data, writing destination records and tracking sync progress.
This data integration case study centred on the connector layer. These connectors pull data out of business services and databases, then write it to analytical destinations. One shared interface handles the configuration and the access checks. Beyond that, it also runs stream discovery and manages the record transfer.
The important moment comes at the destination. A connector has to know which records were actually written before it tells the surrounding platform how far the sync has reached. Our scope stopped at that connector layer and the delivery parts that sit around it.
The challenge
No two sources behave in quite the same way. An API might page through its records one batch at a time, following its own rules. A database reader is a different problem. It brings tables, connection limits and a moving change log to track. Their recovery paths are different as well.
On its own, a source checkpoint does not amount to a finished transfer. The destination has to flush the matching data first. Only then is the checkpoint safe. We carried that link all the way through extraction, batching and acknowledgement.
What we built
Routine API behaviour lives in YAML manifests. A manifest sets out the authentication, the streams and the pagination. A shared runtime reads those definitions, and custom Python and JVM code handles the cases that do not fit. Containers give every implementation the same execution interface.
Databases follow a different route through the system. The read engine supports JDBC and change-data capture, and large tables can be split into sections for concurrent reads. State records tie each stream's progress to a position in the change log. A later run can therefore pick up where the last one stopped.
Technical approach
Records and state move as structured messages. The default is line-separated JSON over standard input and output. Where a path supports it, the connectors can use socket-based JSON or Protocol Buffers instead. Connection checks and discovery run on their own. Neither one needs a full transfer.
The Kotlin engine hands out table sections through serialized resource claims. The pool of available connections caps how much work can start at once. PostgreSQL paths add replication-slot management and resumable reads, so the tie between table progress and the database log holds.
At the far end, the load pipeline counts records against the state messages. A count only reaches the state stage after the destination has flushed, and a reconciler then publishes the completed checkpoints.
That ordering matters, yet it does not make delivery exactly once. A single retry can repeat the same records. Anything downstream has to plan for duplicate records, because this is an at-least-once contract rather than exactly-once.
Project success
A connector author can reuse a manifest for familiar API patterns. For the unusual behaviour they write custom code instead, and the surrounding interface does not change. Database reads can run in parallel inside the connection limits. Saved progress always points back to the writes that the destination has actually finished.
The client ends up with a clear recovery contract. Each connector also carries version and breaking-change metadata, which keeps releases manageable.
Technology stack
Connector definitions
YAML, JSON
Custom connectors
Python, Java, Kotlin
Database extraction
JDBC, Debezium
Load processing
Kotlin Flow, coroutines
JVM services
Micronaut
Packaging and transport
Docker, Protocol Buffers
Other engagements
Recognize the pattern?
If one of these looks like your situation, the discovery call is where we find out how close.
