Asked for a database every system would switch to, we built a stream none of them had to.

Customer records that only changed overnight now reach a dozen systems, in the cloud and on premises, within seconds, and none of them had to switch databases to get there.

The situation

There was no central, authoritative store of customer records. Every night the data lake was reloaded in full from an external customer data source, and everything downstream, the search layer included, rebuilt from it. A change made during the day had nowhere to go until the next reload.

That ruled out what came next. The internal dashboard had to show customer changes as they happened, and the onboarding frontend and the customer portal had to let people update their own details.

What we proposed instead

A database every system switches to makes every system a project: a dozen teams rewriting their reads, on their own schedules, against a schema none of them chose. We turned it around. The database became one source of changes among others, and the stream became the contract. Each system keeps reading from the store it already uses, and changes arrive there.

The search layer stayed the interface for most customer data, and kept its performance. The daily batch kept arriving. It is merged into the stream instead of overwriting everything downstream.

How it holds together

• Every field has one owner. Each source owns specific fields, so the daily batch and a change made an hour earlier never compete for the same value. Merging is not a conflict to resolve.
• Writers see their own change at once. Writes go through one API, and one call both updates the search layer and publishes to the event log. The person who made a change sees it immediately, and every other system follows within seconds.
• New consumers start from the full picture. The event log is a compacted Kafka topic. A system that joins later reads it from the beginning and ends up with the current state of every record.
• The events are ours, not the database’s. Change capture is built into the application rather than read from the database’s internals, so a schema change in a table never leaks to a dozen consumers. We decide what a change means and when it is allowed to break.
• Nobody integrates with Kafka. Stream processors we maintain deliver each consumer’s data in the schema it needs, into the store it already uses, and keep that schema compatible as things change. On-premises systems receive events through a managed messaging service and run their own subscribers.

What it changed

• A change made in the portal is searchable within seconds. It used to take a day.
• A dozen systems, from cloud services to on-premises deployments, receive half a million customer records and every change to them, while only a few talk to the new database directly.
• The analytics stack became a sink instead of a source.
• Alerts and a user-facing activity log were not in the brief. With every change already on a stream, in events we define, each was another consumer.

What we did not do

The nightly reload from the external source still runs; it is merged, not retired. Most systems never moved to the new database, which was the point. And the on-premises subscribers belong to the teams that run those systems: we deliver the events, they own what happens next.

Delivery

Two engineers, end to end. In production for a year and a half, with no downtime.