arendalsoftware engineering

Two prior attempts didn’t ship. Two people delivered it in six months.

A search-driven platform behind several internal and customer-facing applications.

500msaverage search latency, from over 10s
20/200qps sustained, average / peak
1.5TBindexed and searchable
BEFORE10s+AFTER500msSourcesWarehouseBI layerAppSourcesWarehouseIngestSearchAPIAppBEFORE10s+AFTER500msSourcesWarehouseBI layerAppSourcesWarehouseIngestSearchAPIApp

The situation

Data from many source systems was aggregated into a cloud data warehouse. A BI layer sat on top of it and served the application directly.

Any fuzzy search took more than ten seconds. Paginating, sorting or re-filtering cost another ten. The system was read-only, so nothing could be changed from the application itself. Users were abandoning a platform that had already been paid for.

Already tried

Two prior engagements with large systems integrators did not ship. Both worked on making the BI layer faster.

The diagnosis

The access pattern was multi-tenant, with hard tenant isolation rules, and almost every query was a fuzzy full-text search across arbitrary attributes of a rich document. A columnar warehouse behind a BI layer is built for aggregate scans over structured columns. Tuning does not close that gap, because the gap is the workload. The fix was a data store matched to the access pattern.

Before: source systems fed a data warehouse, and a BI layer served the app, with fuzzy search over 10 seconds and no writes. After: ingestion of 1.5TB into Elasticsearch behind a web API with search and mutations, 500ms on average.
Before and after: the BI layer replaced by a search engine and a writable API.

What we built

  • Analysed every access pattern the application actually used.
  • Selected a data store matched to that pattern rather than to the reporting stack.
  • Deployed and ran Elasticsearch clusters across all environments.
  • Built a web API with mutation support, lifting the read-only ceiling.
  • Built ingestion pipelines for 1.5TB of indexed, searchable data.
  • Enforced tenant isolation in an application layer covered by unit and integration tests.
  • Migrated capability by capability. No downtime.

What we did not do

The warehouse stayed. The BI layer stayed for analytical dashboards, and the application population stopped needing seats for it. Data freshness stayed next-day: this work made realtime possible and did not deliver it. We left the existing ingestion sources untouched, and the client’s own data team ran their own validation before sign-off.

Delivery

Two engineers. Six months. No downtime at cutover.

Talk to the person who built it

If your platform has this shape, or an attempt to fix it has already stalled, we will tell you what we think in one call, before any proposal.

DanielCo-Founder

daniel@arendal.ca