A tracking portal that stopped queueing behind itself
Peak hour timeouts on a shipment tracker, caused by a vendor system nobody was allowed to touch.
- p95 response
- 6.8s710ms-89%
- Throughput
- 1.0x3.9x+290%
- Tickets
- 100%46%-54%
Measured over 90 days post launch against the prior 90 days. Source: Datadog APM, April 2025
Context
A customer facing order tracking portal serving 38k daily active transactions, built on a Laravel monolith that queried the warehouse management system directly on every page load.
The constraint
The thing that ruled out the obvious answer.
The WMS was a vendor product with no API and a contractual ban on schema changes. Every optimization had to live outside a system we were not allowed to modify.
Diagnosis
Peak hour timeouts were not a database performance problem. Each page load fired 11 sequential queries against the WMS, and the vendor connection pool was capped at 25 concurrent connections. At peak the portal was queueing behind its own requests, and adding hardware would have changed nothing.
What we rejected, and what we chose instead
Any firm can describe what it built. The useful record is the option that was on the table and did not survive.
Rejected
Read replicas of the vendor database
The vendor licence priced per instance. Three replicas cost more per year than the entire project budget, and the contract gave us no way to negotiate it inside the timeline.
Chosen
Change data capture into a read optimized projection
Slightly stale data, unlimited reads, no vendor contact required. Tracking is a read heavy workload where five seconds of staleness is invisible to the user and freshness can be shown in the interface.
What was actually implemented
- 01Debezium reading the WMS binlog into a Postgres projection shaped for the tracking query
- 02Tracking API rebuilt on Next.js route handlers with a single query per request
- 03Redis for hot shipment lookups during the delivery window
- 04Projection lag surfaced in the operations console so staff could see data freshness
- 05Load tested at three times measured peak before cut over
Measured, not estimated
p95 fell from 6.8 seconds to 710ms. Peak capacity stopped being the operating constraint, and support ticket volume on the question where is my shipment dropped 54% in the first quarter.
Measured over 90 days post launch against the prior 90 days. Source: Datadog APM, April 2025
They focused on bottlenecks before writing a single feature.
More from platforms and integration
Platforms and integration
Custom platforms, ERP and CRM integration, and legacy modernization that does not require stopping the business while it happens.