Degraded API Performance

Incident Postmortem — Broadcast API Degradation

Date: 7/17/2026

Severity: Major

Customer impact: ~2.5 hours of degraded performance with intermittent unavailability, affecting all customers

Full incident window: 6:34 AM (first alerts) – 10:54 AM EDT (resolved)

Status: Resolved

Summary

A newly deployed integration began exercising a previously seldom-used code path, driving substantial volume through an inefficient subscriber lookup query that ran roughly 30x slower than necessary. The slow query held database resources long enough that unrelated processes queued behind it, cascading into connection buildup, memory and CPU exhaustion, and ultimately timeouts and errors across Broadcast. A hotfix to the query restored normal resource consumption and system operation. All customers were affected for approximately 2.5 hours.

Timeline (EDT)

• 6:34 AM — First alerts fire. Transient, with no observable customer impact.
• ~8:00 AM — Degradation becomes noticeable; investigation is escalated.
• 9:00 AM — Incident formally opened as degradation worsens.
• 10:16 AM — Hotfix applied to the subscriber lookup query.
• 10:54 AM — Incident declared resolved after 38 minutes of monitoring confirmed resource consumption and error rates had returned to baseline.

Impact

Degraded performance across Broadcast with intermittent unavailability impacting all customers for approximately 2.5 hours.

Root cause

The API used an inefficient query to retrieve subscriber information. Two problems compounded: the query performed unnecessary string manipulation, and it failed to use the relevant table index, forcing a far more expensive lookup. Together these made execution roughly 30x slower than the fixed version.

This inefficiency was latent because the code path in question was rarely used. When a new integration went live and began routing significant traffic through that path, the slow query started executing at volume. It occupied database resources long enough that other processes had to wait to execute against the database. As more processes queued, the problem cascaded: database connections accumulated, memory and CPU were consumed, and the resulting resource pressure produced timeouts and errors throughout the system.

Resolution

A hotfix corrected the query to eliminate the unnecessary string manipulation and use the table index. Resource consumption dropped to normal levels, and the team monitored for 38 minutes before declaring the incident resolved.