Payment Error Threshold Intelligence
In frontier markets we were always reactive. An operator would go down, the merchant would tell us their transactions were failing, and only then would we find out. I shipped a pipeline that cut failure detection from about 90 minutes to under a minute, then took it further into a model that tells real degradation apart from normal noise and points at whose problem it is.
In frontier markets, being reactive is expensive. We had a NOC team, but we were never predictive about operator performance. An operator would have unplanned downtime, the merchant would message us that their transactions were failing, and only then would we discover the operator was down. Same story when a merchant's own failure rate spiked from a bug on their side. They'd ask us, we'd investigate, then we'd tell them what was wrong. There was a subtler case too. Some error responses, like OTP retries, merchant limit errors, or "user did not approve the transaction," are normal in small amounts. When they spike, they often mean something is actually broken at the operator: OTPs not reaching users, payment requests not popping up. We wanted to catch all three early, alert the merchant the moment something looked wrong on their side or an operator looked down so they could cut traffic, and escalate to the operator at the same time.
In production, I shipped a Python and DataDog pipeline that cut failure detection from about 90 minutes to under a minute. For each merchant I analyzed their specific failure-reason data and set two benchmark rates, a pre-alert level and a full-alert level. When a rate crossed one, DataDog emailed the relevant people based on the rules. For this portfolio version I took the model deeper, because a naive rule like "alert when error rate is over 5%" still floods ops. Some errors are normal, some are merchant config, some are a real operator outage, and one threshold can't tell them apart. I also added an AI copilot that drafts plain-English summaries of each alert and merchant-safe comms, kept well away from the detection itself.
A single threshold pages too much.
For each merchant, operator, and error reason, I baseline the average error rate, how much it normally varies, and how it tracks with success. A pre-alert needs the error rate to rise above its normal range and success to dip below a warning level at the same time. A full alert needs both to breach harder. Requiring two conditions at once is what stops a harmless error spike from paging anyone.
Whose problem is it?
If the trouble is isolated to one merchant and lines up with config or credit signals, it's merchant-side. If errors climb across many merchants while success drops broadly, it's the operator. That call is what turns an alert into something someone can act on, instead of a number on a dashboard.
From 4,002 candidates to 175 real incidents.
On a 30-day holdout across 100 merchants and 2 operators, the success-rate guardrail cut 4,002 single-condition candidates down to 1,269 dual-condition alerts, a 68% reduction. Of those, 175 became full-alert incidents, ranked into a triage queue with the likely owner, how widely it had spread, and an impact score.
From raw error spikes to ranked incidents · 30-day backtest, 100 merchants
−68% noiseThe trick is the dual condition: a page only fires when errors spike and success dips together. That cuts the noise 68% before anything ever reaches a person.