Customer Care Knew Something Was Wrong. Nobody Knew How Much.
Confidentiality Note: Client anonymized. Payment, billing and store vendors are described by role. No revenue figure, fee amount or customer record appears here; the dollar sizing of the leak is the client's and is not published.
The Full Story
It started as a customer care report, which is usually how the expensive ones start. Members were complaining that a monthly charge had failed. Nobody could say whether that was five people or five thousand, because the answer lived across three systems that had never been asked the question together.
The answer was 1,232 failed charges in a single month, across 1,229 unique customers. A 27.7 percent failure rate on a recurring charge. The one piece of good news, and we checked it first, was that nobody had been wrongly charged: every failure happened before money was taken.
The causes were layered. A platform migration run before we were involved had carried payment methods across that did not work, and roughly a quarter of the migrated cards were expired, reissued or otherwise invalid. Separately, 2,839 accounts held cards that simply disagreed between the old payment gateway and the new subscription biller, with no integration path between the two and no gateway code in the backend at all. Then there were the zombies: subscriptions active in the billing vendor and terminated on the store side, retrying forever, quietly inflating the subscriber count everyone was reporting on.
Here is the part we would put first, because it is the part that cost us something to publish. Our own first estimate of the card mismatch leak was about twenty times too high. We had a number, it was wrong, and rather than let it stand we validated it down before anyone acted on it. An investigation that never revises its own headline figure is not an investigation, it is a pitch. The revised number is the one the client planned against.
Remediation came in three parts. The zombie cleanup ran through the billing vendor's API with cohort filters: 519 customers identified, 517 still active in the biller, 495 cancelled, 27 deliberately left alone because they had a recent successful charge, one error. The skip lockdown scanned 24,762 active subscriptions, found 6,180 fee subscriptions of which every single one was skippable, and found that none of them could be changed through the API, because the field is computed from a store wide setting. We reported that as blocked rather than inventing a workaround for it.
And one of our own automated sessions later cancelled 25 subscriptions that the cleanup had deliberately skipped. We reactivated all 25 and verified them green across all three systems: 25 active, 25 with a next charge date set, 25 with the cancellation reason cleared, zero errors, and the legacy platform proven never to have been touched. That is in the record because it happened, and because how a supplier behaves in the twenty minutes after breaking something is the only part of this a client cannot evaluate in advance.
What went to customer care was not a dashboard. It was five lists, each row carrying the action needed, and a briefing in plain English saying which problems engineering was fixing and which ones needed somebody to call a customer.
This ran as a joint team. Our part was the investigation, the tooling and the decision package; the client's own engineers and their operations lead carried their share of it and made the business calls. We are not claiming it alone.
The Challenge
Customer care was fielding complaints about failed monthly charges and had no idea of the scale. Underneath it, a platform migration run before we arrived had carried across payment methods that did not work, and thousands of accounts held cards that disagreed between the old payment gateway and the new subscription biller, with no integration path between the two. Hundreds of subscriptions were zombies: active in the billing vendor, terminated on the store side, retrying forever and inflating the subscriber count. And a fee that was supposed to be mandatory turned out to be skippable from the self service portal.
Our Solution
Investigation tooling and remediation commands run against the client's own billing, store and legacy platform data, alongside their team. A cohort analysis over every failed charge. A cascade payment recovery path. A zombie subscription cleanup executed through the billing vendor's API with cohort filters and safeguards. Five action lists for customer care, each carrying an explicit action needed column, and a plain English briefing that separates what engineering is fixing from what needs a human to pick up the phone.
The Result
The Leak Found, Sized, And Then Sized Down