Screening What A Field Of Thousands Says About Your Product

If you sell through tens of thousands of independent people, you are answerable for what all of them write. Not what you wrote. What they wrote, on their own social accounts, about your product, in their own words, at two in the morning. A small compliance team reads as much of it as it can, against the company's own policy manual and a stack of federal and industry guidance that runs to around nineteen separate documents. The arithmetic does not work and everybody knows it.

The obvious build is a model that reads a statement and says yes or no. That build takes a week and it is worthless, because the first time it says yes to something a regulator would not have, nobody trusts it again and it gets switched off.

So the service has two strategies behind one endpoint. The default compiles the rules into a version controlled artifact at build time and evaluates the statement against that, which means the rules are a reviewable document with a history rather than an instruction buried in a prompt. The second embeds the statement, searches the ingested regulation corpus for the passages that actually bear on it, and evaluates against what it retrieves. The first is predictable and auditable. The second reaches material the rules file does not cover.

The parts that took the time are the parts around it. Every evaluation writes an audit trail, so a verdict can be explained later rather than defended in the abstract. A reviewer feedback loop lets a human disagree with a verdict in a way the system records. There is a curated challenge bank, 1,513 lines of it, holding statements with their expected verdicts, which is the only honest way to answer the question a regulated buyer asks first: how do you know it is right. And there is a tuning interface, so when the compliance officers want to change a rule they change it and re-score, rather than opening a ticket and waiting for us.

That last one is the whole point of the engagement, and it is why the rules corpus went through two documented rounds of review with the compliance team and then a full rewrite. The officers are the authors of record. The model is the executor. We built the machinery and then got out of the way of the people who actually know which claims are permitted.

It runs on their infrastructure, under their own legal organisation's account, with their keys.

We will say plainly what is not here: there is no accuracy rate, no throughput figure and no review time reduction on file anywhere, so we are not quoting one. There is also a scar we would rather disclose than have found. A hard coded model identifier was retired by the vendor and took the service down entirely. It was hotfixed the same day and then structurally prevented by moving the identifier to one place. A version string copied into six files is a production incident waiting on somebody else's release schedule.