The Same Rule Flipped Four Times In Three Weeks

The product answers questions for people who are worried about something, which means the quality of an answer is a judgement call, which means somebody has to make it. A panel of three graded batches of answers, and their notes went straight into the next night's changes.

It did not work, and the way it failed is worth describing precisely, because most teams building this kind of product will recognise it. The same rule flipped four times in three weeks. Reviewers were recorded, in the same batch, on the same answer, one marking it down for mentioning a source and another marking it up. Feedback arrived through three separate channels, none of which kept a record of what had already been handled, so the same report got worked twice and others got dropped entirely.

Then grading sentiment fell off a cliff overnight and the assumption in the room was that the engine had regressed. We went and measured the answers on either side of the cliff instead of arguing about it. They were functionally interchangeable: the same count of recommendations surfaced, the same number of answers leading with a bolded statistic, average lengths of 536, 526 and 510 characters across the three batches. The engine had not changed. The rubric had. People were grading against a standard that had moved without being written down anywhere.

That is the finding that reframed the engagement. The problem was not a model and it was not a prompt. It was that opinions were being applied to production without ever becoming decisions.

So what we built is mostly not code. A binding protocol says you may not change a prompt, a note template or a judge rubric directly from a grading note or a chat message. You synthesize the round, you ratify it, and then you implement it citing the ruling that authorised it. When two pieces of feedback contradict each other you escalate the contradiction rather than patching whichever arrived last. There is a fast lane, deliberately narrow, for safety violations, contradictions with an already ratified rule, and outright bugs.

Underneath it sits a numbered decision ledger, declared the sole authority, where every entry is marked ratified, open or under review, and where implementing either side of an open question is forbidden. That last clause does more work than anything else in the document. It means an open disagreement stays visibly open instead of being silently resolved by whoever touched the code most recently.

The supporting mechanics are small and matter. An engine version is stamped only after that batch's grades come back, so a version always refers to the thing that was actually graded. Behaviour changes sit behind flags. Three triage processes, one per feedback channel, run over a single shared ledger, each scanning from a cursor, deduplicating, routing, and drafting any customer facing reply for approval before it sends.

We will not claim a quality improvement, because no before and after measurement of answer quality exists and inventing one here would be beneath the exercise. What we can show is the search quality benchmark this work runs against: 61 of 61 cases passing across twenty positive, twenty negative and twenty one security and edge cases, on one run at one date, which is how it should be quoted. Rerun it, do not inherit it.