Why an answer about your pet should be refused rather than guessed

There is a kind of software failure that does not look like one. Nothing crashes, no error appears, and a well formed sentence arrives on time. It is simply not true.

Language models produce that failure by construction, and a product that puts one near a medical record has to decide what to do about it. This is what Talli decided, and why.

A prompt is a request, not a guarantee

The obvious approach is to tell the model to behave: answer only from the records provided, refuse when you are not sure, never invent a value. Those instructions are all in the system prompt, and they are under test.

They are also not the reason the product is safe, because a prompt is a request and a model may ignore it. That is not a suspicion. It was measured.

Deliberately degraded photographs of veterinary laboratory reports, with known correct values, were given to a vision model that had been told to write a placeholder rather than guess when a character was not legible. On the worst image it returned 32 fully populated analyte rows, 30 of them wrong, and used the placeholder zero times. The refusal instruction was ignored at every level of degradation.

Two details from that run matter more than the headline.

The first is that none of the wrong values were malformed. Every one was a well formed, plausible medical number, of the kind that passes a schema check, sits inside a chart, and looks entirely normal on a review screen.

The second is what happened when the same page was re-read asking only for a transcription. Of the 19 reference intervals it printed, 16 were invented, and the invented ones were the standard human reference ranges rather than the ones on the page. The model stopped reading and started recalling.

A reference interval is what decides whether a result is flagged high or low. A correct value read against an invented interval produces a wrong reading of a real result, and there is no downstream check that catches it, because there is nothing structurally wrong with it.

So the checks are mechanical

Everything below is enforced in code, where a value would be written or an answer shown. None of it depends on a model choosing to comply.

An answer with no citation is discarded

When you ask a question about a pet, the app retrieves that pet's own rows, numbers them, and asks one question about that numbered list. Every sentence that draws on a record has to end with that record's number.

An answer that cites nothing is discarded and never reaches you, however plausible it reads. A citation number that is not on the list is deleted from the text before the answer is rendered, because an invented citation is exactly what this check exists to catch.

The check works for a specific reason: a model answering from general knowledge has nothing to cite. The failure that is hardest to spot by reading becomes the easiest to catch by machine.

"The record does not say" is a real answer

The second half of that rule is the part people underestimate. If refusal is only a failure state, a system under pressure to be helpful will avoid it.

So when the records do not answer the question, the model is required to say so, name what is missing, and that is returned as a normal answer rather than as an error. An empty record is not a reason to improvise. It is a fact about your record, and it is often the most useful thing the app can tell you.

The same habit runs through the rest of the product. A health area with insufficient inputs says so and names the requirement it did not meet, rather than producing a reassuring score from nothing.

Some questions are refused before a model is called

Ask what is wrong with your dog, ask for a diagnosis, ask what dose to give, and the request is refused before any provider is contacted. There is no model call, so there is no answer to filter.

The refusal says what it is: it cannot diagnose, prescribe or tell you what is wrong, and it can show you what is in the record and help you take it to a veterinarian.

Beyond that hard stop, the assistant does not interpret findings, does not assess severity, risk or urgency, and does not decide whether a value is normal or abnormal. Only a flag printed in the record can say that, and it is reported as the laboratory's flag rather than as the software's opinion. Where a veterinarian or a laboratory wrote an assessment, it can be quoted and attributed, because the words are theirs.

A fact that does not quote the document is dropped, not downgraded

When a document is read, each fact has to quote the page it came from, and the value has to appear inside its own quote. The comparison collapses whitespace and does nothing else. No case folding, no fuzzy matching, because both would let a paraphrase through, and a paraphrase is precisely what the check is for.

A fact that fails is discarded rather than kept with a lower score. There is no confidence at which an unquotable fact becomes filable, so lowering its score would only be a quieter way of keeping it.

There is a related rule about what a document is actually saying. A due date is not a record of something that happened. A billing line is not evidence that a treatment was given. Something declined was never given. Only facts in the right form are allowed to create a clinical row, which is why a reminder table in a visit summary does not turn into a list of tests your animal had.

A reading the animal could not have produced is held

A weight or a temperature is checked against the animal's own physiology and against the nearest reading on its own chart before it is written. If it fails, the row is not written, the chart is not touched, and the reading is reported with the bound it failed so that a person can settle it.

It is not dropped silently, not written at a lower confidence, and not corrected for you. The document may be right and the chart wrong. Deciding that is a person's job.

The limit of that check is stated rather than glossed over: weight and temperature have bounds today, and other measurement kinds do not yet.

Nothing read by a machine eye is applied on its own

Anything that came from optical character recognition goes to review unconditionally, at any confidence score. That is a rule about where the text came from, not a threshold, and it is deliberately not allowed to become one.

The measurement behind it: on degraded veterinary pages the confidence of correct reads and the confidence of incorrect reads formed two heavily overlapping distributions, so no cut-off exists that keeps the good numbers and drops the bad ones.

And character recognition does not fail the way a parser fails. It does not return nonsense that a schema rejects. It substitutes one plausible character for another and hands back a well formed wrong number. A body weight of 31 kg read as 11 kg validates, matches the expected type, carries a respectable score, and is a 20 kg lie in a medical record. That is the kind of number a veterinarian doses against.

What this costs you

Honesty about the trade is part of the argument. All of this means you are asked more often.

A document that could have been filed silently instead waits for one tap. With no model provider configured at all, which is a supported way to run the whole product, most documents need that tap, and the documentation says so in as many words: that is the policy working rather than a defect.

The asymmetry that justifies it is written into the source as a comment, and it is the shortest summary of the whole product. A value declined costs the user one tap. A wrong value applied silently becomes part of a medical record that a veterinarian may act on.

The number that is not on this site

There is no accuracy percentage for the AI path anywhere on this site, and there will not be one. The only committed measurement run of the document corpus was made with no provider configured, so any percentage would be an invention, and an invented accuracy figure is the same class of error as an invented laboratory value.

What can be said is what the checks are, where they run, and what happens when they fail. That is written the same way this article is: as behaviour, not as a promise.