Is Your AI Hallucinating?

We ship an API that returns a hallucination score for every token that a frontier language model generates. Before describing how it works, here is the most important thing about it: the percentage score that we return is not actually the probability that the token is wrong.

The obvious problem is that frontier models do not expose the hidden states that we would need to probe to reveal their thoughts. Hmmm...

You cannot probe GPT or Claude. Their APIs return only text. There is no layer to read, no residual stream to inspect, and no amount of prompting changes that. Every technique in this space has to route around the same wall.

Our answer is a white-box proxy model. We run an open-weight model we can control end to end, which means we have full access to its activations at every layer. The probe is trained on and reads from this proxy model.

To be clear then, our API actually returns information about a proxy model rather than the model you have called.

We took a dataset of hundreds of thousands of input/output pairs from GPT-5.4 Nano, containing a mixture of both reliable and hallucinated claims. Averaged across the dataset, 3% of Nano's claims were hallucinated. We then replayed Nano's outputs through Gemma-4 and trained our probes against those labels on Gemma-4's hidden states.

We scored the probe on an unseen validation set, counting a case as accurate when the probe's verdict matched that of the judge LLM. Agreement exceeded 95%.

So what does our API actually return?

What our scores measure is the likelihood that the proxy model is confused while producing the output generated by your called model. A high score means that, as the proxy model processed that token, its internal state closely resembled the internal states it typically exhibits when it is confused.

The upshot is that our API works with black box frontier models. Point your base URL at us and swap the API key. We handle the model call from there. You get back the usual response from your model of choice along with two extra fields.See an example below:

{
  "choices": [{ "message": { "content": "Cardinal Robert Prevost was elected pope after a conclave attended by 135 cardinals." } }],
  "tokens": ["Cardinal", " Robert", " Prevost", " was", " elected", " pope", " after", " a", " conclave", " attended", " by", " 135", " cardinals", "."],
  "scores": { "hallucination": [0.01, 0.02, 0.03, 0.02, 0.01, 0.04, 0.02, 0.01, 0.03, 0.05, 0.08, 0.72, 0.31, 0.01] }
}

Scores stream and each span carries its own number, so we can highlight a suspect span while the model is still writing.

Here, the model correctly identifies Cardinal Robert Prevost as the newly elected pope, but is confused about how many cardinals attended the conclave (2 of the 135 total couldn’t make it). For more examples, have a play with the demo that’s up on our website.

Why white-box proxy?

Earlier in this piece the use of a more 'intelligent' frontier LLM as a judge was described. Using an LLM as a judge actually has a slightly higher accuracy rate than this white box model interpretability approach. So why not just use LLM as a judge in all cases?

It boils down to cost, measured both in money and time.

The first problem is that models are categorically bad at marking their own homework for the same reason the mistake was made in the first place. So using an LLM as a judge is ineffective if it's just a second pass in the same model. It has to be a second model, of at least equal intelligence, running over everything the first one wrote. So now you’re paying for two frontier models. Not ideal.

By way of contrast, our approach runs a small open-weights proxy instead, and we use a number of inference tricks to keep costs low. The result is a fraction of the compute spend of a second frontier model. We pass the model costs through at zero margin, so the only thing you pay us for is the probe itself.

Then there is the time cost of the LLM as a judge. A judge has to read a finished response before it can rule on it, which means it has to be run sequentially. You have to wait for both models to finish. Our probe scores each token as it is produced, so a suspect claim is flagged while the sentence is still being written.

Our approach amounts to a sensible trade. You give up a few points of accuracy in exchange for a fraction of the cost and real time operation.

Yeah, but Anthropic’s next model will fix this, right?

Well we can’t rule it out, but probably not, no. Newer models hallucinate less but a system whose core operation is "generate plausible text" will also generate more convincing falsehoods.

The other thing is that a hallucination isn't truly a failure of training. It is actually a result of how the loss / reward function is designed.

Pretraining scores you on predicting the next token. A fabricated citation scores well, because it's what a document like that would plausibly say next. Nothing in the objective function actually knows the difference between true and likely. Post-training doesn’t help matters. Reward models learn from human preferences and humans typically like confident answers and don’t often check their sources.

As an aside, OpenAI have still never replaced their superalignment team. Model interpretability just isn’t a focus for the big labs - for valid commercial reasons. AI capabilities are what the investors are paying for after all.

What we claim

No one can promise to be able to solve hallucination.

Our value proposal is that you can get a per-token signal from a normal API call, and use it to efficiently decide which claims are worth a verification pass. That is especially useful if your alternatives are reading everything, or reading nothing.

What we want from you, dear reader

We have essentially arbitrarily chosen what threshold we assign something to be a hallucination. It’s a function of your tolerance for false positives and we suspect it varies by use case. We would love to hear back from you as to how it looks for your specific industry!

Docs: https://www.telluvian.ai/docs Demo: https://www.telluvian.ai/demo

Pricing, in full

Scoring costs $1.00 per 1M completion tokens. That is the only price we set.

Model tokens are passed through at cost. You pay what the provider charges us, with no markup and no commission. We are not making margin on inference, and that’s not going to change.

Output tokens only. The probe works as the model writes, so a 100k-token prompt with a short answer pays the surcharge on the short answer. To give you some example inference pricing see the table below (these change over time):

Model	Input	Output	Probe	Total
google/gemma-4-31B-it	2.00¢	4.00¢	0.50¢	6.50¢
anthropic/claude-sonnet-4-5	3.00¢	7.50¢	0.50¢	11.00¢
deepseek/deepseek-r1	0.70¢	1.25¢	0.50¢	2.45¢
deepseek/deepseek-chat	0.26¢	0.51¢	0.50¢	1.27¢

That is 10k prompt tokens and 5k completion tokens. Send include_scores: false and the probe fee is not charged.

Billing is prepaid in dollars. No subscription. Balances are integer micro-dollars and every charge is computed in integers, and only rounded up once at the end. When your balance hits zero you get a 402 and nothing is charged automatically. Unused credit is refundable.