AI Portal

Mech Interp as a Portal to AI

The AI spend problem is real and under-addressed despite being solvable with techniques we already have.

Motivation

AI use has skyrocketed and token prices have collapsed. Per-token inference prices for state-of-the-art capability have been falling by orders of magnitude each year and Gartner projects a further ~90% drop in provider-side costs by 2030. There is reason to believe that the cost problem is solving itself.

However, it just isn't. Global enterprise AI spending is projected to hit ~$407B in 2026, up roughly 35% year-on-year, with enterprise LLM API spend passing $8.4B in 2025 and on track to double. The mechanism seems to follow Jevons paradox: usage is growing faster than unit prices are falling. It's easy to see why, agentic workflows burn 5-30x more tokens per task than a chatbot query, RAG inflates context 3-5x per call, and systems like always-on monitoring agents consume compute whether or not a human is asking anything. This usage increase has made itself known on enterprise budget lines:

Uber's engineering wing reportedly blew its entire annual AI coding budget a third of the way into the year. Its CEO, Dara Khosrowshahi, said the cost of the company's AI initiatives was "getting harder to justify" relative to output.
Microsoft reportedly cancelled most of its internal Claude Code licenses, partly over cost, six months after rolling them out.
Axios reported one company spent half a billion dollars in a single month after failing to set usage limits.
Sam Altman, on stage at Intelligence at Work: "My company spent my entire 2026 budget in Q1". This has since become a meme.
Jed Dougherty (SVP at Dataiku) warns: "We're all burning compute" describing an industry where most organisations are still in the "I hope this pans out" stage.
Meta and Amazon both discontinued internal token-consumption leaderboards after employees started "Tokenmaxxing".

The shocking truth is that roughly 95% of enterprise AI usage still runs on the most expensive frontier models. This includes work that clearly doesn't require frontier capability. This decadence is not a hardware problem nor a pricing problem. It is an optimization problem, specifically the kind you can solve with software.

What can be done?

The standard brace of efficiency ideas is well known. It's worth going through why each one, in practice, underdelivers.

Quantisation and distillation.

Effective if you self-host but these methods are incompatible with using frontier models over an API, where you can't touch the weights. Distilling your own model also means settling for a certain level and accepting that the frontier moves away from you.

Self-hosting an open model.

Self hosting can work out 50-85% cheaper for sustained high-volume workloads when combined with routing. Unfortunately, it is only large enterprises that can make use of this economy of scale.

Prompt caching.

This method is easy and proven to work: a 2026 PwC evaluation found real-world savings of 41-80% across major providers. And yet most enterprises still aren't using it systematically, because it requires restructuring prompts in a unified manner across the business. The lack of commonplace prompt caching is probably the most glaring example of a missed efficiency due to coordination failure rather than a technical failure.

Speculative decoding, batching, custom kernels.

These are examples of provider-side optimisations. They lower the provider's cost curve; Gartner is explicit that these savings will not be fully passed on to enterprise customers. That means that we won't benefit from efficiency gains that happen on someone else's GPU.

Just write shorter prompts.

Keep it simple, stupid. Tokens feel cheap, and no employee that I've ever met treats them as a scarce resource. Output tokens are priced at ~4x input tokens so every design decision that lengthens generation compounds quickly.

Rationing

This one is common practice, it means things like usage caps, license clawbacks, telling employees to 'use the cheap model' etc. The thing is rationing reduces spend by reducing value delivered. This is clearly an unhappy compromise.

What we can do

We think the interesting frontier is per-request optimisation: treating every single inference call as a decision problem over model, context, and compute. Mechanistic interpretability is typically looking to answer "what is the model thinking?" from an AI safety perspective but it turns out the same question, answered per-token in production, can provide serious efficiency gains.

Prompt pre-processing

Most production prompts are mostly redundant: boilerplate system prompts, stale conversation history, RAG chunks etc. By implementing pre-processing steps, such as compression, we can limit context to the tokens that actually shift the output distribution. Whilst this is definitely treating a symptom rather than the illness, this is a quick fix to problems like context rot and model costs blowing up with session length.

Model routing

Not every query needs a frontier model, but 95% of them get one anyway. Users typically have the option to use a simpler model, they just don't use it. Whether this is out of a desire to get the best answer or plain laziness doesn't really matter to us, the important thing is they overwhelmingly will choose the best model available regardless of the prompt. A clear efficiency gain would be to provide some sort of router layer that can classify each request by difficulty and route it accordingly. By way of example, a scheme may direct to Haiku-class for the easiest 50%, Sonnet for the next 40% and Opus-class for the hardest 10% (forget Fable you don't need it). Typical enterprise distributions along these lines cut per-query costs 60-80% against a frontier-only baseline. The problem here is setting up the router itself: predicting task difficulty before you've gotten the answer is a nontrivial problem.

Optimal reasoning

As it turns out, transformers are susceptible to overthinking. This recent LessWrong post Fragile Correctness: Cases of reasoning harming performance by Toby Pullan looks at models that reach the correct answer partway through their chain of thought and then overthink their way back out of it. This is, I think, a surprisingly relatable trait for a machine to have. It reminds me of more than a few exam questions I got wrong over the years. Toby's research shows that simple linear probes on hidden states can partially detect this overthinking and stop the reasoning early, which recovers accuracy while using fewer tokens. In other words, interpretability tooling can make the model both cheaper and more accurate at the same time.

Reasoning swap

Reasoning swap involves decoupling the architect from the builder. Have an expensive model produce the plan: the reasoning trace, the decomposition, the spec etc, and then hand execution to a small or specialised model. Since output tokens cost ~4x more than input tokens, moving the generation phase onto cheap models will cut down the bill. I think this one makes intuitive sense; you wouldn't pay an architect to do a builder's work.

What now

Every deployment sits somewhere on a three-dimensional curve between cost, latency, and performance, and most enterprise users haven't yet gone looking to optimise their position on it. The current standard practice of using frontier models for everything is the only reason crude rationing works at all: there's so much slack that even blunt cuts save money. But rationing just trades performance for cost. It can't actually move you toward an optimum.

What would move us towards an optimum is a central portal that every AI request flows through, one that can compress, route, monitor reasoning, and swap models under the hood automatically, without adding significant latency or any complexity for the user. The idea is that your application makes the same API call it made yesterday and the portal just passively makes it cheaper.

That portal is what we're building now. As we've discussed above, each individual problem is tractable, and the tooling to solve them already exists. The idea is to create an infrastructure layer that makes AI cheaper and better for everyone whilst running in the background.

If you'd like to get in contact with us, drop a line to hello@telluvian.ai.