Insights
Notes on production AI - context engineering, agentic orchestration, and what it takes to ship bots that do not hallucinate their way into trouble. Written by someone who has done it at scale.
July 13, 2026
Embeddings, geocoding, web re-indexing - all of it is expensive and predictable. Do it in the request and you pay for it in latency on every query. Push it offline and leave only cheap indexed lookups on the request path, and the answer stays fast even on a huge database. The principle: what changes rarely doesn't get computed at runtime.
Read articleJuly 13, 2026
Every tool call is a round-trip - produce a result, return it, let the model continue. When the things that would normally call a tool (follow-up suggestions, entity links, prompts) are emitted directly in the streamed answer as inline tags, the answer becomes one pass. Time-to-first-token drops, because you're not computing anything up front.
Read articleJuly 13, 2026
The fastest token is the one already prepared. Model providers cache the prompt prefix - but only the part that doesn't change. Every dynamic blob at the front cancels the whole gift. A static system prompt and per-turn injection behind the cached prefix is the cheapest latency win you'll find.
Read articleJuly 13, 2026
Three seconds to answer is a budget, not an average. And the budget bleeds through a lot of small leaks - a fresh TLS handshake on every embed, an agent-loop restart after a failed tool, an extra model-call after the user already got what they needed. None of them moves the number alone, but together they decide whether you make the limit.
Read articleJune 25, 2026
Most teams treat evaluation as a scoreboard - a number you glance at and feel good or bad about. The frontier idea is to wire the eval back into the system as an input that rewrites it: traces go to independent judges, fixes get proposed automatically, and one human approves. It's AI all the way down, with a single gate that isn't.
Read articleJune 24, 2026
Before a bot can answer well, it has to know what kind of question it's facing. Skip that step and chitchat gets a database query, 'last month' goes to SQL untranslated, and every request runs the same dumb path. The fix is a cheap, decisive first move: classify the intent, extract the entities, resolve the time.
Read articleJune 23, 2026
Your knowledge base is enormous and the model's context window is tiny. The trick isn't a bigger window - it's an hourglass: compress everything ahead of time, then expand only what one question actually needs.
Read articleJune 22, 2026
You compressed your knowledge into clean indexes. Now comes the moment everything hinges on: given a question, can the bot actually find the handful of tables it needs among a thousand? Miss the right one and nothing downstream can save you - so at the retrieval step, recall beats precision.
Read articleJune 21, 2026
Hand-writing the association layer is the right move for your first dozen tables. It does not scale to a thousand - or to a new client schema every month. The answer is to let the model draft the glossary from real data, then have a human curate it.
Read articleJune 20, 2026
A user asks for 'employees.' Your database calls them H_OSOBA. No model bridges that gap on its own. The fix is the highest-leverage, least glamorous artifact in the whole system: a living glossary that maps human language to your schema.
Read articleJune 19, 2026
An association layer teaches a bot your vocabulary - 'employee' means this table. But some questions aren't about words; they're about relationships and business concepts no single table spells out. That's where a flat dictionary ends and a knowledge graph begins.
Read articleJune 18, 2026
Hand an agent total freedom and it wanders off and wrecks the task. Lock it in rigid if-else and it can't handle anything real. The production sweet spot is a fixed pipeline with autonomous pockets - rails the agent can't leave, and real decisions inside them.
Read articleJune 17, 2026
A bot that doesn't learn from use is frozen at launch quality - and launch quality is the worst it should ever be. The difference between a bot that goes stale and one that sharpens every week is a feedback loop that turns each interaction into signal.
Read articleJune 16, 2026
A query that runs cleanly can still be wrong. The fix isn't a smarter model - it's a loop: generate, probe the data, let a judge decide if the answer makes sense, and feed concrete failures back until it does.
Read articleJune 14, 2026
The judge loop is about being right. This is about the prior question: what does a bot do when it isn't sure? The worst ones barrel ahead and answer anyway. A good one has a repertoire - clarify, default, hedge, or hand off - and knows which to reach for.
Read articleJune 13, 2026
Most of what an agent does is trivial. Resolving a code, matching a value, extracting an entity - none of it needs your most expensive model. Defer the lookups, route the grunt work to cheap fast models, and save the genius for the genuinely hard part.
Read articleJune 11, 2026
The best customers for a data bot - hospitals, banks - are the ones who legally can't let data leave the building. That constraint shapes everything: where the model runs, what the query layer is allowed to do, and how you prove the whole thing behaves.
Read articleJune 6, 2026
The most common question we get on a scoping call: should the bot be fine-tuned on our data, or use retrieval? Here’s how we actually decide.
Read articleJune 5, 2026
A confident wrong answer is worse than no answer. Hallucination isn’t magic - it’s a set of failure modes you can design against.
Read article