Connected Is Not Retrievable

There's a sentence that gets said in a lot of AI projects, usually with relief: "We connected it." We loaded the handbook. We pointed it at the knowledge base. We ingested the policy documents. Everyone exhales, because connecting the data felt like the hard part, and now it's done.

Here's the problem. Connecting a document and being able to answer from that document are two completely different things, and the gap between them is where a surprising number of AI systems quietly fail. The data is in there. The system can't reach it. And the worst part is that nothing looks broken.

How "in the system" and "reachable" come apart

If you don't build these systems, the mental model is simple: you give the AI your documents, and now it knows them. The reality has more moving parts, and each one is a place things silently break.

When you add a document to an AI system that answers from your data, it goes through a pipeline. The document gets split into chunks. Each chunk gets converted into a numerical representation — an embedding — that captures its meaning. Those embeddings get stored in a way that lets the system, at question time, go find the chunks most related to what someone asked and feed them to the model. That retrieval step is the whole game. The model can only answer well from what gets retrieved and handed to it. If the right chunk doesn't come back, the model never sees it, and it answers from whatever else it has — or from thin air.

So there are several ways a document can be fully "connected" and still be dark to the system:

It got loaded but never registered with the part of the system that actually searches — it's sitting in storage the query path doesn't look at. It got chunked in a way that split the answer across two pieces so no single chunk looks relevant. Or — the quiet killer — the embeddings were made with one model and the questions get embedded with a different one, so the two don't line up, and retrieval recall silently collapses toward zero. The document is right there. The search reaches past it every single time.

In every one of these cases, you did connect the document. It's genuinely in the system. It's just not retrievable, and those are not the same status.

The tell: it answers from the wrong thing

Here's why this is so dangerous, and why "we connected it" is such a comforting lie. When retrieval fails, the system almost never goes silent. It answers anyway — from a more general source. Its base training. A broader, blander document that did come back. The old version of the policy that's still floating around in there.

So you ask about your specific refund policy and you get a reasonable-sounding, generic answer about refund policies. You ask about your company's specific procedure and you get a plausible industry-standard version of that procedure. It reads fine. It's fluent, it's confident, and it's wrong in exactly the way that's hardest to catch — because it's not obviously wrong, it's just not yours. It answered the question. It just didn't answer it from the document you spent all that effort connecting.

Nobody notices, because nobody's checking whether the answer came from the right place. They're checking whether there's an answer, and there always is. The document you added might as well not exist, and you'll believe it's working for months.

The discipline: measure retrieval before you add more

The fix isn't glamorous, and that's probably why it gets skipped. You have to measure retrieval directly, with real questions, before you trust it and before you pile on more data.

The tool for this is a golden set: a list of real questions your users will actually ask, each paired with the specific document — ideally the specific passage — that should answer it. Twenty questions is enough to start. Fifty is better. You write them from how real people actually phrase things, not from how the documents are titled, because those are different and the difference is exactly what breaks retrieval.

Then you check the boring, decisive thing: for each question, did the system retrieve the right source? Not "was the answer good" — that's downstream and it lies to you. Did the correct chunk come back in the results at all? If it didn't, the answer was luck or fabrication regardless of how good it sounded. This measurement catches every failure mode above, because all of them show up the same way: the right document doesn't come back for the questions it should answer.

You run this before you connect the next data source, not after you've connected ten and something feels off. Because here's what happens otherwise: retrieval quality degrades as you add data — more documents means more chances for the wrong thing to look relevant — and if you were never measuring it, you have no idea when it broke or which addition broke it. You just know that six months in, the answers got mushy, and now you're debugging a haystack you built yourself.

"We connected it" is not a status

The thing I want owners to take away from this is small and it will save you real money: "we connected it" is not a status. It's the start of the work, not the end of it. When a vendor or an internal team tells you the data's connected, the right follow-up isn't "great" — it's "show me it answering ten real questions from the right sources."

If they can't show you that, they don't actually know whether it works. They know the ingestion job ran. That's a different, much smaller fact than the one they're implying, and the gap between the two is where AI projects go to quietly disappoint everyone.

Connecting data is easy — it's mostly a button now. Making that data reliably retrievable, and knowing that it is, is the actual engineering. Anyone can load your documents. The question worth asking is whether the system can find them again when it matters.

The golden set is cheap insurance

I know "build a golden set and measure retrieval" sounds like the kind of engineering discipline that only big teams can afford. It isn't. Twenty real questions in a document, each with a note about which source should answer it, is an afternoon of work — and most of that afternoon is just writing down questions your customers already ask you every week. You don't need a tool or a specialist to do the first version. You need someone who knows the business to sit down and be honest about what people actually ask and where the true answer lives.

That small, unglamorous artifact is the cheapest insurance you'll buy on an AI project. It turns "we connected it, seems to be working" — a feeling — into "it retrieved the right source on eighteen of twenty questions, and here are the two it missed" — a fact you can act on. When a vendor tells you the data's connected, you hand them the twenty questions and ask them to run it in front of you. If the system answers from the right places, you've verified something real. If it doesn't, you found out before you bet the customer relationship on it, not after. Either way, the afternoon paid for itself.

If you've stood up an AI tool on your own data and you're not certain it's answering from the right sources, that's exactly the thing worth checking before you trust it. See how we work.