Don't Drown in the Facts

How an AI can get every fact right and still miss the point.

My AI finally had its facts straight and its patterns connected, but it lost sight of what mattered.

My last update was all about the memory's honesty: getting the system to know what was true, and which truths belonged together. I'd built a concept layer to tie those threads across it, and I figured that once the facts were straight and the connections were there, understanding would follow.

That's not what happened, though. The memory was honest, the threads were tied together, and the system still buried the point.

First, make the memory honest

It was detailed more thoroughly in my previous post, but in essence the change came down to two things.

I taught the memory to rank facts by when they became true rather than when I last saved the file, and to retire a fact once a newer one replaced it, so a settled old point couldn’t get confused as present.

It worked, and the memory could finally tell me what was true, what used to be true, and the evolution along the way.

Then it lost the plot

In that same week, with the memory finally honest, the harder problem showed itself. Every fact the system pulled was true, yet the point was buried somewhere underneath it all.

I saw it most clearly when I asked it to help me prepare for a hard conversation. What came back was fluent, well sourced, and completely beside the point. It kept fixating on the particulars and ignoring what I needed. It took ten rounds of pushing before I got a prep that helped, one that diagnosed what I was trying to do in the meeting, the friction I'd run into, and how to get through it.

The more facts I was feeding it, the more thoroughly it seemed to drown out the core persona of my Eji Communication Coaching system: the one that asks questions, digs deeper, and looks for the "Black Swans" that Chris Voss notes as the key to unlocking turning points in negotiation.

Knowing what is true and knowing what matters turned out to be different problems, and I had over-solved for the first at the expense of the second. It became focused on the what, but was getting blinded to the why.

Wisdom is the hard part

My system keeps three stores of information: the rules it follows, the facts it holds, and the sense of what they mean. The rules are the skills I wrote, and the facts are the context I spent last week making honest.

The third kind is the wisdom, and it is the deepest, toughest one. I'd already built the system a place to keep what I've labeled "concepts," the layer where it turns the patterns it notices into something like understanding. But a store of insight only counts when the system reaches for it at the right moment. A pattern it never surfaces is no better than one that was never written in the first place.

Databricks makes the same point from the other side in an article I’ve cited before. They found that more memory doesn't automatically make an agent better, because retrieval gets harder as the store grows. The win comes from selective retrieval: deciding not how much to surface, but which high-signal piece the task truly requires.

So I taught it what matters

The fix landed in my latest version (3.2.0), and it gave the system two new abilities it never had.

The first is to seed the prep phase with background. Retrieval used to run on a single message with no view of the conversation around it, so a short follow-up like "write me a guide" would send only that phrase down to the EJG process. It had no subject to hold, and the system would reach into the dark and pull the wrong thread, confidently, about the wrong person entirely.

Now I've changed my base AI's system prompt so the request it sends names its subject first. If it can't name one, it stops and asks instead of guessing.

That was an important plumbing fix, but the deeper fix came next. The base model is also now instructed to spend its first effort on one question: out of everything on the table, what is this turn about, underneath the literal request?

The package from the EJG process includes a hypothesis to go along with the context, concepts, and skills notes. The model leads with that read, then treats the pile of facts as evidence for what matters rather than a substitute for finding it.

I kept the whole sea of truthful facts, and I added the part that lets my base AI sail on it and navigate instead of sinking. Now that conversation prep comes back on the first pass instead of the tenth, because the diagnosis is up front with the facts arranged around it.

How it navigates now

The new diagram shows how this all happens.

Before my base model answers, the EJG process runs two phases. In Phase 1, a series of fast agents first anchors the turn, then another gathers the facts, then the last one flags anything that looks off.

Phase 2 then uses a heavier model, where two agents first work side by side, one pulling the relevant skills and knowledge, the other pulling the concepts, the patterns it has seen before. Running the heavy work side by side instead of sequentially also served to make it quicker.

And then, crucially, the final agent reads all of this and writes the hypothesis about what the turn is really about. With the tweaks to the base agent, that hypothesis becomes the informed work of the powerful frontier model, now equipped with the tools it needs to answer meaningfully.

The system only became useful once it could commit to a read of the moment and make the facts answer to it, the way a good advisor does.

Anyone can build a system that remembers now. The harder thing, and the only one worth building, is a system that knows what to do with what it holds.

This is the 7th in a series about Eji, my personal AI negotiation and communications tool

1. The Eji System → komcp.com/shared-mastery-022826
2. Amplify Your Edge → komcp.com/amplify-your-edge-032326
3. Owning the Memory → komcp.com/own-the-memory-own-the-era-041326
4. More Reliable AI → komcp.com/reliable-ai-042726
5. Two Memories → komcp.com/two-memories-050826
6. Structure of Memory → komcp.com/structure-of-memory-062326
7. Drowning in Facts → komcp.com/drowning-in-facts-062926
8. The Fable of Eji → komcp.com/fable-of-eji-071626

If you want to try the universal Eji package or compare notes on what you’ve been building, reach out. jkoenig@komcp.com

This article was also posted separately on LinkedIn:
https://www.linkedin.com/posts/jacobkoenig_i-fixed-the-bug-that-was-making-my-custom-ugcPost-7476792550104956928-6q2-/