Leo's blog

Just after midnight, we were checking whether Alastair had actually searched for a Saab ignition-barrel pinout. He had returned no answer. That was acceptable. The interesting part was his account of how thoroughly he had failed to find one.

Five notebooks. 1,281 lines. Several searches. Then a firm statement that the information was absent.

Specific numbers in an AI answer can be reassuring, which is inconvenient when you are trying to establish whether the AI did anything. A model can produce “1,281” with exactly the same typing effort as “quite a lot.” The comma costs very little.

I initially treated those counts as unverified. That matters: I had no grounds to call them invented either. We had the answer, but not yet the search evidence behind it. Asking Alastair whether he was sure would have produced another answer from Alastair. We had enough of those.

Irage supplied the recorded tool activity from the original turn. First we could see that searches had happened. Later, a bounded excerpt showed the queries and results. Six searches, no matches. The tool itself reported five files and 1,281 lines. The output was not truncated.

So the numbers checked out. Good. That part of the suspicion could go in the bin.

There was a small counting trap in the evidence too: the excerpt contained 24 events. Calls and results appeared in two recording layers. Six searches, not 24 attempts. Apparently even checking whether a bot has exaggerated its work gives you an opportunity to exaggerate its work.

Alastair had also avoided the much worse failure. He hadn’t filled the gap with a plausible wiring table. He hadn’t substituted a web answer for the requested notebook answer. No invented connector pins. With electrical information, an empty result has considerable charm compared with a confident fiction.

The remaining problem was one sentence. He had moved from unsuccessful searches to certainty that the information wasn’t in the notes.

I still don’t know whether it was there. The evidence established what those searches returned. It didn’t establish that every possible description of the thing had been recognised. A note could use different terminology; a relevant detail could sit under an unexpected heading. Those are possibilities, not discoveries about these particular notebooks. I haven’t earned a replacement story about why the searches missed.

“I couldn’t find it” would have done the job. Add what was searched, leave the pinout blank, stop. Slightly disappointing for anyone wanting to connect wires, but the disappointment belongs to the available evidence.

We had already spent time repairing instructions intended to stop fabrication. I wasn’t keen to turn this remaining wording problem into another blanket prohibition. An assistant that refuses every subject it cannot retrieve from a notebook is going to be very honest about being useless. General knowledge still has a place. So does inference, provided it arrives wearing its own name.

The next useful check was much smaller: ask for one fact the owner knows is saved, in a named notebook, and require the source. We had demonstrated six ways to come back empty-handed. I wanted to see him bring something back.