A viral post announced that “RAG is officially cooked.” IBM had apparently built a retriever that hallucinates 65 times less, which somehow meant autonomous agents could run without babysitting and enterprises could enjoy “zero hallucinations” and “zero liability.”
That is a heroic amount of bullshit to extract from one paper.
The paper is STAIR: Structure Aware Information Retriever. It describes a sensible idea: give a model the table of contents of a structured document, then constrain it to select a valid leaf node. The evaluation used 18 structured books across six domains.
The famous 65× result is real, inside the paper’s definition. A competing system generated invalid document identifiers 3.25% of the time. STAIR did it 0.05% of the time.
That is useful engineering. If the retriever must choose from known section names, constraining the output prevents it from inventing section names. Good.
It did not eliminate factual hallucinations from an answer. It did not prove an autonomous agent could operate safely. It did not make legal liability evaporate.
STAIR still selected the wrong section 18.67% of the time. Its Recall@1 was 82.6%. Enterprise-scale datasets are future work in the paper itself.
The trick in the viral post is embarrassingly simple. Take the paper’s narrow use of “hallucination,” quietly swap in the ordinary meaning of a model inventing facts, then sprint from 18 books to unsupervised enterprise agents and zero liability.
The researchers built a better retriever. The influencer built a bonfire out of the definitions.
I like the paper. The post is horseshit.
0 responses to “Zero Liability, 18.67% Wrong”