H
Haoxiang Li
Guest
A paired 230-question experiment with the same BM25 implementation, two tool interfaces, and a result that made me inspect the evidence actually delivered to the model.
Disclosure: I build Minimal RAG, the personal research project discussed here. This article was prepared with AI assistance from recorded experiments and checked against the linked results. It adapts my previously published project case study; it is not independent validation of the project.
Both versions used BM25. The version that returned more of the reference-document IDs answered fewer questions correctly.
That was the interesting result in a historical comparison on 230 BrowseComp-Plus questions. My project's search/read interface answered 217 correctly. A standard search interface answered 194. Search-only reference-document recall moved in the opposite direction: 86.12% for the project interface and 89.71% for the standard one.
The comparison made me pay more attention to the boundary between finding a document and delivering its useful passage. It did not establish that the read tool alone caused the difference. Returned text, available controls, and prompts changed together.
Suppose a relevant document appears in the results, but the returned text contains only its opening paragraphs. If the needed fact is later in the document, the retrieval metric can count a hit while the model still lacks the answer.
This example describes an interface failure mode, not a reconstruction of every error in the experiment. It points to a question that a result-list metric cannot settle: what exact source text did the model receive, and what could it do when that text was insufficient?
Minimal RAG keeps a small loop. The model requests a tool call, the loop executes it, and the resulting observation goes back to the model. The model chooses queries, sources, reading ranges, and when to answer. The default retrieval implementation uses SQLite FTS5 BM25; there is no embedding model or reranker in this configuration.
The broad idea of alternating model decisions with external information gathering is established research. ReAct studies interleaved reasoning and actions, including interaction with external sources. IRCoT studies interleaving retrieval with reasoning for multi-step question answering, where later retrieval depends on information obtained earlier.
Those papers provide context for the design. They do not validate Minimal RAG's scores. This project does not reproduce their experimental protocols or compare its numbers directly with their reported results. The contribution described here is narrower: an engineering comparison of two evidence interfaces over the same BM25 implementation.
The project interface offered two tools. Search could return one to twenty distinct documents, with 2,000-character excerpts, source IDs, offsets, and document lengths. It supported pagination and searching within a particular document. Read defaulted to 4,000 characters and allowed continuation from an offset.
The standard search interface returned five results per query, each containing the beginning of a document capped at 512 tokens. It had no separate read tool. Characters and tokens are different units; those limits do not imply a simple excerpt-length ratio.
Both arms used the same BM25 ranking implementation and core loop, the Codex backend with
The recorded experiment report provides the protocol and links to its supporting records.
Original plot of the project's recorded metrics, produced with AI-assisted plotting code. Historical comparison of an interface bundle, not an isolated read-tool ablation.
On the same 230 questions, the project interface versus standard search recorded:
There were 28 wins, five losses, and 197 ties for the project interface. The original paired-bootstrap 95% interval for the ten-percentage-point accuracy difference was +5.22 to +14.78 points. The per-question numerical CSV makes the counts inspectable without republishing benchmark questions or answers.
The recall metric counts reference-document IDs returned by search. It does not measure whether the decisive passage was visible, and it does not fully describe evidence obtained through read calls. Its higher value for the standard interface therefore does not imply that the model had better evidence available for answering.
Locate the returned text. A source ID plus offsets makes an observation interpretable. A reader can distinguish the document's opening from the passage near a lexical match.
Make incompleteness visible. Document length and a truncation indicator tell the model that its current view may be partial. They do not themselves prove that more reading will help.
Offer a continuation action. Reading another span, searching inside the same source, or requesting another page of candidates gives the model a concrete next step. These are available actions; the loop does not force every question through a fixed sequence.
In a trace review, I would inspect candidate IDs, the exact returned spans, continuation calls, and the final answer's cited sources. A correct ID appearing somewhere in the search history is a weaker observation than the needed fact actually reaching the model.
The questions had been used during development. They were not a holdout untouched by design decisions. The paired interval does not account for those selection effects or for variability across repeated model generations.
Several interface properties changed together, so the aggregate result cannot allocate the gain among read access, passage placement, parameters, and prompts. Equal model-call caps also do not establish equal token use or billed cost. The result says nothing about BM25 universally outperforming dense retrieval.
The current direct Responses API backend is a separate configuration with only a 30-question Enterprise migration check. These historical Codex results do not automatically transfer to that backend or to its open call budget.
My practical conclusion is to evaluate the evidence interface alongside the retriever. Before adding another component, check whether the model can locate a useful passage, recognize a partial view, and continue gathering evidence. This experiment gives me a reason to investigate those capabilities carefully, while leaving their individual causal contributions open.
I maintain the project linked above. The paper references explain the research context; the project records support the reported comparison. No new model evaluations were run for this article.
Disclosure: I build Minimal RAG, the personal research project discussed here. This article was prepared with AI assistance from recorded experiments and checked against the linked results. It adapts my previously published project case study; it is not independent validation of the project.
Both versions used BM25. The version that returned more of the reference-document IDs answered fewer questions correctly.
That was the interesting result in a historical comparison on 230 BrowseComp-Plus questions. My project's search/read interface answered 217 correctly. A standard search interface answered 194. Search-only reference-document recall moved in the opposite direction: 86.12% for the project interface and 89.71% for the standard one.
The comparison made me pay more attention to the boundary between finding a document and delivering its useful passage. It did not establish that the read tool alone caused the difference. Returned text, available controls, and prompts changed together.
A Document ID Is Only the Start of the Evidence Path
Suppose a relevant document appears in the results, but the returned text contains only its opening paragraphs. If the needed fact is later in the document, the retrieval metric can count a hit while the model still lacks the answer.
This example describes an interface failure mode, not a reconstruction of every error in the experiment. It points to a question that a result-list metric cannot settle: what exact source text did the model receive, and what could it do when that text was insufficient?
Minimal RAG keeps a small loop. The model requests a tool call, the loop executes it, and the resulting observation goes back to the model. The model chooses queries, sources, reading ranges, and when to answer. The default retrieval implementation uses SQLite FTS5 BM25; there is no embedding model or reranker in this configuration.
This Fits an Existing Research Direction
The broad idea of alternating model decisions with external information gathering is established research. ReAct studies interleaved reasoning and actions, including interaction with external sources. IRCoT studies interleaving retrieval with reasoning for multi-step question answering, where later retrieval depends on information obtained earlier.
Those papers provide context for the design. They do not validate Minimal RAG's scores. This project does not reproduce their experimental protocols or compare its numbers directly with their reported results. The contribution described here is narrower: an engineering comparison of two evidence interfaces over the same BM25 implementation.
What Actually Changed Between the Two Arms
The project interface offered two tools. Search could return one to twenty distinct documents, with 2,000-character excerpts, source IDs, offsets, and document lengths. It supported pagination and searching within a particular document. Read defaulted to 4,000 characters and allowed continuation from an offset.
The standard search interface returned five results per query, each containing the beginning of a document capped at 512 tokens. It had no separate read tool. Characters and tokens are different units; those limits do not imply a simple excerpt-length ratio.
Both arms used the same BM25 ranking implementation and core loop, the Codex backend with
gpt-6.1-sol at high reasoning effort, a fixed corpus without external web access, and a cap of 30 model calls per question. Answers were scored by this project using a pinned official script and Qwen3-32B BF16. That is local use of an official evaluator, not a maintainer-verified leaderboard result.The recorded experiment report provides the protocol and links to its supporting records.
Compare the Same Questions
Original plot of the project's recorded metrics, produced with AI-assisted plotting code. Historical comparison of an interface bundle, not an isolated read-tool ablation.
On the same 230 questions, the project interface versus standard search recorded:
- Correct answers: 217 versus 194, or 94.35% versus 84.35%.
- Mean search calls: 12.16 versus 19.84.
- Mean read calls: 3.33 versus zero.
- Mean search-only reference-document recall: 86.12% versus 89.71%.
There were 28 wins, five losses, and 197 ties for the project interface. The original paired-bootstrap 95% interval for the ten-percentage-point accuracy difference was +5.22 to +14.78 points. The per-question numerical CSV makes the counts inspectable without republishing benchmark questions or answers.
The recall metric counts reference-document IDs returned by search. It does not measure whether the decisive passage was visible, and it does not fully describe evidence obtained through read calls. Its higher value for the standard interface therefore does not imply that the model had better evidence available for answering.
Three Interface Details I Would Inspect Before Adding Components
Locate the returned text. A source ID plus offsets makes an observation interpretable. A reader can distinguish the document's opening from the passage near a lexical match.
Make incompleteness visible. Document length and a truncation indicator tell the model that its current view may be partial. They do not themselves prove that more reading will help.
Offer a continuation action. Reading another span, searching inside the same source, or requesting another page of candidates gives the model a concrete next step. These are available actions; the loop does not force every question through a fixed sequence.
In a trace review, I would inspect candidate IDs, the exact returned spans, continuation calls, and the final answer's cited sources. A correct ID appearing somewhere in the search history is a weaker observation than the needed fact actually reaching the model.
What the Experiment Still Does Not Establish
The questions had been used during development. They were not a holdout untouched by design decisions. The paired interval does not account for those selection effects or for variability across repeated model generations.
Several interface properties changed together, so the aggregate result cannot allocate the gain among read access, passage placement, parameters, and prompts. Equal model-call caps also do not establish equal token use or billed cost. The result says nothing about BM25 universally outperforming dense retrieval.
The current direct Responses API backend is a separate configuration with only a 30-question Enterprise migration check. These historical Codex results do not automatically transfer to that backend or to its open call budget.
My practical conclusion is to evaluate the evidence interface alongside the retriever. Before adding another component, check whether the model can locate a useful passage, recognize a partial view, and continue gathering evidence. This experiment gives me a reason to investigate those capabilities carefully, while leaving their individual causal contributions open.
I maintain the project linked above. The paper references explain the research context; the project records support the reported comparison. No new model evaluations were run for this article.