In RAG, Ten Sources Are Not Ten Pieces of Evidence

[ilias konstantinidis]

The Retrieval Count Is Not the Evidence Count​

A typical retrieval pipeline has a simple mental model: find relevant chunks, rank them, pass the best ones to the model, and let agreement among them increase confidence. That works reasonably well when the retrieved sources are genuinely independent.

The internet is not independent.

News sites syndicate reporting. Blogs paraphrase documentation. Corporate announcements get rewritten by trade publications. Wikipedia pages cite articles that cite the same original document. Social posts repeat screenshots from a single account. Even technically different documents can belong to the same information chain.

Suppose a source A makes a claim. B, C, and D repeat A. Four other pages summarize B and C. A search engine retrieves all seven secondary pages plus A itself.

A naive system sees eight supporting documents. An influence-aware system sees something closer to one upstream lineage with multiple descendants.

Text deduplication does not solve this. B and C may use completely different wording while depending on the same upstream evidence. Semantic similarity does not solve it either: two genuinely independent eyewitness reports can be semantically similar, while two dependent reports can look linguistically different.

The problem is epistemic dependence, not duplicate text.

Why This Matters for RAG​

RAG systems are often introduced as an antidote to hallucination: retrieve external evidence, ground the model in documents, and cite the sources. That is useful, but grounding only helps if we understand what the retrieved material actually represents.

If ten retrieved documents all inherit one error, retrieval can turn a single bad claim into something that looks strongly corroborated. Worse, the system may become more confident precisely because the error has propagated widely.

This is the same mechanism behind an echo chamber. Repetition is mistaken for independence.

In my recent research on influence-aware evidence aggregation, I built controlled scenarios to measure this effect. The two headline results were large: a confidence-inflation ratio of 4.00 in a common-parent echo chamber and 10.00 in a duplicated cascade. But those numbers only mean something if the experimental setup is visible. So here is the machinery behind them.

What the Experiment Actually Did​

The benchmark was deliberately synthetic. The point was not to imitate the whole internet. It was to isolate one variable: what happens to confidence when multiple supporting items are not actually independent?

Each evidence item carried a small confidence increment and a dependence label. For the controlled benchmark, an item could contribute as:

• independent: weight 1.0
• unknown: weight alpha, set to 0.5 in the main runs
• dependent: weight 0.0
• partially dependent: an explicit weight between 0 and 1

Those weights were summed into an Effective Independent Evidence count, or EIE. The system then converted EIE into an independence factor:

gamma = min(1, EIE / K)

where K is the amount of independent evidence required for the confidence update to receive full credit. The raw support from a batch was the sum of its evidence increments. The dependence-aware update multiplied that raw support by gamma before adding it to the prior confidence.

In compact form:

raw_increment      = sum(evidence increments) 
adjusted_increment = gamma * raw_increment 
C_aware            = min(1, C_prior + adjusted_increment)

The naive comparator did exactly what many simple aggregation schemes effectively do: it treated every supporting item as independent and added every increment in full.

The inflation ratio measured how much more support the naive model added relative to the dependence-aware model:

IR = (C_naive - C_prior) / (C_aware - C_prior)

An IR of 1.0 means there was no inflation. An IR of 4.0 means the naive model added four times as much confidence as the dependence-aware model.

Four dependency regimes​

I tested four controlled scenarios.

1. Fully independent evidence. Ten evidence items were all marked independent. Each added 0.05 support, with prior confidence 0.30 and K = 10. Because EIE reached 10, gamma = 1.0. Both the naive and dependence-aware models ended at 0.80. Inflation ratio: 1.00.

This is the control case that matters most. The method did not penalize genuinely independent evidence.

2. Duplicated cascade. Ten apparent evidence items were supplied, but only one was independent; the other nine were dependent duplicates. Each item again carried a 0.05 increment, the prior was 0.10, and K = 10.

The naive model counted all ten increments:

C_naive = 0.10 + (10 x 0.05) = 0.60

The dependence-aware model saw EIE = 1, so gamma = 1 / 10 = 0.10. It credited only one tenth of the raw increment:

C_aware = 0.10 + (0.10 x 0.50) = 0.15

The inflation ratio was therefore:

IR = (0.60 - 0.10) / (0.15 - 0.10)    
   = 0.50 / 0.05    
   = 10.00

That is where the 10.00 figure came from. It is not a universal constant. It is the consequence of a deliberately extreme stress test in which nine of ten apparent confirmations were duplicates.

3. Common-parent echo chamber. This scenario used twelve apparent sources, but only three counted as genuinely independent units; the other nine were modeled as dependent on shared upstream provenance. Each item contributed 0.04, the prior was 0.20, and K = 12.

Naive aggregation treated all twelve items as independent and ended at 0.68. The dependence-aware model had EIE = 3, so gamma = 3 / 12 = 0.25 and ended at 0.32.

IR = (0.68 - 0.20) / (0.32 - 0.20)    
   = 0.48 / 0.12    
   = 4.00

That is the 4.00 result. The apparent source count was twelve; the modeled independent support was three.

4. Mixed ecosystem. The less extreme scenario used 18 evidence items: four independent, five unknown, six dependent, and three partially dependent with weight 0.4. With alpha = 0.5, K = 12, a 0.04 increment per item, and prior confidence 0.25, the effective independent evidence was:

EIE = 4 + (5 x 0.5) + (6 x 0) + (3 x 0.4)
    = 7.7  

gamma = 7.7 / 12       
      = 0.6417

The naive model reached 0.97. The dependence-aware model reached 0.71. Inflation ratio: 1.56.

The full controlled comparison looked like this:

Scenario	Naive	Bayesian independence	Source-discounted	Dependence-aware	IR
Fully independent	0.80	0.76	0.80	0.80	1.00
Duplicated cascade	0.60	0.45	0.15	0.15	10.00
Common-parent echo chamber	0.68	0.63	0.32	0.32	4.00
Mixed ecosystem	0.97	0.86	0.56	0.71	1.56

The important pattern is not that one formula always produces a lower number. It is that the gap opens specifically when the evidence structure contains dependence, while disappearing in the fully independent control.

A check outside the synthetic benchmark​

I also tested a semi-realistic provenance case using 500 claims from the FEVER fact-verification corpus. FEVER does not provide ground-truth influence graphs, so I used a deliberately conservative heuristic: evidence sentences from the same Wikipedia article were treated as sharing page-level provenance.

Under that setup, mean confidence fell from 0.268 under naive aggregation to 0.229 under the dependence-aware method, with a mean inflation ratio of 2.72. Cluster alerts fired in 1.6% of claims. The dataset itself was not especially duplication-heavy - the mean duplicate-page fraction was only 8.4% - which is useful because it shows the mechanism was not only reacting to the deliberately extreme synthetic cascades.

None of these values should be read as a universal calibration for every RAG system. They are outputs of a specified evidence model and specified parameters. The engineering result is narrower and more useful: when confidence grows with supporting-document count, dependence structure can materially change the result, and a system can preserve full credit for genuinely independent evidence while discounting repeated lineage.

Represent the Lineage, Not Just the Documents​

The practical fix starts with a graph.

Instead of representing evidence as a flat list of retrieved chunks, represent known or suspected influence relationships between them. Nodes can be documents, claims, publishers, authors, datasets, press releases, repositories, or other upstream entities. Edges represent known dependence: copied from, cites, syndicates, derives from, republishes, or shares a common parent.

Now the system can distinguish four retrieved documents from only two upstream origins.

You do not need a perfect causal map of the internet. That would be impossible. Partial dependence knowledge is still useful. Some relationships are explicit: hyperlinks, citations, canonical URLs, wire-service attribution, repository forks, quoted sources, or shared dataset provenance. Others can be inferred probabilistically and marked as uncertain rather than silently treated as independent.

That last distinction matters. “Unknown” should not automatically mean “independent.” It should mean exactly what it says: the relationship is unknown.

Count Effective Evidence, Not Raw Evidence​

Once dependence is represented, the next step is to stop using raw document count as a proxy for corroboration.

Imagine five documents supporting a claim. If all five are independent, the system should preserve the benefit of five independent observations. If four are descendants of one source and only the fifth is independent, the effective evidence count should be much closer to two than five.

There are many ways to formalize this. The method I tested uses an estimate of effective independent evidence and dampens confidence growth when the evidence graph contains strong dependence. The implementation details can vary. The design constraint is what matters: confidence should grow with informational diversity, not merely with repetition.

Unknown Dependence Is a First-Class State​

Real systems will rarely know the full provenance graph. That does not make the approach useless. It means uncertainty about dependence has to be represented explicitly.

A source can be marked as independent, dependent, or unknown relative to another source. Unknown relationships can receive a configurable weight between full independence and full dependence. The system can then expose that assumption in its audit trail rather than burying it in an implicit default.

This also gives operators useful monitoring signals. A claim supported by eight documents but only two known lineages is different from a claim supported by eight sources whose relationships are mostly unknown. Both deserve different treatment from eight demonstrably independent sources.

What I Would Add to a Production RAG Pipeline​

If I were implementing this in a production knowledge system, I would add five things before changing the language model itself.

1. A source registry Give every source a stable identity and record publisher, author, canonical URL, timestamps, and known upstream relationships.

1. Claim-level provenance Keep the link between a generated answer, the supporting chunks, their documents, and any upstream entities. Do not flatten this into a list of citations after generation.
2. Dependence labels Represent known-independent, known-dependent, and unknown relationships explicitly. Avoid the convenient fiction that missing metadata means independence.
3. Confidence damping If several supporting items share a lineage, prevent them from contributing the same confidence gain as independent observations.
4. An audit view Show operators why the system thinks evidence is diverse. A confidence score without its evidence structure is hard to debug and even harder to trust.

None of this requires the LLM to reason perfectly about provenance. Much of the work belongs in the retrieval and evidence layer: metadata extraction, graph construction, deterministic aggregation, and monitoring.

That is important because this is not fundamentally a prompting problem.

Citations Are Not Corroboration​

Modern AI interfaces increasingly show citations, which is a good development. But citation count can create a false sense of security if users read five links as five independent confirmations.

A citation answers: “Where did this statement come from?”

Corroboration asks a harder question: “How many genuinely distinct evidential paths support it?”

Those are not the same thing.

A good RAG system should be able to say not only, “I found ten relevant sources,” but also, “Seven appear to derive from the same upstream report, two are independent, and one has unknown provenance.” That is much closer to the information a human analyst actually needs.

The Internet Is a Graph. Our Evidence Models Should Be Too​

Retrieval systems flatten the web because lists are convenient. Search results are lists. Vector databases return ranked lists. Context windows receive lists of chunks. The underlying information environment, however, is a network.

When we ignore that network, popularity can masquerade as corroboration. One claim copied a hundred times can outweigh three genuinely independent observations simply because it occupies more documents.

The fix is not to distrust repeated information automatically. Repetition is useful. It can show dissemination, salience, and consensus. It just should not be counted as independent evidence unless we have reason to believe it is independent.

Your RAG system may retrieve ten sources. Before turning that into confidence, ask one more question: how many information lineages did it actually retrieve?