Knowledge Hub: Multi-Source RAG System

Most RAG systems are easy to demonstrate.

Put a document into a vector database, retrieve a few chunks, pass them to an LLM, and generate an answer.

The difficult part starts when the system has to answer a more uncomfortable question:
What happens when the knowledge is no longer one document?

That was the problem I wanted to explore in M2 of my personal AI engineering project. In M1, I built Knowledge Vault, a controlled retrieval system around a single PDF. That constraint was intentional. It gave me a clean environment to study extraction, chunking, embeddings, retrieval, reranking, evaluation, and failure analysis.

The experiments taught me something important:

Retrieval quality is not just about finding semantically similar text. It is about finding evidence that is actually useful for answering the question.

M2 took that idea into a much less controlled environment.

Instead of one PDF, the system needed to work across PDFs, Markdown, source code, GitHub repositories, and technical articles.

That changed almost everything.

The problem was no longer simply:

“Can I retrieve the right passage?”

It became:

“Can I preserve the identity, structure, and provenance of knowledge as it moves through heterogeneous sources, and can I measure how different retrieval strategies behave against that knowledge?”

That became Knowledge Hub.

🔗 Github Repository: https://github.com/Jaival-Suthar/knowledge-hub

M1 solved a controlled problem. It created a bigger one.

Knowledge Vault had a useful property: the corpus was predictable. There was one source. The source had a document structure. The chunks represented passages from that document. Dense retrieval was enough to establish a strong baseline. But real engineering knowledge doesn’t look like that.

A personal knowledge base might contain:

PDFs
Markdown notes
source code
Git repositories
technical articles
configuration files
tests
documentation
project metadata

A function inside a TypeScript file is not conceptually the same thing as a paragraph in a PDF. A GitHub repository is not a document. A technical article is not necessarily stored locally. A code identifier can be highly important even when its semantic embedding is not particularly close to the query.

So I defined the M2 problem around three requirements:

Source diversity — ingest heterogeneous knowledge without creating source-specific retrieval systems.
Retrieval diversity — compare semantic and lexical retrieval instead of assuming one method is sufficient.
Evidence traceability — preserve enough provenance to understand where every retrieved chunk came from.

The architecture therefore had to separate ingestion from retrieval.

The central design decision: one canonical representation

The most important architectural decision in M2 was not Qdrant, BM25, RRF, Tree-sitter, or FastAPI. It was the canonical data model.

I wanted source-specific ingestion to disappear once data entered the retrieval system.

The core representation became:

Document
   ↓
Chunk
   ↓
Provenance

Every ingestion adapter could do whatever was necessary to understand its source. But once the content became canonical Document and Chunk objects, retrieval should not care whether the original source was:

PDF
Markdown
TypeScript
GitHub
Hashnode

This gave the system a clean boundary:

That separation ended up being one of the most important parts of the project. Without it, every retrieval strategy would eventually become coupled to every source type.

The ingestion problem was harder than it looked

Supporting another source is not simply adding another parser.
Each source has a different definition of structure.
That became very obvious during M2.

Markdown

Markdown has semantic structure that plain text extraction would throw away. Headings, lists, tables, blockquotes, and code blocks all carry meaning. The Markdown ingestion path therefore became heading-aware and token-aware while preserving important structural boundaries.

The goal wasn’t to create the smallest possible chunks. It was to create chunks that remained meaningful when retrieved independently.

Code is not prose

This was one of the biggest differences from M1. For a PDF, a paragraph is a reasonable unit of meaning. For source code, arbitrary token windows are often a poor abstraction. A function should ideally remain identifiable as a function. A class should retain its class context. An interface should not be separated from the information that defines it.

That led to AST-aware code ingestion using Tree-sitter.
The parser extracts semantic symbols such as:

classes
methods
functions
interfaces
types
enums
structs
namespaces

These symbols become the basis for semantic chunks.

The distinction matters because the retrieval system should be able to answer a query such as:

“Where is authentication implemented?”

with a meaningful implementation unit rather than an arbitrary slice of source code.

The ingestion pipeline was validated against a real code corpus.
For the PerfEngine validation:

1,928 ZIP members were inspected
252 supported files were identified
252 files were successfully parsed
730 AST symbols were extracted
730 semantic chunks were produced
252 canonical documents were created

That validation was important because it tested the ingestion contract against a non-trivial repository rather than a toy example.

ZIP ingestion also became a security boundary

Once repositories and project archives entered the system, ingestion was no longer just a parsing problem.
It became an input validation problem.
The ZIP ingestion path therefore explicitly handled cases such as:

path traversal
absolute paths
symlinks
encrypted archives
extraction limits

This is an example of something that doesn’t appear in a typical RAG architecture diagram.

But ingestion is an attack surface.

If the system accepts arbitrary archives, extraction behavior needs to be treated as part of the engineering design rather than as a convenience function.

GitHub is not a document

A GitHub repository introduced another layer of complexity.
A repository has:

a remote identity
a ref
a commit
a directory structure
multiple files
multiple languages
generated files
tests
configuration
documentation

The ingestion pipeline therefore uses shallow clones, explicit refs, commit validation, bounded timeouts, isolated temporary workspaces, and cleanup.

More importantly, provenance records the repository context.
A retrieved code chunk should not merely say:

"function authenticate()..."

It should be possible to understand where that code came from.

For the RippleTalk repository, the validation run processed 38 files and produced 25 semantic chunks.

The retrieval layer could then treat those chunks exactly like any other canonical chunks.

Articles introduced a different kind of source boundary

Technical articles are closer to Markdown than source code, but they introduce an acquisition problem. The source may live remotely.
Its content may change. Its identity is not simply a local filesystem path.
For Hashnode articles, the ingestion path therefore separates acquisition from parsing and chunking.

The existing acquisition layer retrieves the article’s public Markdown representation. That representation then passes through the existing article parser and article-aware chunker before being converted into the canonical Knowledge Hub representation. The canonical layer preserves the original chunk content, deterministic identity, source URI, heading context, ordering, and provenance.

The important design principle remained the same:

Source-specific complexity should stop at the ingestion boundary.

After ingestion, retrieval should operate on the same canonical representation.

M0 and M2 have deliberately different responsibilities

One architectural clarification became particularly important while building M2.

The local LLM from M0 is not the retrieval engine.
Knowledge Hub’s dense retrieval uses an embedding model:

BAAI/bge-small-en-v1.5

and stores vectors in Qdrant.

BM25 provides the lexical retrieval path. The Qwen3 model from M0 belongs to the inference/generation side of the larger system.

That separation matters because it allows retrieval experiments to be measured independently of generation. If retrieval quality changes, I want to know that the retrieval system changed.

I don’t want improvements or regressions to be hidden behind a different generated answer.

This was one of the principles carried forward from M1:

Measure retrieval separately from inference.

Knowledge Hub therefore does not require the local LLM to perform dense retrieval.

Dense retrieval became the baseline again

Since dense retrieval was already established in M1, I kept it as the semantic baseline. The embedding model converts chunks and queries into vector representations. Qdrant performs similarity search.

The retrieval contract returns ranked canonical chunks rather than source-specific objects.

But this time, I did not want to stop at establishing a baseline. I wanted to understand what different retrieval strategies contributed.

That led to a controlled experiment comparing:

Dense retrieval
BM25 retrieval
Dense + BM25 with Reciprocal Rank Fusion

The initial benchmark deliberately used the controlled PDF corpus.

This was important because I wanted to isolate retrieval behavior before introducing the additional variability of the complete multi-source knowledge base.

The evaluation contained:

35 total questions
30 answerable questions
5 unanswerable questions
70 explicit gold/acceptable evidence records

The 30 answerable queries were used for the Dense/BM25/Hybrid retrieval comparison.

Consider a query containing:

HybridRetriever
rrf_k
candidate_k
content_role

These are not ordinary natural-language concepts.
They are identifiers.

A semantic model may understand their general meaning, but exact lexical matching can be extremely valuable.

This led to the next experiment:
What does BM25 contribute when dense retrieval already exists?

BM25 became the second retrieval signal

The BM25 implementation indexes chunk content together with provenance metadata.

The tokenizer was intentionally designed to preserve useful identifiers while also handling forms such as camelCase.

That matters for technical corpus.
A query containing:

CrossEncoder reranking

should be able to match the exact terminology present in source code or technical documentation.

The benchmark showed a clear tradeoff.

BM25

Dense retrieval remained stronger on this benchmark, particularly in the top-ranked results. BM25, however, was substantially faster and still recovered a meaningful amount of relevant evidence at larger candidate depths.

The important observation was not simply that one retriever performed better than the other.

The more useful question was:

“Do they retrieve different useful evidence?”

Before building hybrid retrieval, I measured complementarity

I compared the candidate sets produced by Dense and BM25 over the 30 answerable benchmark queries.

The results were:

Dense only       5
BM25 only        2
Both            19
Neither          4

The average candidate intersection was approximately:

4.77 / 20

with a mean Jaccard similarity of:

0.1393

More importantly:

BM25 rescued Dense: 2 queries
Dense rescued BM25: 5 queries

The two retrievers were not producing identical candidate sets.
They were seeing the corpus differently.
That gave me a concrete reason to combine them.

Hybrid retrieval therefore became an experimental hypothesis rather than an architectural assumption:

If the two retrievers expose different evidence, can rank fusion recover useful evidence that either method would miss on its own?

RRF: combining different retrieval signals

For the hybrid system I used Reciprocal Rank Fusion.

The important property of RRF is that it doesn’t require the raw scores from different retrieval systems to be directly comparable.

Dense similarity scores and BM25 scores have different meanings and scales.

Instead, RRF operates on rank positions.
Conceptually:

Dense ranking ──┐
                ├──> RRF ──> unified ranking
BM25 ranking ───┘

The implementation uses 1-based ranks and merges results by canonical chunk_id.

The same candidate depth is passed to both retrievers.
This gives the experiment a controlled structure:

Query
  ├── Dense → top K candidates
  └── BM25  → top K candidates
             ↓
            RRF
             ↓
        unified candidates

Then the downstream evidence pipeline can operate on the fused result.

The hybrid result was more interesting than simply picking a winner

The controlled PDF benchmark produced these results:

The result was not simply that Hybrid retrieval was “better.”

The more precise observation was that Hybrid retrieval achieved the highest Recall@20 on this benchmark, while Dense retrieval retained stronger early-ranking metrics.

Specifically:

Recall@20
Dense    80.00%
BM25     70.00%
Hybrid   86.67%

Hybrid therefore recovered relevant evidence within the top 20 for more queries than either individual retriever.

At the same time:

MRR
Dense    0.5322
Hybrid   0.4615

nDCG@5
Dense    0.4768
Hybrid   0.3930

So the additional candidate coverage from RRF did not translate into stronger top-ranked retrieval on this benchmark.

Latency also increased because the Hybrid retriever executes both retrieval paths:

Dense    23.06 ms
BM25      0.73 ms
Hybrid   32.64 ms

This made the result more interesting than a simple winner-takes-all comparison.

The experiment provided evidence that combining the two retrieval signals could increase deeper candidate coverage, while the ranking metrics showed that the fused ranking behaved differently from Dense retrieval at the top of the list.

That distinction is important.

If I looked only at Recall@20, Hybrid appeared to provide a clear coverage gain.

If I also looked at Recall@1, MRR, and nDCG@5, the result became more nuanced.

This is exactly why retrieval evaluation needs multiple metrics rather than a single “accuracy” number.

RRF sensitivity: was the result dependent on one constant?

Once Hybrid retrieval showed higher Recall@20, I wanted to know whether that behavior depended heavily on the RRF constant.

Instead of treating rrf_k = 60 as a magic value, I evaluated:

k = 10
k = 20
k = 40
k = 60
k = 100
k = 200

The sensitivity matrix was:

The interesting part was the stability.
Across a 20× range of RRF constants:

Recall@1  = 36.67%
Recall@10 = 76.67%
Recall@20 = 86.67%

remained unchanged.

The variation was concentrated in the top five positions.
So I did not interpret k=40 as a universal optimum.
The narrower conclusion was:

On this benchmark, the RRF constant had little effect on candidate coverage. Its measurable effect was concentrated in the ordering of the highest-ranked results.

That was useful because it showed that the observed Hybrid coverage was not dependent on finding one “magic” RRF constant.

Retrieval depth changed the interpretation

The hybrid experiment reinforced an idea from M1:

Candidate retrieval and final ranking are different problems.

A system can successfully retrieve the correct evidence into its candidate pool without placing that evidence near the top.

That means:

Candidate Recall
        ≠
Final Ranking Quality

This distinction becomes especially important once multiple retrieval systems are fused.

Hybrid retrieval can improve the breadth of the candidate pool while still requiring another mechanism to determine which evidence should appear first.

That is where structural filtering and reranking enter.

Retrieval needed to understand source structure

M1 had already shown that semantic relevance does not automatically imply evidentiary usefulness.

M2 made that problem broader.

A knowledge base can contain:

navigation
metadata
implementation
documentation
tests
configuration
reference material
actual evidence

A table of contents can be semantically related to a question.
A test can mention an API without explaining how it works.

A configuration file can contain the exact keyword from a query while still being poor evidence for the user’s question.

So M2 introduced canonical content roles:

navigation
metadata
documentation
implementation
evidence
reference
test
configuration

These roles became part of the retrieval metadata rather than being thrown away during ingestion.

Structural eligibility was deliberately applied after RRF

The final retrieval pipeline became:

Dense Retrieval ──┐
                  ├──> RRF
BM25 Retrieval ───┘
          ↓
Structural Eligibility
          ↓
Metadata Filtering
          ↓
Cross-Encoder Reranking
          ↓
Final Evidence

The ordering is intentional.

I didn’t want one retriever to make a source-level decision independently and then prevent another retriever from contributing.

Both retrieval signals first contribute candidates. Then the system applies source-aware controls. Unknown or missing structural roles are preserved rather than aggressively discarded. That makes the filtering policy conservative.

Metadata filtering added explicit control

The next requirement was more operational.
Sometimes semantic relevance is not enough.
If I know that I want evidence from:

source_type = github
language = typescript
repository = RippleTalk

the retrieval system should be able to express that constraint directly.
M2 therefore supports metadata filtering across dimensions such as:

source_type
project
language
repository
path
content_role

The semantics are:

AND across dimensions
OR within a dimension

For example:

language = typescript
AND
repository = RippleTalk

while multiple accepted repositories can be expressed as an OR within the repository dimension.

This is different from relevance. It is retrieval control. That distinction became increasingly important as the corpus grew.

More sophisticated ranking did not automatically mean better ranking

M1 already taught me to be skeptical of the assumption that adding a reranker automatically improves retrieval.

M2 reinforced it.
The pipeline supports an optional BGE cross-encoder reranker.
The reranker scores query/chunk pairs directly and reorders the candidate set.

Conceptually:

retrieved candidates
        ↓
(query, chunk)
        ↓
cross-encoder
        ↓
reranker score
        ↓
new ranking

It sounds like an obvious improvement. The benchmark did not support that assumption. In the evaluated configuration, reranking reduced the measured ranking metrics while adding substantial latency.

That made the decision straightforward:
The reranker remains optional rather than becoming a mandatory stage.

This is an important engineering outcome.

A component does not deserve a permanent place in the architecture simply because it is more sophisticated.

If a component increases complexity and latency without improving the measured objective, the correct response is to treat it as an experiment rather than an unquestioned dependency.

The controlled PDF experiment was only the first stage

The single-PDF benchmark was useful for controlled comparison. But M2 was ultimately designed for heterogeneous sources. So I created a combined Knowledge Hub corpus and a separate multi-source gold-evidence dataset.

The multi-source dataset contained 21 answerable questions spanning articles, GitHub/code, and project sources, with explicit gold evidence attached to the questions. The dataset was established before the final source-aware retrieval experimentation.

This changed the experiment.
The question was no longer:

“Which retriever performs best on one controlled document?”

It became:

“Does the retrieval architecture continue to behave coherently once different source types share the same knowledge space?”

After the multi-source gold questions were in place, I implemented the remaining retrieval stages and evaluated them progressively as an ablation rather than assuming that every additional stage would improve the system.

The multi-source retrieval ablation

The final experiment evaluated six configurations over the same 21 answerable questions:

The complete E1–E6 ablation was based on the 21-question multi-source benchmark.

The results were more interesting than a simple “more stages means better retrieval”.

E3 — Hybrid + RRF reached 100% Recall@5, with MRR 0.7143 and nDCG@5 0.6980. This provided empirical evidence that Dense and BM25 were complementary on this heterogeneous corpus.

E4 — Structural eligibility did not change the measured ranking metrics on this benchmark. Recall@5, MRR, and nDCG@5 remained identical to E3, while latency increased only from 27.47 ms to 28.32 ms. That means the benchmark did not demonstrate a retrieval-quality improvement from structural filtering, even though the stage provides deterministic structural quality control.

E5 — Cross-encoder reranking produced the most surprising result. Recall@5 fell from 100% to 95.24%, MRR from 0.7143 to 0.6742, and nDCG@5 from 0.6980 to 0.6642, while latency increased from 28.32 ms to 5532.58 ms. The reranker therefore remained an optional component rather than becoming a mandatory stage.

E6 — Source-aware metadata filtering preserved Recall@5 at 95.24% and Recall@10/20 at 100%, while slightly changing MRR from 0.6742 to 0.6766 and nDCG@5 from 0.6642 to 0.6663. Its larger architectural value is explicit control over retrieval scope by provenance, rather than a large recall improvement on this benchmark.

The important conclusion is therefore not that every stage improved retrieval.

It is:

Different retrieval stages have different roles, and empirical evaluation is necessary to determine whether each stage actually contributes to retrieval quality.

The two evaluation stages therefore answered different questions:

Controlled PDF benchmark
        ↓
Isolate retrieval behavior
        ↓
Dense / BM25 / Hybrid
        ↓
RRF-k sensitivity
        ↓
Understand ranking vs coverage

Multi-source gold-evidence benchmark
        ↓
Evaluate heterogeneous sources
        ↓
Hybrid retrieval
        ↓
Structural eligibility
        ↓
Cross-encoder reranking
        ↓
Source-aware metadata filtering
        ↓
Measure the contribution and cost of each stage

Keeping these experiments separate prevented me from treating one benchmark as representative of the entire system.

The retrieval contract became more important than the retrievers

One architectural lesson from M2 was that retrieval implementations should not own the entire data model. Dense retrieval knows about vectors. BM25 knows about lexical terms. RRF knows about ranks. The reranker knows about query-document pair scores. None of them should need to understand the full ingestion architecture.

That is why the shared retrieval contract is built around the canonical Chunk and its provenance. The retrieval layer can therefore evolve independently.

Today:

Dense
BM25
Hybrid

can operate over the same representation.

The important part is not the number of retrievers.
It is that they agree on what a retrieved piece of evidence is.

Evaluation became part of the architecture

One of the biggest changes from M1 to M2 was the evaluation harness.
The controlled PDF benchmark contains:

35 total questions
30 answerable questions
5 unanswerable questions
70 explicit gold/acceptable evidence records

The evaluation measures:

Recall@1
Recall@5
Recall@10
Recall@20
MRR
nDCG@5
latency

The goal was not to produce a single leaderboard number.
The goal was to make retrieval changes measurable.
A retrieval experiment should answer:

What changed?
Why did it change?
Did candidate coverage improve?
Did ranking improve?
What did it cost?

That mindset became one of the most useful parts of the project.

The API became the system boundary

Once the ingestion and retrieval layers were stable, I needed a clean way to expose the system. That became the FastAPI layer.

The API exposes:

GET  /health
POST /search
POST /retrieve
POST /ingest
GET  /documents
GET  /sources

The important design choice was to keep the API thin.
It does not contain a second retrieval implementation.
It does not duplicate ingestion logic.
It does not recreate the metadata model.
It calls into the existing canonical pipeline.

The architecture therefore becomes:

                  ┌───────────────┐
                  │    FastAPI    │
                  └───────┬───────┘
                          │
                ┌─────────▼─────────┐
                │ Canonical Retrieval│
                │      Pipeline      │
                └─────────┬─────────┘
                          │
        ┌─────────────────┼─────────────────┐
        │                 │                 │
      Dense              BM25            Filters
        │                 │                 │
        └─────────────────┼─────────────────┘
                          │
                        Qdrant

Swagger/OpenAPI makes the subsystem directly inspectable without requiring a separate client.

That matters because the project is no longer just an experiment running through scripts.

It now has a stable interface.

Persistence also changes the architecture

Qdrant is not treated simply as a temporary vector store. The API startup path hydrates the corpus view from persisted data.

That gives the service a useful property:
The API does not need to reconstruct the entire corpus from scratch every time it starts. The canonical documents and chunks remain persisted, while the service reconstructs the operational view it needs.

This is another example of separating:

source acquisition

from:

runtime retrieval

What M2 actually completed

At the end of M2, Knowledge Hub can:

ingest PDFs
ingest Markdown
ingest source code
parse code using Tree-sitter
ingest GitHub repositories
ingest Hashnode articles
preserve canonical provenance
persist canonical chunks in Qdrant
perform dense retrieval
perform BM25 retrieval
combine retrieval signals with RRF
apply structural eligibility
apply metadata filters
optionally rerank candidates
evaluate retrieval with a reproducible benchmark
expose the system through FastAPI
expose OpenAPI/Swagger for interactive inspection

The final validation suite reached:

324 passed
6 skipped

The system was also validated through real corpus ingestion, GitHub ingestion, Hashnode ingestion, PerfEngine parsing, Uvicorn/API startup, Swagger/OpenAPI access, PDF retrieval, Ruff, compile checks, and repository diff validation.

That was the point where I considered M2 complete.

What M2 intentionally does not solve

There is one boundary I want to make explicit.

Knowledge Hub does not automatically synchronize changing sources.

If a GitHub repository changes, the system does not yet automatically detect the new commit and incrementally update only the affected documents.

If an article changes, the system does not automatically determine which chunks became stale.

If a source disappears, deletion propagation is not automatic.
For the completed M2 system, ingestion is explicit.
That is intentional scope control.

Future work

Knowledge Hub does not yet include the following capabilities. They are planned as productionization work and are not part of the completed M2 implementation. They fall into two areas.

Incremental indexing

Today, ingestion is explicit. Making the system maintain itself as sources change would require tracking identity and version at every level:

content_hash
document identity
chunk identity
source version

Each document and chunk would be classified by change state:

NEW
MODIFIED
UNCHANGED
DELETED

For GitHub, this would mean tracking the repository, branch/ref, commit SHA, and file path, so that a new commit updates only the affected documents.

For articles, it would mean tracking article identity, content hash, and update/version information, so that stale chunks can be detected and replaced.

Deletion would also propagate, so that a source that disappears no longer leaves evidence behind in the index.

Retrieval Inspector

The second area is observability. Rather than presenting only the final answer, the Retrieval Inspector would expose the retrieval pipeline as a sequence of stages:

QUERY
  ↓
Dense Results
  ↓
BM25 Results
  ↓
RRF Results
  ↓
Structural Filtering
  ↓
Metadata Filtering
  ↓
Reranking
  ↓
Final Evidence

At each stage, it would show what the candidate set looked like and what changed, with enough information to answer a question like:

Why did this chunk rank here?

Together, these two additions would move Knowledge Hub from a system that can be measured at a point in time to one that stays current and can be explained continuously.

The biggest lessons from M2

**Multi-source retrieval is an ingestion problem before it is a retrieval problem
**It is tempting to think that adding more sources simply means adding more documents to the vector database.

It doesn’t.

Different sources have different structures, identities, acquisition mechanisms, and security considerations.

If those differences are not handled before retrieval, the retrieval layer becomes polluted with source-specific assumptions.

The canonical Document → Chunk → Provenance model was therefore more important than any individual retriever.
**Dense retrieval is powerful, but semantic similarity is not the whole search problem
**
Dense retrieval performed strongly on the benchmark. But the complementarity analysis showed that BM25 and Dense did not retrieve the same candidates. This was particularly relevant for technical terminology and identifiers. The lesson isn’t that BM25 replaces embeddings.

It is that different retrieval signals encode different notions of relevance.
Hybrid retrieval changed coverage more than ranking

I did not want to add BM25 simply because “hybrid search is what production systems do.” The candidate overlap gave me an empirical reason to test it. Then the benchmark showed where the hybrid system actually helped:
```
Recall@20:
Dense   80.00%
Hybrid  86.67%
```
while top-ranked metrics did not improve.

The RRF sensitivity experiment added another layer.

Across k=10 through k=200, Recall@10 and Recall@20 remained unchanged. The variation was concentrated in the top-five ordering.

So the more accurate conclusion is:

On this benchmark, hybrid retrieval improved deeper candidate coverage more than it improved ranking quality.

That is a much more useful result than calling hybrid retrieval universally better.
More ranking sophistication can make a system worse
The reranker was technically capable. That did not make it automatically useful. On the evaluated configuration, it reduced measured ranking metrics while increasing latency.

That is a reminder that architecture should follow measured objectives. Not the other way around.
Provenance is not metadata decoration
Once multiple sources enter the system, provenance becomes part of retrieval correctness.

Knowing that a chunk came from:
```
repository
→ commit
→ file
→ symbol
```
or:
```
article
→ section
→ chunk
```
changes how confidently the evidence can be interpreted.

Without provenance, retrieval can tell you what matched. With provenance, it can also tell you where it came from.
Retrieval quality needs multiple measurements
Recall@20 answered one question.
MRR answered another.
nDCG@5 answered another.
Latency answered a completely different one.

The hybrid experiment made this obvious. One metric suggested a gain.
Other metrics showed that the gain did not translate into better top-ranked evidence.

The RRF sensitivity experiment showed that even a parameter sweep can affect ranking without changing deeper candidate coverage. That is why I don’t want to reduce retrieval evaluation to a single number.

What changed from M1 to M2?

M1 was about understanding retrieval failure inside a controlled document.

M2 was about making the retrieval system survive heterogeneity.
The progression looks roughly like this:

M1
Single PDF
   ↓
Dense Retrieval
   ↓
Reranking
   ↓
Structural Failure Analysis
   ↓
Passage-Level Evaluation

M2 expanded the problem:

Multiple Sources
   ↓
Canonical Ingestion
   ↓
Source-Aware Provenance
   ↓
Dense + BM25
   ↓
RRF Hybrid Retrieval
   ↓
RRF-k Sensitivity
   ↓
Structural Eligibility
   ↓
Metadata Filtering
   ↓
Optional Reranking
   ↓
Reproducible Evaluation
   ↓
FastAPI Boundary

The interesting part is that M2 did not simply add features.
It changed the questions I could ask about the system.

The real outcome of M2

When I started M2, I thought the primary problem was:

“How do I make RAG work across more sources?”

By the end, that no longer felt like the right question.
The harder problem was:

How do I preserve evidence identity across heterogeneous sources while keeping retrieval strategies interchangeable, measurable, and inspectable?

That led to several design decisions that I would not have made from a typical RAG tutorial:

source-specific ingestion with source-agnostic retrieval
canonical chunks with explicit provenance
AST-aware code chunking
lexical retrieval alongside dense retrieval
empirical candidate complementarity analysis
RRF rather than score normalization
RRF sensitivity evaluation
structural eligibility after candidate fusion
metadata filtering as explicit retrieval control
optional rather than mandatory reranking
passage-level evaluation
retrieval/inference separation
a thin API boundary over the actual system

The biggest lesson is probably the same one that emerged from M1:
The difficult part of RAG is not making the pipeline produce an answer.

The difficult part is being able to explain why the system retrieved what it retrieved, measure where it fails, change one component at a time, and know whether the change actually improved the objective.

M1 taught me how to investigate retrieval failure inside one document.
M2 forced me to do that across an entire knowledge base.

And that is a much more interesting engineering problem.

Command Palette