RAG Gateway: from documents to cited answers

The problem: fluent does not mean informed

A capable local model still cannot answer from a handbook it has never seen. RAG Gateway connects assistants to a searchable knowledge base, retrieving evidence at question time and returning it to the model with source labels.

The goal is to make retrieval reusable. Open WebUI and Hermes can use the same ingestion and retrieval service rather than implementing separate knowledge pipelines.

From document to answer

The pipeline has two distinct stages.

During ingestion, the Gateway extracts text, divides it into passages, requests embeddings from MongoDB’s AI Models API, and stores passage text, metadata, and vectors in MongoDB Atlas.

At question time, it embeds the query, retrieves passages filtered to the selected tenant, and assembles an evidence-augmented prompt for Ollama. Answers can reference retrieved sources using labels such as [S1] and [S2].

Open WebUI / Hermes
         |
   FastAPI Gateway
         |
         +-- MongoDB AI Models API: embeddings
         +-- MongoDB Atlas: passage storage and search
         +-- Ollama: answer generation

This is a reference application architecture, not a map of a private deployment.

A gateway instead of separate integrations

The shared API brings ingestion, retrieval, tenant selection, and evidence assembly into one place. Clients select the public model name rag-local; the Gateway routes generation to the configured Ollama model.

The project includes several ways to understand and operate that pipeline:

• A Document Loader for previewing and ingesting supported files from uploads or allowlisted directories.
• Chunk management for inspecting documents and passages by tenant.
• A Playground for retrieval-only inspection and RAG chat.
• A retrieval evaluation workbench comparing retrieved document IDs with labeled expected sources.

These interfaces turn retrieval into something that can be inspected, rather than an invisible step before generation.

Comparing retrieval options

The project supports vector retrieval, optional hybrid retrieval, and optional reranking. Hybrid search combines vector and lexical rankings using Reciprocal Rank Fusion. It requires a separate Atlas Search index.

The Playground compares eight combinations of hybrid search, reranking, and reasoning. A useful experiment keeps the corpus, model, top-k, and generation settings fixed, varies one factor at a time, and separates cold calls from warm runs.

In the published example, all eight configurations found the three expected source documents for one question. Recorded time ranged from 5.1 seconds with all three options off to 38.2 seconds with reranking and high reasoning enabled.

That example illustrates a cost of additional processing for one question. It does not establish which configuration is best overall. Repeated runs across a representative question set are needed to choose defaults.

Evaluate several kinds of correctness

Retrieval quality and answer quality answer different questions. The retrieval workbench reports hit rate, recall, mean reciprocal rank, and latency; it does not grade whether the generated answer is correct.

The broader evaluation approach considers factual lookup, semantic and lexical matching, multi-source synthesis, conflicting updates, false premises, multilingual queries, and abstention when evidence is absent.

Citation validity, temporal precedence, answer completeness, and tenant boundaries should be evaluated separately. Tenant-isolation tests need dedicated test tenants and credentials; a correct answer in one authorized tenant is not evidence of isolation.

The private evaluation documents and expected answers are not reproduced on this site.

Where the data goes

“Local-first” describes an important part of the design, but the pipeline uses cloud services:

Tenant-scoped keys can restrict configured tenant IDs, but this version does not implement document-level ACLs or an end-user identity provider. Authentication and network controls remain part of the deployment design.

Lessons and current limits

The value of the Gateway is a shared, inspectable retrieval path. Source labels and evaluation tools make it easier to understand where an answer came from and where the pipeline needs work.

The current implementation uses character-based chunking, which does not preserve Markdown section structure. Managed search indexes must be provisioned separately. Retrieval scores from vector-only and hybrid search are not directly comparable, and embeddings, managed retrieval, and optional reranking can contribute to cost.

The full English article includes the detailed evaluation method, operating notes, interface screenshots, and additional limitations. The repository contains the implementation.

RAG Gateway is an independent personal project. It is not affiliated with, sponsored by, or endorsed by MongoDB, Inc. This article is an editorial adaptation of the public project documentation.