{"id":6022,"date":"2026-04-13T17:27:52","date_gmt":"2026-04-13T16:27:52","guid":{"rendered":"https:\/\/upcloud.com\/global\/?post_type=tutorial&p=6022"},"modified":"2026-04-13T17:27:52","modified_gmt":"2026-04-13T16:27:52","slug":"building-self-hosted-rag-system-open-source-tools","status":"publish","type":"tutorial","link":"https:\/\/upcloud.com\/global\/resources\/tutorials\/building-self-hosted-rag-system-open-source-tools\/","title":{"rendered":"Building a Self-Hosted RAG System with Open-Source Tools"},"content":{"rendered":"\n

Large language models are trained on a fixed dataset and don\u2019t have access to new or private data by default. This makes them unreliable for up-to-date or context-specific questions. This issue is what Retrieval-Augmented Generation<\/em> solves. Developers combine a language model with vector search so applications can answer questions using their own documents.<\/p>\n\n\n\n

Introductions<\/h2>\n\n\n\n

Most tutorials show this workflow using hosted APIs for embeddings, vector search, and model inference because it allows a working prototype to appear within minutes. And while that convenience works well early on, teams often begin to reconsider the architecture once real usage starts. Request-based pricing grows with traffic, documents move through several external services, and core application logic ends up tied to specific AI platforms. It also becomes harder to control where data is processed, how it is stored, and how portable the system is across environments.<\/p>\n\n\n\n

As a result, open-source tooling and self-hosted stacks become more relevant. Teams start looking for ways to keep sensitive data within their own infrastructure, reduce reliance on vendor-specific services, and retain the flexibility to move across providers. That shift leads to a practical question in engineering discussions: can the same system run using open-source tools on infrastructure the team controls?<\/p>\n\n\n\n

The answer today is increasingly yes. Modern inference servers expose OpenAI-compatible APIs, embedding models run comfortably on standard machines, and PostgreSQL can perform vector search through the pgvector extension. With object storage holding source documents and a small API service orchestrating retrieval and generation, the entire RAG pipeline can run on ordinary infrastructure rather than managed AI platforms. This reduces vendor lock-in and cost variability, but it also means taking on responsibility for operating and scaling the system, and performance may not match fully managed services without careful tuning or dedicated hardware.<\/p>\n\n\n\n

This tutorial walks through building that stack using open-source components on UpCloud. Instead of focusing on RAG theory, the goal is to show how realistic it is to run a minimal but practical system using an open LLM runtime, a local embedding model, PostgreSQL with pgvector, and object storage. By the end, you will have a working self-hosted RAG pipeline that mirrors the architecture many teams now deploy in production.<\/p>\n\n\n\n

What a Self-Hosted RAG Stack Looks Like<\/h2>\n\n\n\n

Before diving into the implementation, it helps to understand the main components that make up a typical RAG system. Even in self-hosted environments, most deployments follow a similar structure with a few clearly defined layers.<\/p>\n\n\n\n

1. LLM Runtime<\/h3>\n\n\n\n

The language model runtime is responsible for generating answers. In self-hosted setups, this usually means running an open-source model behind an inference server that exposes an API.<\/p>\n\n\n\n

Several runtimes now provide OpenAI-compatible endpoints<\/em>, which allow applications to interact with local models using the same SDKs commonly used for hosted AI services. Popular options include vLLM<\/a>, Ollama<\/a>, and llama.cpp<\/a>.<\/p>\n\n\n\n

vLLM is typically used for GPU-backed inference, while CPU-focused setups are better suited to runtimes like llama.cpp or Ollama. Running larger models (such as Mistral-7B) purely on CPU is possible but tends to be significantly slower, so CPU mode is often used for demos or small-scale setups rather than production deployments.<\/p>\n\n\n\n

2. Embedding Model<\/h3>\n\n\n\n

Embeddings convert text into numerical vectors that capture semantic meaning. These vectors allow the system to find relevant information even when the wording of a query differs from the source document.<\/p>\n\n\n\n

Common open models include BGE<\/a>, E5<\/a>, and various sentence-transformer models. They can run locally using Python libraries such as sentence-transformers<\/a>, making them easy to integrate into ingestion pipelines or query processing workflows.<\/p>\n\n\n\n

3. Vector Search<\/h3>\n\n\n\n

The vector database stores embeddings and performs similarity searches to find relevant document chunks.<\/p>\n\n\n\n

There are many systems designed for this task, including Qdrant<\/a>, Milvus<\/a>, and OpenSearch<\/a>. In this tutorial, we use pgvector<\/a>, a PostgreSQL extension that adds vector search capabilities directly to a relational database. It is widely supported, simple to deploy, and sufficient for many RAG workloads.<\/p>\n\n\n\n

4. Document Storage<\/h3>\n\n\n\n

The original documents used by the system are usually stored separately from the vector index. Object storage<\/a> works well for this layer because it handles large files efficiently and keeps the source material independent from derived embeddings.<\/p>\n\n\n\n

During ingestion, documents are retrieved from storage, split into smaller chunks, and converted into embeddings before being stored in the vector database. This design allows documents to be reprocessed later if the chunking strategy or embedding model changes.<\/p>\n\n\n\n

5. Retrieval and Application Layer<\/h3>\n\n\n\n

The final layer is a small application service that orchestrates the RAG workflow.<\/p>\n\n\n\n

This service receives user queries, generates query embeddings, searches the vector database, assembles relevant document context, and sends a prompt to the language model. Frameworks such as FastAPI<\/a>, Flask<\/a>, or lightweight Node.js<\/a> APIs are commonly used to implement this layer.<\/p>\n\n\n\n

Together, these five components form the core of most RAG systems.<\/p>\n\n\n\n

High-Level Architecture<\/h2>\n\n\n\n

Now that you know what the core components are, it helps to see how they interact during a typical request. A RAG system works by retrieving relevant information from a document index and using that information to guide the language model\u2019s response.<\/p>\n\n\n\n

At a high level, the flow looks like this:<\/p>\n\n\n\n

\"-\"<\/figure>\n\n\n\n

When a user submits a question, the API service first converts the query into an embedding using the same model that was used during document ingestion. That embedding is then used to search the vector database for document chunks with similar semantic meaning.<\/p>\n\n\n\n

The retrieved chunks are combined with the user\u2019s question to construct a prompt. This prompt is sent to the language model running on the inference server, which generates a response grounded in the retrieved information.<\/p>\n\n\n\n

This tutorial is intentionally minimal. It is designed to show the core mechanics of RAG without adding too many moving parts. In production systems, this pipeline is usually extended with better chunking strategies, reranking steps, caching, structured prompt templates, and monitoring around retrieval quality and latency.<\/p>\n\n\n\n

Most teams also place a decision layer in front of retrieval, using either lightweight heuristics or an LLM to classify the query first. That layer can decide whether to trigger RAG at all, which index or tenant-specific corpus to search, and whether the request should be handled by a normal model response, a retrieval workflow, or another tool altogether. Let\u2019s now understand the infrastructure needed to build this setup.<\/p>\n\n\n\n

Infrastructure Used in This Tutorial<\/h2>\n\n\n\n

With the architecture in place, the next step is choosing the infrastructure needed to run it. A self-hosted RAG system does not require an overly complex environment. In many cases, a small number of services are enough to support a fully functional pipeline.<\/p>\n\n\n\n

The most important component is compute for running the language model inference server. Depending on the model size, this can run either on CPUs or GPUs.<\/p>\n\n\n\n

A typical CPU-based setup might look like this:<\/p>\n\n\n\n