What is RAG and Why Your AI Assistant Needs It

The Hallucination Problem in LLMs

LLMs are powerful tools: they can write code, summarize documents, and carry on conversations. But they don’t know what’s true — all they do is continue text based on probabilities. That’s why an LLM might confidently say something untrue, like: “Shakespeare published a book in 1923.” That’s a hallucination. And if such a mistake happens in a customer chat where the return policy must be stated accurately — it becomes a business risk.

How RAG Works

RAG solves this problem. It injects up-to-date context into the LLM prompt — context retrieved from your own knowledge base. The process includes three steps:

1. We receive the user’s question.
2. We search for relevant fragments in the knowledge base using vector search.
3. We insert those fragments into the model’s prompt — and get a meaningful, accurate answer.
   

In this way, we don’t retrain the model — we enrich its knowledge on the fly.

Vector Search and Embeddings

RAG performs semantic search, not keyword matching. Each text is converted into a vector in a multidimensional space — this is called an embedding. Texts that are similar in meaning have vectors that are close to each other. For example, “herring” and “what swims in the sea?” will be nearby, while “giraffe” will be far away.

To get these vectors, we first extract the text from the source files: PDFs, audio, CRM documents. This can be done manually, with parsers, or even with an LLM. The key is to obtain clean, analyzable text. Because, as we say in RAG: trash in — trash out.

Hands-on With Directual

We implement a RAG subsystem in Directual — no coding required:

• Knowledge base — an interface for uploading files, parsing, cleaning, and vectorizing them.
• AI Assistant — a chat where the user interacts with the LLM, and the model receives extended context from the knowledge base.
  

We use ready-made plugins for embeddings, speech recognition, PDF parsing, and ChatGPT integration. Everything is configured visually, step by step. Under the hood: scenarios, sockets, and a well-thought-out data structure.

The result is a fully functional RAG system where:

• the user sends a message in the chat;
• the assistant gathers the conversation history;
• searches the knowledge base for relevant data;
• and forms a precise, personalized response with source references.
  

What’s Next?

In this second lesson, we’ve learned:

• how RAG works;
• why it’s essential for reliable AI assistants;
• what vector search and embeddings are;
• how to prepare and process data;
• and how to build a basic RAG system without writing code.
  

In the next — third — lesson, we’ll dive into advanced topics:

• text chunking,
• structured output and JSON formatting,
• chain of thought reasoning,
• logprobs,
• and even running models locally.
  

If you want to build production-grade AI solutions — be sure to move on to Part 3!