RAG in production: infrastructure, data security and why your chatbot answers wrong

Where the answer gets lost: seven failure points

RAG (retrieval-augmented generation) looks simple: documents are split into chunks and converted to vectors, the most similar chunks are retrieved for a query, and the model composes an answer from them. In production, however, the answer can get lost in several places, and each one needs a different fix.

A good map is provided by the study Seven Failure Points When Engineering a Retrieval Augmented Generation System (Barnett et al., January 2024), based on three case studies from research, education and biomedicine. It describes seven failure points:

The first three points are purely a matter of data and retrieval. Swapping the language model for a newer one does nothing about them. The study's authors also reach two conclusions worth treating as default assumptions: a RAG system can only be validated during operation, and its robustness cannot be designed up front but evolves over time.

Retrieval: hybrid search, reranking and chunking

Pure vector search captures meaning well but fails on exact strings – a contract number, a product code, the acronym of an internal system. Full-text search (BM25) is the opposite. Hybrid search runs both queries in parallel and merges the results. Azure AI Search uses Reciprocal Rank Fusion for this: each document receives a score of 1/(rank + k) for its position in each list, and the scores are summed; according to the documentation, the algorithm works best with a small value of k, such as 60.

How much difference the individual layers make was measured by Anthropic in its post on Contextual Retrieval (September 2024). The metric was the share of queries for which the relevant chunk did not make it into the top 20 chunks:

• embeddings enriched with document context: a drop from 5.7% to 3.7% (35% lower),
• after adding contextual BM25: a drop to 2.9% (49% lower),
• after adding reranking: a drop to 1.9% (67% lower).

These are a vendor's measurements on its own datasets, so the absolute numbers on your documents will come out differently. The order of steps, however, applies generally: hybrid first, then reranking, and only then model tuning.

Chunking is not a detail

The seven-failure-points study notes that chunking sounds trivial, yet its quality affects the embeddings and therefore how well chunks match queries. A typical problem: a paragraph pulled out of an internal policy does not say which policy, which version or which branch it applies to. Contextual Retrieval solves this by adding to each chunk, before indexing, a short description of where it sits in the document.

Agentic RAG

Compound questions (“compare the terms of two contracts”) cannot be covered by a single index query. The agentic approach breaks the query into sub-queries, runs them in parallel and merges the results. In Azure AI Search, according to documentation updated in September 2026, agentic retrieval is generally available in REST API version 2026-04-01, while LLM-based query planning and answer synthesis remain in preview. Expect higher latency and cost per query – each sub-query is reranked separately – and the fact that another LLM call becomes part of a chain you have to measure.

Vector database, pgvector or a managed service

The choice of store is often made first, yet it has less influence on answer quality than chunking and reranking. Operational criteria matter more: who will run it, how filtering by permissions works and whether the store supports hybrid search.

The pgvector details come from the project documentation. For Bedrock Knowledge Bases, the AWS documentation lists the supported stores as OpenSearch Serverless, managed OpenSearch Service clusters, S3 Vectors, Aurora with pgvector, Neptune Analytics, Pinecone, Redis Enterprise Cloud and MongoDB Atlas. Hybrid search, however, is according to the same documentation supported only for Amazon RDS, OpenSearch Serverless and MongoDB with a filterable text field; elsewhere the query silently falls back to pure semantic search. By default, the service returns at most five chunks.

The counterargument to “just use pgvector”: as soon as you need high-quality full-text search in Czech, result fusion and reranking, you are writing and maintaining a piece of a search engine yourself. If the team lacks the capacity, a managed service works out cheaper despite the higher running cost.

Data security: permissions must be enforced at retrieval time

The most common RAG security flaw is not an exotic attack but an index with broader permissions than the user asking the question. Ingestion runs under a service account that reads everything, and the chatbot then happily summarizes payroll documents for anyone. OWASP ranks this among the top risks in LLM08:2025 Vector and Embedding Weaknesses: unauthorised access and data leakage, cross-context leaks in a shared vector database, embedding inversion and data poisoning. As defenses it lists fine-grained access control with logical partitioning of datasets, source validation, data classification and immutable retrieval logs.

The principle is simple: the permission filter belongs in the index query, not in the prompt. The instruction “do not reveal confidential documents” is not a security control. What never reaches the model's context, the model cannot leak.

In Azure AI Search, the security filter pattern is generally available: group identifiers are stored with each document at indexing time, and the application adds them to the query filter.

POST /indexes/securedfiles/docs/search?api-version=2026-04-01
{
  "search": "contract termination terms",
  "filter": "group_ids/any(g:search.in(g, 'group_id1, group_id2'))"
}

The documentation states explicitly that in this pattern the identity is just a string in the filter – no authentication or authorization takes place on the service side. Your application is therefore responsible for building the filter correctly. The variants in which the service compares the user's token with permissions inherited from the source (ACLs and RBAC for ADLS Gen2, ACLs from SharePoint, Microsoft Purview sensitivity labels) are, according to the access control overview, still in preview as of September 2026.

Indirect prompt injection: the document as an attack vector

By design, RAG puts text into the model's context that neither the user nor the developer wrote. OWASP LLM01:2025 describes this as indirect prompt injection: the model accepts input from an external source, and the content in it alters the model's behavior. OWASP also states that neither RAG nor fine-tuning fully eliminates this vulnerability.

EchoLeak showed that this is not theory. The vulnerability CVE-2025-32711, published on 11 June 2025, is described in the NVD as AI command injection in M365 Copilot that allows an unauthorised attacker to disclose information over a network. Microsoft gave it a CVSS score of 9.3 (critical), NIST 7.5. According to an analysis published on arXiv (Reddy and Gujral, September 2025), a crafted e-mail and no user action were enough: the attack chained a bypass of the prompt injection classifier, a bypass of link redaction via Markdown reference-style links, auto-loaded images and a Teams proxy allowed by the content security policy.

The second category is knowledge base poisoning. The paper PoisonedRAG (Zou et al., accepted at USENIX Security 2025) reports a 90% attack success rate when injecting five malicious texts per target question into a knowledge base of millions of texts; according to the authors, the defenses they evaluated were insufficient. This is a lab result, but the lesson is practical: whoever can write to the sources you index can influence the answers.

None of these measures is sufficient on its own; in the EchoLeak case, both input and output filters were bypassed. The goal is that a successful injection has nothing to reach and no way to get data out.

Ingestion as a pipeline, evaluation as a test

Ingestion is a build. It has inputs (documents, parser version, chunking strategy, embedding model), an output (the index), and it should be reproducible. The same rules apply as for any other CI/CD pipeline: configuration in version control, separate environments, controlled access to secrets and the ability to roll back to a previous version of the index.

• Version the chunking and the embedding model. Changing either of them means re-indexing everything; plan it as a migration with a parallel index and a cutover.
• Handle deletions and duplicates. An outdated version of a policy in the index is a common source of “wrong” answers that are actually faithful – just to an old document.
• Store metadata. Source, owner, validity date, classification and groups with access; without them you can neither filter nor cite.
• Check content at the input. Scan for secrets and personal data before they reach the index.

Evaluation: measure retrieval and generation separately

Without measurement, you cannot tell whether a chunking change helped. The Ragas framework offers RAG metrics for context precision and context recall (retrieval quality) and faithfulness and response relevancy (answer quality). Faithfulness is defined as the share of claims in the answer that are supported by the retrieved context, out of all claims in the answer.

Two caveats. LLM-based metrics call a model themselves, so they are not deterministic and they cost money. And high faithfulness does not mean a correct answer: the model can faithfully summarize an outdated or planted document. That is why you also need a set of questions with known correct answers and sources, put together with people from the business, and run it as a regression test on every pipeline change.

In operation, log for every answer the query, the user's identity, the IDs of the retrieved chunks, and the index and prompt versions. This is the basis for debugging as well as for incident investigation; more on the topic in our article on security monitoring and logging. Keep in mind, though, that query logs themselves contain sensitive data and need the same protection as the source documents.

What to do on Monday

If the audit shows that documents are scattered, without owners and without classification, start there. RAG on top of disorganised data only makes the mess accessible faster – the same pattern as with infrastructure that is not ready for AI.

Key takeaways

• Most wrong answers originate before the model. Missing content, poor result ranking and loss during context assembly are data and retrieval problems.
• Hybrid search and reranking are the foundation, not an add-on. In Anthropic's measurements, together with adding context to chunks, they reduced the share of failed retrievals from 5.7% to 1.9%; verify the benefit on your own data.
• Permissions are enforced in the index query. The prompt is not a security control, and synchronization lag is a security parameter.
• Every indexed document is a potential instruction. EchoLeak demonstrated a zero-click data leak from a production system; defend yourself by separating sources, applying least privilege and restricting output.
• Ingestion is a pipeline and evaluation is a test. Version everything, measure retrieval and generation separately, and expect validation to happen only in operation.
• Choose the store based on operations. pgvector, Azure AI Search and Bedrock Knowledge Bases all work; the difference is what the service does for you and what is still only in preview.