How to build a document Q&A system on ASP.NET Core and Azure OpenAI. RAG architecture, Azure AI Search, chunking strategy, citations, and real cost estimates.
How to build a document Q&A system on ASP.NET Core and Azure OpenAI. RAG architecture, Azure AI Search, chunking strategy, citations, and real cost estimates.
The most common AI request we hear from businesses with existing ASP.NET Core applications in 2026 is some version of: "We have thousands of documents contracts, manuals, policies, knowledge base articles and our users can't find anything. Can AI fix this?"
The answer is yes, and the pattern that fixes it has a name: Retrieval-Augmented Generation (RAG). This guide explains what RAG actually is, how it works inside an ASP.NET Core application, what the implementation looks like in C#, and what it realistically costs and takes to build based on real engagements rather than vendor marketing.
A document Q&A system lets users ask questions in natural language and get answers drawn from your actual documents not from the AI model's general training data. The distinction is critical.
When you ask GPT-4o a general question, it answers from its training data knowledge baked in during model training, which ends at a cutoff date and doesn't include your proprietary documents. A document Q&A system with RAG changes this: it retrieves relevant passages from your documents first, then gives those passages to GPT-4o as context, then asks GPT-4o to answer the user's question based on that context. The answer comes from your documents, not from GPT-4o's general knowledge.
This means:
RAG has two distinct phases that are easy to confuse. Understanding both is essential for scoping a real project.
Phase 1: Indexing (Runs Once, Then on Updates)
Before any user can ask questions, the documents need to be processed and stored in a form the system can search quickly. This happens offline not during a user's query.
Your documents (PDF, Word, HTML, SharePoint)
↓
Document parsing
(Azure Document Intelligence for PDFs/Word)
↓
Text chunking
(split into 300–800 token segments with overlap)
↓
Embedding generation
(Azure OpenAI text-embedding-3-large API)
↓
Vector storage
(Azure AI Search stores text + embedding vector)What happens at each step:
Document parsing
Extracting clean text from your source formats. PDFs created digitally extract cleanly. Scanned PDFs require OCR (Azure Document Intelligence handles this). Word and Excel files extract via the DocumentFormat.OpenXml SDK or Azure Document Intelligence. SharePoint content is accessible via the Microsoft Graph API.
Text chunking
Splitting documents into segments small enough for the embedding model to process meaningfully. This is deceptively important: chunks that are too small lose context; chunks that are too large dilute relevance. A common approach for most business documents is 500-token chunks with 100-token overlap the overlap ensures content at chunk boundaries isn't lost.
Embedding generation
Converting each text chunk into a vector (a list of floating-point numbers) that represents its semantic meaning. Azure OpenAI's text-embedding-3-large produces 3,072-dimensional vectors. Semantically similar text produces similar vectors this is what makes semantic search work.
Vector storage
Azure AI Search stores both the original text chunk and its vector. At query time, it can find the most semantically similar chunks to a user's question in milliseconds.
Phase 2: Query (Runs on Every User Question)
User question: "What is our policy on remote work expenses?"
↓
Embed the question
(same Azure OpenAI embedding model)
↓
Vector search in Azure AI Search
(find top 3–5 most similar document chunks)
↓
Build the prompt
(question + retrieved chunks as context)
↓
Send to Azure OpenAI GPT-4o
↓
Return answer with citations
"Based on the Remote Work Policy (page 3): Employees may claim..."The key insight: GPT-4o never sees your entire document corpus. It only sees the 3–5 chunks that Azure AI Search identified as most relevant to the specific question. This keeps costs low (fewer tokens per query), keeps answers focused, and prevents the model from being distracted by irrelevant content.
Here is what the actual C# implementation looks like, using Microsoft Agent Framework 1.0 and Azure AI Search together.
Document Indexing Service
public class DocumentIndexingService(
SearchClient searchClient,
EmbeddingClient embeddingClient,
DocumentAnalysisClient documentAnalysisClient)
{
public async Task IndexDocumentAsync(
Stream documentStream,
string documentName,
string documentId,
CancellationToken ct = default)
{
// Step 1: Extract text using Azure Document Intelligence
var operation = await documentAnalysisClient
.AnalyzeDocumentAsync(
WaitUntil.Completed,
"prebuilt-read",
documentStream,
cancellationToken: ct);
var content = operation.Value.Content;
// Step 2: Chunk the text
var chunks = ChunkText(content, maxTokens: 500, overlapTokens: 100);
// Step 3: Generate embeddings and index each chunk
var indexDocuments = new List<SearchDocument>();
for (int i = 0; i < chunks.Count; i++)
{
var embedding = await embeddingClient
.GenerateEmbeddingAsync(chunks[i], ct);
indexDocuments.Add(new SearchDocument
{
["id"] = $"{documentId}-chunk-{i}",
["documentId"] = documentId,
["documentName"] = documentName,
["chunkIndex"] = i,
["content"] = chunks[i],
["contentVector"] = embedding.Value.ToFloats().ToArray(),
["indexedAt"] = DateTimeOffset.UtcNow
});
}
// Step 4: Upload to Azure AI Search
await searchClient.UploadDocumentsAsync(indexDocuments, ct);
}
private static List<string> ChunkText(
string text, int maxTokens, int overlapTokens)
{
// Approximate tokenization split on sentence boundaries
// For production, use Microsoft.ML.Tokenizers for accurate counts
var sentences = text.Split(
new[] { ". ", ".\n", "!\n", "?\n" },
StringSplitOptions.RemoveEmptyEntries);
var chunks = new List<string>();
var current = new StringBuilder();
var currentTokens = 0;
foreach (var sentence in sentences)
{
var sentenceTokens = sentence.Length / 4; // rough approximation
if (currentTokens + sentenceTokens > maxTokens
&& current.Length > 0)
{
chunks.Add(current.ToString().Trim());
// Keep the last overlap worth of content
var words = current.ToString().Split(' ');
var overlapWords = words
.TakeLast(overlapTokens / 5)
.ToArray();
current.Clear();
current.Append(string.Join(" ", overlapWords));
currentTokens = overlapTokens;
}
current.Append(sentence).Append(". ");
currentTokens += sentenceTokens;
}
if (current.Length > 0)
chunks.Add(current.ToString().Trim());
return chunks;
}
}Document Q&A Query Service
public class DocumentQAService(
SearchClient searchClient,
EmbeddingClient embeddingClient,
ChatClient chatClient)
{
public async Task<QAResponse> AskAsync(
string question,
string? filterByDocumentId = null,
CancellationToken ct = default)
{
// Step 1: Embed the question
var questionEmbedding = await embeddingClient
.GenerateEmbeddingAsync(question, ct);
// Step 2: Search for relevant chunks
var searchOptions = new SearchOptions
{
VectorSearch = new VectorSearchOptions
{
Queries =
{
new VectorizedQuery(
questionEmbedding.Value.ToFloats())
{
KNearestNeighborsCount = 5,
Fields = { "contentVector" }
}
}
},
// Hybrid search: vector + keyword for better recall
SearchMode = SearchMode.Any,
Select = { "id", "documentName", "chunkIndex", "content" },
Size = 5
};
// Optional: filter to a specific document
if (filterByDocumentId != null)
searchOptions.Filter =
$"documentId eq '{filterByDocumentId}'";
var searchResults = await searchClient
.SearchAsync<SearchDocument>(question, searchOptions, ct);
// Step 3: Collect retrieved chunks with source info
var retrievedChunks = new List<RetrievedChunk>();
await foreach (var result in searchResults.Value.GetResultsAsync())
{
retrievedChunks.Add(new RetrievedChunk
{
DocumentName = result.Document["documentName"].ToString()!,
ChunkIndex = (int)result.Document["chunkIndex"],
Content = result.Document["content"].ToString()!,
Score = result.Score ?? 0
});
}
if (!retrievedChunks.Any())
{
return new QAResponse
{
Answer = "I couldn't find relevant information in the " +
"available documents to answer this question.",
Sources = []
};
}
// Step 4: Build the prompt with retrieved context
var context = string.Join("\n\n---\n\n",
retrievedChunks.Select((c, i) =>
$"[Source {i + 1}: {c.DocumentName}]\n{c.Content}"));
var systemPrompt = """
You are a helpful assistant that answers questions based
strictly on the provided document context.
Rules:
- Answer only from the provided context
- If the context doesn't contain the answer, say so clearly
- Cite the source document name when referencing information
- Be concise and accurate
- Do not add information not present in the context
""";
var userMessage = $"""
Context from documents:
{context}
Question: {question}
Answer based strictly on the context above.
Cite the source document for any information you provide.
""";
// Step 5: Get the answer from Azure OpenAI
var response = await chatClient.CompleteChatAsync(
[
new SystemChatMessage(systemPrompt),
new UserChatMessage(userMessage)
],
cancellationToken: ct);
return new QAResponse
{
Answer = response.Value.Content[0].Text,
Sources = retrievedChunks
.Select(c => new DocumentSource
{
DocumentName = c.DocumentName,
ChunkIndex = c.ChunkIndex
})
.DistinctBy(s => s.DocumentName)
.ToList()
};
}
}
public record QAResponse
{
public required string Answer { get; init; }
public required List<DocumentSource> Sources { get; init; }
}
public record DocumentSource
{
public required string DocumentName { get; init; }
public required int ChunkIndex { get; init; }
}ASP.NET Core API Endpoints
// In Program.cs using Minimal APIs for the Q&A endpoints
var docsGroup = app.MapGroup("/api/documents")
.RequireAuthorization(); // enforce your existing auth
// Upload and index a document
docsGroup.MapPost("/index", async (
IFormFile file,
DocumentIndexingService indexingService,
ICurrentUserService currentUser,
CancellationToken ct) =>
{
var documentId = Guid.NewGuid().ToString();
await using var stream = file.OpenReadStream();
await indexingService.IndexDocumentAsync(
stream, file.FileName, documentId, ct);
return Results.Ok(new { documentId, message = "Indexed successfully" });
})
.DisableAntiforgery(); // for multipart form upload
// Ask a question
docsGroup.MapPost("/ask", async (
AskRequest request,
DocumentQAService qaService,
CancellationToken ct) =>
{
var response = await qaService.AskAsync(
request.Question,
request.DocumentId,
ct);
return Results.Ok(response);
});
record AskRequest(string Question, string? DocumentId = null);Registration in Program.cs
// Azure AI Search
builder.Services.AddSingleton(_ =>
new SearchClient(
new Uri(builder.Configuration["AzureAISearch:Endpoint"]!),
builder.Configuration["AzureAISearch:IndexName"]!,
new DefaultAzureCredential())); // Managed Identity no key needed
// Azure OpenAI embeddings and chat
var openAIClient = new AzureOpenAIClient(
new Uri(builder.Configuration["AzureOpenAI:Endpoint"]!),
new DefaultAzureCredential());
builder.Services.AddSingleton(_ =>
openAIClient.GetEmbeddingClient("text-embedding-3-large"));
builder.Services.AddSingleton(_ =>
openAIClient.GetChatClient("gpt-4o"));
// Document Intelligence for PDF/Word parsing
builder.Services.AddSingleton(_ =>
new DocumentAnalysisClient(
new Uri(builder.Configuration["DocumentIntelligence:Endpoint"]!),
new DefaultAzureCredential()));
// Application services
builder.Services.AddScoped<DocumentIndexingService>();
builder.Services.AddScoped<DocumentQAService>();The Azure AI Search Index Configuration
The index schema needs to be created before documents can be indexed. For a production document Q&A system:
public static async Task CreateIndexIfNotExistsAsync(
SearchIndexClient indexClient,
string indexName)
{
var fields = new List<SearchField>
{
new SimpleField("id", SearchFieldDataType.String)
{ IsKey = true },
new SimpleField("documentId", SearchFieldDataType.String)
{ IsFilterable = true },
new SearchableField("documentName")
{ IsFilterable = true, IsSortable = true },
new SimpleField("chunkIndex", SearchFieldDataType.Int32)
{ IsSortable = true },
new SearchableField("content"),
new SimpleField("indexedAt",
SearchFieldDataType.DateTimeOffset)
{ IsSortable = true },
new VectorSearchField("contentVector", dimensions: 3072,
vectorSearchProfileName: "myHnswProfile")
};
var index = new SearchIndex(indexName)
{
Fields = fields,
VectorSearch = new VectorSearch
{
Profiles =
{
new VectorSearchProfile(
"myHnswProfile", "myHnsw")
},
Algorithms =
{
new HnswAlgorithmConfiguration("myHnsw")
{
Parameters = new HnswParameters
{
M = 4, // graph connectivity
EfConstruction = 400, // index quality
EfSearch = 500, // query quality
Metric = VectorSearchAlgorithmMetric.Cosine
}
}
}
},
SemanticSearch = new SemanticSearch
{
Configurations =
{
new SemanticConfiguration("mySemanticConfig",
new SemanticPrioritizedFields
{
ContentFields = { new SemanticField("content") }
})
}
}
};
await indexClient.CreateOrUpdateIndexAsync(index);
}Pure vector search finds semantically similar content it excels when users ask questions using different words than the document uses. Pure keyword search finds exact matches it's essential when users search for specific terms (product codes, names, regulation numbers) that need to match exactly.
Hybrid search combines both and consistently outperforms either alone:
var searchOptions = new SearchOptions
{
// Vector search component
VectorSearch = new VectorSearchOptions
{
Queries =
{
new VectorizedQuery(questionEmbedding)
{
KNearestNeighborsCount = 10,
Fields = { "contentVector" }
}
}
},
// Keyword search component runs simultaneously
// (the 'question' parameter in SearchAsync is the keyword query)
SearchMode = SearchMode.Any,
QueryType = SearchQueryType.Semantic, // re-ranks with semantic model
SemanticSearch = new SemanticSearchOptions
{
ConfigurationName = "mySemanticConfig",
SemanticQuery = question // used for semantic re-ranking
},
Select = { "id", "documentName", "chunkIndex", "content" },
Size = 5
};
// The question string is the keyword query
// The vector search runs in parallel
var results = await searchClient.SearchAsync<SearchDocument>(
question, searchOptions, ct);This three-stage approach (vector retrieval → keyword retrieval → semantic re-ranking) gives Azure AI Search's best retrieval quality and is what Microsoft recommends for production document Q&A systems.
One of the most common design oversights in document Q&A systems is failing to enforce document-level access control. If your ASP.NET Core application restricts which documents different users can see, the Q&A system must respect those same restrictions otherwise it becomes an authorization bypass.
public async Task<QAResponse> AskAsync(
string question,
ClaimsPrincipal user,
CancellationToken ct = default)
{
// Get the document IDs this user is entitled to access
// — from your existing authorization service
var allowedDocumentIds = await _documentAuthService
.GetAccessibleDocumentIdsAsync(user);
if (!allowedDocumentIds.Any())
return new QAResponse
{
Answer = "You don't have access to any documents.",
Sources = []
};
// Build a filter — only search within authorized documents
var filter = string.Join(" or ",
allowedDocumentIds.Select(id =>
$"documentId eq '{id}'"));
var searchOptions = new SearchOptions
{
Filter = filter, // enforced at the search layer
// ... rest of search options
};
// ... rest of query logic
}The authorization filter runs at the Azure AI Search layer before results are returned to the application. This is the correct pattern because it prevents unauthorized document chunks from ever being retrieved, regardless of what the user asks.
RAG systems produce incorrect answers. Understanding when and why is essential for setting realistic expectations and designing appropriate fallbacks.
Common failure modes:
Low confidence retrieval
The user's question doesn't match any document content well, so Azure AI Search returns loosely related chunks, and the AI generates an answer that sounds plausible but is based on the wrong context. Fix: Return a confidence threshold if the highest retrieval score is below a defined threshold, respond with "I couldn't find relevant information" rather than generating an answer.
Conflicting information across documents
If your document corpus has two documents with contradictory information, the AI may produce a hedged or incorrect answer. Fix: Include source citations in every answer so users can verify against the original document.
Multi-step reasoning across documents
"What is the total cost of all the items on contract C-2024-001?" requires the AI to identify relevant line items across potentially many chunks and sum them. This is hard for current models to do reliably. Fix: Identify these structured-data queries explicitly and handle them with a database query rather than AI retrieval.
Recent documents not yet indexed
If a user uploads a document and immediately asks a question about it, the indexing pipeline may not have completed. Fix: Return an indexing status indicator per document and gracefully handle queries against documents still being processed.
For a practical reference a mid-sized professional services firm with 5,000 policy and contract documents, averaging 15 pages each, with staff of 200 asking an average of 10 questions each per month (2,000 monthly queries):
Initial indexing:
Monthly running costs:
Total monthly running cost: approximately $300/month
For 200 staff members, that is $1.50 per user per month compared to the staff time currently spent searching manually through the same documents. Read more: Cost to Add AI to ASP.NET Core App.
How accurate is a document Q&A system?
Accuracy depends primarily on three factors: document quality (clean, well-structured text performs significantly better than scanned PDFs or inconsistently formatted documents), chunking strategy (correct chunk size for your document type and typical question length), and the clarity of the user's question. For well-structured corporate documents with clear questions, modern RAG systems using GPT-4o achieve 85–92% accuracy on factual questions with answer verification against the source document. For ambiguous questions or poorly structured documents, accuracy drops significantly.
Can the system handle multiple languages?
Yes. Azure OpenAI GPT-4o handles multilingual input and output natively you can ask questions in English about documents written in French, Spanish, or German. Azure AI Search's semantic search supports multiple languages. For the best retrieval accuracy on non-English documents, use language-specific chunking and consider whether the text-embedding-3-large model (which has strong multilingual support) is sufficient for your language pair, or whether a language-specific embedding model is needed.
How do we keep the index current when documents are updated?
Document updates should trigger re-indexing of the affected document. The standard pattern in an ASP.NET Core application is to delete all existing chunks for the document ID, then re-run the indexing pipeline with the updated content. For SharePoint or OneDrive sources, Microsoft Graph API webhooks can trigger re-indexing when a document changes. For uploaded documents, trigger re-indexing on the document upload endpoint after processing.
Can we prevent the AI from answering questions not in our documents?
Yes through the system prompt instructions (the Rules section in the implementation above) combined with a retrieval confidence threshold. The "Answer only from the provided context" instruction tells the model to stay within the retrieved content. The confidence threshold prevents low-relevance retrieval from triggering an answer at all. Neither is foolproof a determined user can sometimes elicit off-topic responses but together they substantially restrict the AI to your document corpus.
Is the document content sent to OpenAI for training?
No not when using Azure OpenAI Service specifically. Azure OpenAI does not use your prompts, completions, or document content for model training. Your data stays within your Azure subscription. This is distinct from using OpenAI's direct API (api.openai.com), which has different data handling terms. The Azure OpenAI terms explicitly state: "Your prompts (inputs) and completions (outputs), your embeddings, and your training data are not used by Microsoft or OpenAI to train, retrain, or improve the base models."
How long does it take to build a document Q&A system on our existing ASP.NET Core app?
For a focused initial version upload a document, index it, ask questions about it a working proof of concept against your actual documents is typically achievable in the first sprint (2 weeks of development after a discovery phase). A production-ready system with authorization enforcement, hybrid search, confidence thresholds, citation UI, and document management takes 6–10 weeks total depending on your existing ASP.NET Core architecture's readiness for the integration.
Document Q&A is the AI feature that delivers the clearest, most immediate ROI for most businesses it solves a problem every organization with more than a few hundred documents recognizes, it integrates into existing ASP.NET Core applications without requiring a rebuild, and the Azure infrastructure that makes it work is within most SMB budgets at well under $500/month to run.
The implementation is well-understood engineering rather than research. The patterns above RAG with Azure AI Search, hybrid search, authorization enforcement at the retrieval layer, confidence thresholds are production-tested and represent current best practice for .NET AI development.
If you're ready to build a document Q&A system on your existing ASP.NET Core application, our AI integration team can assess your document corpus and existing architecture and give you a realistic scope before any development commitment is made.
