Choose RAG Chunk Size by Answer Length

Choose RAG chunk size by the evidence an answer needs, then test candidate sizes on questions from your own workload. A fact supported by one sentence may be easy to retrieve in a small chunk; a technical explanation may need a larger passage to preserve its conditions and context. Select a size only after checking whether the retrieved text supports a good answer at an acceptable cost.

The Fraunhofer IAIS study tested nonoverlapping chunks of 64, 128, 256, 512 and 1,024 tokens across several question-answering datasets. Its concise, fact-based questions generally favored 64–128 tokens, while broader technical responses often favored 512–1,024. The results concern retrieval: document structure and embedding model also affected the preferred size, and the study did not directly score generated answers.

Classify questions by the evidence they require

Build a small evaluation set from questions the system is expected to answer. Include short factual requests, technical explanations and questions about long documents. For each one, write an expected answer and mark every source span needed to support it. If a technical answer depends on a definition and an exception elsewhere in the document, mark both spans.

Answer length is a useful starting signal, but evidence location makes the distinction operational. A brief answer can still depend on dispersed passages; a long answer might summarize one self-contained section. Record whether its evidence is local, crosses a chunk boundary or appears in separate parts of a document. This lets you distinguish a failure to find the right passage from a failure to retrieve enough of it.

Run a small, controlled matrix

Use the five tested token sizes as candidates rather than defaults. For an initial screen, index the same corpus at 64, 128, 256, 512 and 1,024 tokens, then retrieve the top two and top five chunks for each size. Run each combination against all three question groups. That produces 30 size, depth and question-type cells; the numbers describe a proposed experiment, not a claimed benchmark result.

Keep text extraction, tokenizer, embedding model, retriever, prompt and generation model fixed. Rebuild the index for each chunk size.
Hold the overlap policy fixed, preferably starting without overlap, so it does not obscure the effect of size. Record actual chunk lengths and source locations.
Save the ranked chunks returned for every question. Inspect boundaries where a retrieved passage includes an answer term but loses the condition that makes it correct.

An undergraduate-textbook experiment compared sentence, paragraph, page and chapter segmentation while varying how many passages were retrieved. That is a different segmentation method from fixed token windows, but it illustrates why retrieval depth belongs in the comparison: changing the number of passages changes the context supplied to the generator.

After the initial screen, rerun the strongest candidates under the same context-token cap. Retrieving five large chunks supplies more text than retrieving five small ones, so equal chunk counts do not make an equal-cost comparison. Log how many chunks fit under the cap and whether truncation removes required evidence.

Score retrieval before generating answers

For each configuration, measure whether the marked evidence appears in the returned context. A single-span question can be scored as an evidence hit at the chosen retrieval depth. For a question needing several spans, record both the proportion recovered and whether all required spans are present. Keep the rank of the first useful chunk: evidence that appears only at the bottom of a longer result list may disappear under a tighter context cap.

Inspect misses by question group rather than relying on one average. Small chunks may split an instruction from its qualifying condition. Large chunks may make a precise fact harder to rank because much of the passage concerns something else. More retrieved chunks may recover a missing span, but the added text can also crowd the prompt. These are different failure modes and call for different changes.

Databricks’ RAG evaluation guidance treats retrieval quality, response quality, token use and latency as separate measures. That separation matters here: a correct answer on a few questions does not establish that the retriever reliably finds the evidence, and high retrieval recall does not establish that the generated answer uses it correctly.

Test the answer and account for cost

Generate answers for the leading configurations using the same model and prompt. For factual questions, judge correctness and support in the retrieved text. For technical explanations, also judge whether the answer includes necessary conditions and avoids unsupported additions. If people review the outputs, hide the chunk configuration from them so the setting itself does not influence their judgment.

Keep a worksheet with separate columns for evidence coverage, answer correctness, answer support, indexed chunk count, retrieved tokens, generation input tokens and response time. Larger chunks can mean fewer stored vectors while sending more text per query. Overlap can increase repeated context and indexing work; test it later if boundary failures show a reason to add it.

Compare configurations within each question group before weighting the groups by their frequency in the actual workload. A setting that excels on short facts may be a poor choice for technical questions. If those groups need sharply different sizes, a separate retrieval route or expansion from a small matched passage to its surrounding section is worth testing under the same answer and cost measures.

Make the choice reproducible

Choose the least costly configuration that meets the evidence-coverage and answer-quality thresholds appropriate to the application. Keep the evaluation questions, marked source spans, corpus version, chunking settings, retrieval depth and context cap with the result. Those details make a later comparison meaningful when the documents, embedding model or mix of questions changes.