Skip to content
JavaAgentic

Type at least two characters. Try “RAG”, “pgvector” or “tool calling”.

LangChain4j Text Splitters

Chunk documents effectively in LangChain4j: the recursive splitter, chunk size and overlap tuning, splitting code and markdown, and why chunking is the highest-impact decision in RAG.

Intermediate4 min readUpdated
On this page

Chunking is the highest-leverage decision in a RAG system and the one people spend the least time on. The chunk is the unit of retrieval — if the answer is split across two chunks, or buried in a chunk full of unrelated text, no embedding model or prompt can recover it. This tutorial covers LangChain4j's splitters and how to tune them.

Key Takeaways

  • The chunk is the unit of retrieval; chunking caps what your system can ever find.
  • The recursive splitter respects natural boundaries — the right default.
  • Chunk size trades precision against context; overlap prevents boundary cuts.
  • Tune against a real question set, not intuition. Different content wants different sizes.

The recursive splitter

The default choice. It splits on the largest natural boundary that keeps chunks under the target size — paragraphs first, then sentences, then words.

Recursive splitting
// 500-token chunks, 100-token overlap. A sensible starting point for prose.
DocumentSplitter splitter = DocumentSplitters.recursive(500, 100);
 
List<TextSegment> segments = splitter.split(document);

Because it prefers paragraph and sentence boundaries, related sentences stay together far more often than with a blind fixed-length cut.

Chunk size: the core trade-off

// Small chunks: precise matches, but a chunk may lack the context that makes
// it meaningful. Good for FAQ-style content with self-contained answers.
DocumentSplitters.recursive(200, 40);
 
// Large chunks: rich context, but the relevant sentence is diluted by
// surrounding text, weakening the match. Good for narrative documents.
DocumentSplitters.recursive(800, 150);

Content-specific splitting

Different content has different natural boundaries.

ContentApproachWhy
ProseRecursive, 300-600 tokensParagraph boundaries carry meaning
Reference / API docsRecursive, 500-800 tokensKeep a whole entry together
Source codeSplit by function or classNever split mid-function
MarkdownSplit on headingsHeadings are semantic boundaries
Tables / CSVWhole rows with the headerA row without its header is meaningless
Splitting by paragraph then sentence, with size limits
// You can compose splitters for finer control.
DocumentSplitter splitter = DocumentSplitters.recursive(
        512,                    // max segment size in tokens
        64,                     // overlap
        new OpenAiTokenizer());  // count tokens accurately for this model

Counting tokens correctly

Chunk sizes are in tokens, and tokenisation differs by model. Pass the right tokenizer so your target sizes are accurate:

Tokenizer tokenizer = new OpenAiTokenizer("gpt-4o-mini");
DocumentSplitter splitter = DocumentSplitters.recursive(500, 100, tokenizer);

Without the correct tokenizer, "500 tokens" is an estimate that can be off by enough to matter for context-window budgeting. See tokenization and context windows.

Metadata survives splitting

Each segment inherits the document's metadata, so filtering and citation still work at chunk level:

List<TextSegment> segments = splitter.split(document);
// Each segment carries the parent document's metadata (sourceId, tenantId, ...)
// plus its position, so you can cite "chunk 3 of handbook.pdf".
segments.forEach(s -> System.out.println(s.metadata().getString("sourceId")));

Tuning against a question set

The only reliable way to choose chunk parameters is to measure retrieval quality on real questions:

// For each candidate configuration, ingest, then check: for a fixed set of
// questions, is the correct passage in the top-k results?
for (var config : List.of(
        DocumentSplitters.recursive(300, 60),
        DocumentSplitters.recursive(500, 100),
        DocumentSplitters.recursive(800, 150))) {
 
    reingest(config);
    double hitRate = evaluateRetrieval(goldenQuestions);
    log.info("config={} hitRate={}", config, hitRate);
}

Pick the configuration with the best hit rate on your content and your questions. This measured approach beats any rule of thumb. See testing AI applications.

Next

Frequently Asked Questions

What chunk size should I use for RAG?
Start at 300-600 tokens with 10-20% overlap for prose. Smaller chunks give precise matches but lose surrounding context; larger chunks carry context but dilute the signal that makes a passage retrievable. Code, tables and structured documents usually want larger chunks that keep their structures intact. Measure against a real question set rather than guessing.
What does chunk overlap do?
Overlap repeats a portion of text between adjacent chunks so that an answer spanning a chunk boundary is not cut in half. Around 10-20% of chunk size is the useful range. Too much overlap fills your retrieved results with near-duplicate chunks of the same passage, which crowds out other relevant content.
What is the recursive splitter in LangChain4j?
DocumentSplitters.recursive tries to split on the largest natural boundaries first — paragraphs, then sentences, then words — falling back only as needed to hit the target size. This keeps semantically coherent units together better than a naive fixed-length split, which is why it is the recommended default.
Why is chunking the most important part of RAG?
Because retrieval quality caps the whole system, and chunking determines what can be retrieved. A chunk that splits a definition from its term, or buries the answer in irrelevant surrounding text, cannot be retrieved well no matter how good your embeddings or prompt are. Get chunking right first; everything downstream depends on it.

Related tutorials