Skip to content
JavaAgentic

Type at least two characters. Try “RAG”, “pgvector” or “tool calling”.

LangChain4j Document Loaders

Load documents into LangChain4j from files, URLs, S3, GitHub and more, and parse PDF, DOCX and HTML with Apache Tika — the ingestion front-end for any RAG pipeline in Java.

Intermediate3 min readUpdated
On this page

Ingestion begins with getting documents into Document objects. LangChain4j separates loading (fetching bytes from a source) from parsing (turning bytes into text), which lets you combine any source with any format. This tutorial covers the common loaders and the metadata discipline that makes retrieval work later.

Key Takeaways

  • A loader fetches bytes; a parser extracts text. Combine them freely.
  • Apache Tika parses PDF, DOCX, HTML and dozens of formats through one parser.
  • Load from filesystem, URL, S3, GitHub and more, each a separate module.
  • Attach metadata at load time — it is the only thing you can filter or cite on later.

Loading from the filesystem

Load and parse a directory
// Apache Tika handles almost every format, so it is the sensible default parser.
DocumentParser parser = new ApacheTikaDocumentParser();
 
// A single file:
Document doc = FileSystemDocumentLoader.loadDocument(
        Path.of("handbook.pdf"), parser);
 
// A whole directory (recursively), matching a glob:
List<Document> docs = FileSystemDocumentLoader.loadDocumentsRecursively(
        Path.of("./knowledge-base"), parser);

Parsers

The parser turns bytes into text. Tika is the generalist; specialised parsers exist for control.

// Apache Tika: PDF, DOCX, PPTX, XLSX, HTML, ODT, RTF and many more.
DocumentParser tika = new ApacheTikaDocumentParser();
 
// Plain text, when you know the format and want no dependency overhead.
DocumentParser text = new TextDocumentParser();

Loading from remote sources

Each source is a separate module you add as needed.

Amazon S3
DocumentSource source = AmazonS3DocumentLoader.builder()
        .region("us-east-1")
        .build()
        .loadDocument("my-bucket", "docs/handbook.pdf", parser);
GitHub
Document readme = GitHubDocumentLoader.builder()
        .gitHubToken(System.getenv("GITHUB_TOKEN"))
        .build()
        .loadDocument("owner", "repo", "main", "README.md", parser);
A URL
Document page = UrlDocumentLoader.load(
        "https://example.com/article", new TextDocumentParser());

Attaching metadata — do not skip this

Metadata is what makes filtered retrieval and citations possible later. It must be added at load time, because it cannot be reconstructed afterwards.

Enriching documents with metadata
public Document loadWithMetadata(Path path, String tenantId) {
    Document doc = FileSystemDocumentLoader.loadDocument(path, new ApacheTikaDocumentParser());
 
    // Everything you might later filter on or cite must be recorded now.
    doc.metadata()
            .put("sourceId", path.getFileName().toString())
            .put("tenantId", tenantId)
            .put("loadedAt", Instant.now().toString())
            .put("path", path.toString());
 
    return doc;
}

From loading to the full pipeline

Loading is the first stage. The EmbeddingStoreIngestor chains loading, splitting, embedding and storing:

The full ingestion pipeline
List<Document> documents = FileSystemDocumentLoader.loadDocumentsRecursively(
        Path.of("./knowledge-base"), new ApacheTikaDocumentParser());
 
EmbeddingStoreIngestor.builder()
        .documentSplitter(DocumentSplitters.recursive(500, 100))
        .embeddingModel(embeddingModel)
        .embeddingStore(embeddingStore)
        .build()
        .ingest(documents);

Splitting is covered in LangChain4j text splitters, embedding in embedding models, and storage in embedding stores.

Handling large ingestions

For big corpora, load and ingest in batches rather than holding everything in memory:

try (Stream<Path> paths = Files.walk(Path.of("./knowledge-base"))) {
    paths.filter(Files::isRegularFile)
            .collect(batchesOf(50))          // your batching helper
            .forEach(batch -> {
                List<Document> docs = batch.stream()
                        .map(p -> loadWithMetadata(p, tenantId))
                        .toList();
                ingestor.ingest(docs);       // embed and store this batch
            });
}

Making ingestion idempotent

Re-running an import must not double your chunks. Delete a source's existing content before re-adding it, keyed by the metadata you attached:

embeddingStore.removeAll(metadataKey("sourceId").isEqualTo(sourceId));
ingestor.ingest(freshDocuments);

Duplicate chunks quietly wreck retrieval — the same passage occupies several of your top results and crowds out the answer.

Next

Frequently Asked Questions

How do I load a PDF into LangChain4j?
Use a document loader with a parser. FileSystemDocumentLoader.loadDocument(path, new ApacheTikaDocumentParser()) reads a PDF and extracts its text into a Document. Apache Tika handles PDF, DOCX, PPTX, HTML and dozens of other formats through one parser, which is why it is the usual default.
What is the difference between a loader and a parser in LangChain4j?
A loader fetches raw bytes from a source — the filesystem, a URL, S3, GitHub. A parser turns those bytes into text, handling the file format. You combine them: a loader gets the file, a parser extracts its content. This separation lets you load from anywhere and parse any format independently.
Can LangChain4j load documents from S3 or GitHub?
Yes. LangChain4j provides loaders for Amazon S3, Azure Blob Storage, Google Cloud Storage, GitHub and URLs, each as a separate module. They fetch the raw documents; you pair them with a parser such as Apache Tika to extract text.
Should I attach metadata when loading documents?
Yes, always. Metadata such as source ID, title, URL, tenant and version is the only thing you can filter or cite on later, and it cannot be recovered after ingestion. Attach it at load time, err on the side of recording more rather than less.

Related tutorials