Chunking strategies for RAG over financial documents

Most RAG tutorials chunk with RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200) and move on to the interesting parts. That default is fine for blog posts and quietly terrible for financial documents.

Building an ESG scoring platform that retrieves across sustainability reports for 2,000+ companies, chunking turned out to be the single highest-leverage decision in the pipeline — more than the embedding model, and far more than the LLM.

Why financial documents break naive chunking

Sustainability reports, annual filings and disclosure documents have properties that fixed-size splitting handles badly:

Tables carry the actual answer. Emissions figures, board composition, energy consumption — the numbers live in tables. Split a table at character 1000 and you get half a header row in one chunk and orphaned numbers in another. Neither retrieves usefully, and the second is worse than useless: a chunk of bare numbers with no column context is actively misleading to an LLM, which will happily attribute them to the wrong metric.

Meaning depends on distant context. A chunk reading "we reduced this by 34% year over year" is retrievable and worthless. Reduced what? The subject was in a heading two pages up.

Layout is not reading order. Multi-column layouts, sidebars and footnotes mean naive PDF text extraction interleaves unrelated sentences. Bad chunking here is downstream of bad parsing — no chunk strategy rescues scrambled input.

Boilerplate dominates by volume. Legal disclaimers and repeated headers/footers are a large fraction of the text and near-identical across documents. Left in, they pollute the index and crowd out real content in retrieval.

Structure-aware splitting beats character counts

The shift that mattered: split on document structure first, and use size only as a constraint within a structural unit.

Concretely, the hierarchy is: section → subsection → paragraph, and only fall back to character splitting when a single paragraph exceeds the limit. Sections are the natural retrieval unit because that's how these documents are written and how questions are asked of them.

from langchain_text_splitters import RecursiveCharacterTextSplitter

# Split on document structure first; character count is the last resort.
splitter = RecursiveCharacterTextSplitter(
    separators=["\n## ", "\n### ", "\n\n", "\n", ". ", " "],
    chunk_size=1200,
    chunk_overlap=150,
    length_function=len,
)

The ordering of separators is the whole trick and it's easy to miss: the splitter tries them in order, so it only falls back to splitting mid-sentence when a structural unit genuinely doesn't fit.

Extract tables separately

The highest-value change I made was to stop treating tables as text at all. Extract them during parsing, then serialise each table as its own chunk with headers preserved:

def table_to_chunk(table, company: str, section: str, page: int) -> dict:
    """Serialise a table so every row keeps its column context."""
    headers = table.columns.tolist()
    rows = [
        " | ".join(f"{h}: {row[h]}" for h in headers)
        for _, row in table.iterrows()
    ]
    return {
        "text": f"Table from {section}\n" + "\n".join(rows),
        "metadata": {
            "company": company,
            "section": section,
            "page": page,
            "content_type": "table",
        },
    }

Repeating the header on every row is redundant to read and exactly right for retrieval: each row becomes independently meaningful, so Scope 1 emissions: 12,400 tCO2e retrieves correctly even when separated from its table.

Give every chunk enough context to stand alone

The rule I'd keep: a chunk must make sense in isolation, because that's how it will be read.

Cheapest effective version — prepend the breadcrumb to the chunk text itself, not just the metadata:

chunk_text = f"{company} — {section} > {subsection}\n\n{raw_chunk}"

That "34% reduction" chunk becomes Acme Corp — Environmental Performance > Emissions followed by the text. Now it's retrievable by an embedding and interpretable by the model.

Metadata alongside is what makes filtering possible, and filtering matters enormously at multi-company scale: a question about one company should never retrieve another's chunks. Pre-filtering on company before vector search does more for precision than any reranking.

{
    "text": chunk_text,
    "metadata": {
        "company": "Acme Corp",
        "fiscal_year": 2025,
        "section": "Environmental Performance",
        "content_type": "narrative",   # or "table"
        "source_page": 47,
    },
}

Strip boilerplate before indexing, not after

Repeated headers, footers and legal disclaimers should be removed at parse time. A practical heuristic that works well across a large corpus: any line appearing on more than ~70% of pages within a document is chrome, not content.

This is unglamorous and pays off immediately — the index shrinks, embedding costs drop, and retrieval stops surfacing disclaimers.

Evaluate chunking directly, not through the LLM

The mistake I'd warn against: judging chunking by looking at final answers. Too many variables. When output is wrong you can't tell whether retrieval missed the chunk, or the chunk was retrieved and the model misread it.

Measure retrieval on its own. Build a modest set of question → known-correct-source-passage pairs and track recall@k: how often the right chunk appears in the top k. A few dozen hand-labelled examples are enough to make chunking decisions empirical rather than aesthetic.

If recall@10 is poor, no amount of prompt engineering saves you — the model never sees the answer.

What I'd do first on a new corpus

1. Fix parsing before touching chunking. Scrambled extraction can't be recovered downstream.
2. Route tables through a separate path from narrative text.
3. Split on structure; treat size as a constraint, not the strategy.
4. Prepend breadcrumbs so chunks stand alone, and attach metadata for filtering.
5. Build a small recall@k evaluation set before tuning anything.

Chunking is unglamorous compared to model selection, and on documents like these it's where retrieval quality is actually won or lost.

Related: building the RAG pipeline that ships to Kubernetes.