HTML, DOCX and slides fail by yielding too much, not too little

A PDF withholds structure by design: its content stream stores glyph positions and no notion of a column, its text layer guarantees characters and not their order, and a grid printed on one of its pages survives extraction as whitespace. Point the same ingestion at a documentation site, a Word contract or a quarterly deck and the problem inverts. HTML carries headings, list structure and real table cells. A .docx carries named paragraph styles, so a heading is genuinely marked as a heading rather than inferred from a font size. A .pptx carries titles, bullet levels and speaker notes. The structure is present and faithful, and none of it is what goes wrong.

What goes wrong is everything that arrives with it. A web page is not an article; it is an article inside a template. A contract under revision is not a clause list; it is a clause list plus every version of every clause somebody edited, plus what the reviewers said about them. A deck is not a document; it is a bag of shapes with a drawing order and a second body of text nobody displays. Each of these is legitimate content of the file, indistinguishable from the part you wanted by any rule about markup, and each of them reaches an index unless something is written to stop it.

Boilerplate is a retrieval problem before it is a tidiness problem#

Here is a chunk that a support assistant retrieved for the question "how long do I have to request a refund":

Skip to main content Products Pricing Docs Company Log in Get started We use
cookies and similar technologies to improve your experience, analyse traffic
and personalise content. Accept all Reject non-essential Manage preferences
Refunds Home / Support / Billing / Refunds On this page Overview Eligibility
How to request Processing times Related articles Refunds — Overview Customers
on monthly plans may request a full refund within 14 days of a charge.
Requests after 14 days are handled at the discretion of ... Was this page
helpful? Yes No Still need help? Contact support © 2024 Northwind. Privacy
Terms Cookies Status

The answer is in there — fourteen days, monthly plans. It occupies 21 of the chunk's 132 words. The other 111 are the same navigation, cookie notice, breadcrumb and footer that appear on all 3,400 pages of the site, which means this chunk is 84% identical to 3,399 others in the index.

Those numbers close on each other, and it is worth seeing that they do. The answer's 21 words and the template's 111 add to 132 with nothing left over, which means the chunk has exactly two parts and no third category to argue about. The template's share of it is 111 in 132, or 84%; the answer's share is 21 in 132, or the 16% that is left. The comparison set is 3,399 rather than 3,400 because the page being compared is itself one of the 3,400. So the 84% is not a rough estimate of overlap, it is a partition of one chunk into a part every page has and a part only this page has.

That is the inversion. Four chapters of PDF have been about information the format never carried; HTML, DOCX and PPTX carry a great deal of structure faithfully, and the failures come from what arrives alongside it.

Boilerplate is the part of a page that is the same on every page of the site — the navigation, the cookie notice, the breadcrumb trail, the feedback widget, the footer — and it is present in the markup exactly as legitimately as the article text is. Three separate things go wrong when it enters a chunk, and only the third is obvious.

The pooled embedding of the chunk above is dominated by tokens about navigation, cookies and support links, so it sits closer to the centroid of the whole site — the average of every chunk vector the site produced — than to any specific topic. Measured on that documentation corpus, stripping boilerplate raised recall@5, the share of questions whose answer appears in the top five retrieved chunks, on a 90-question eval from 0.62 to 0.79, with no change to the embedder, the chunker or k. The second effect is that thousands of near-identical chunks compete with each other for the same top-k slots, so a query about billing can return five chunks from five different pages that agree on 84% of their content and disagree on the 16% that matters, which wastes the retrieval budget on redundancy. The third is token cost, and it is the least important of the three: at 111 wasted words per chunk and five chunks per query, roughly 740 tokens of every request are footer.

The first of those three is the one people accept without being able to reconstruct, so it is worth doing slowly. An embedding of a chunk is pooled from the chunk's tokens, which means the vector it produces can be read as a shared part contributed by text every page has plus a distinctive part contributed by text only this page has. Every chunk on the site carries the same shared part, because every page carries the same template. When a query vector is compared against two candidate chunks by inner product, the shared part contributes the same amount to both scores, so on its own it cannot change which of the two ranks higher — it cancels out of the difference. What it does instead is take up room. Retrieval compares direction rather than magnitude, so each vector is scaled to unit length before scoring, and scaling divides the distinctive part by a length the shared part helped inflate. The distinctive part therefore controls a smaller fraction of a fixed score budget than it would have on its own. That is the mechanism: not that boilerplate points the vector somewhere wrong, but that it compresses the range within which the useful signal has to do its discriminating.

That also explains the second effect, which otherwise reads as a separate complaint. If most of every vector is shared, then most of every pairwise similarity between chunks is shared too, and chunks from unrelated pages sit close together. A top-five list drawn from a set whose members are all near each other is a list of near-duplicates, and five slots spent on five copies of one template is four slots that returned nothing.

Boilerplate removal is a solved research problem with a twenty-year history in web search, and the tools are mature. trafilatura, Mozilla's readability, and resiliparse all implement variations of the same idea, scoring subtrees of the DOM — the document object model, which is the tree of nested elements a parser builds from the markup — by text density and link density, and keeping the dominant content block.

import trafilatura

raw = open("refunds.html", encoding="utf-8").read()
main = trafilatura.extract(
    raw,
    include_tables=True,
    include_links=False,
    favor_recall=True,
    output_format="markdown",
)
print(main[:200])

## Refunds

### Overview

Customers on monthly plans may request a full refund within 14 days of a
charge. Requests after 14 days are handled at the discretion of ...

Text density and link density are the right two signals for a reason worth stating. Navigation is a list of links with two or three words between the tags, so its ratio of link text to total text is close to one. Body prose is mostly sentences with the occasional link, so its ratio is near zero. Those two distributions barely overlap, which is what makes a threshold on the ratio work at all — and it is also the reason the method degrades on the page types named later in this chapter, where the body itself is a dense table of short cells and starts to look, by density alone, like a navigation block.

Generic extractors are the right default for a crawl of the open web. For a RAG corpus they usually are not, because a RAG corpus is typically a handful of known sources rather than the whole internet, and for a known source you can name the container instead of inferring it.

from selectolax.parser import HTMLParser

SOURCE_RULES = {
    "docs.northwind.example": {"keep": "main article", "drop": ["nav", ".toc", ".feedback"]},
    "help.northwind.example": {"keep": "#kb-body",  "drop": [".related", ".breadcrumb"]},
}

def extract(html: str, host: str) -> str:
    rules = SOURCE_RULES[host]
    tree = HTMLParser(html)
    for sel in rules["drop"]:
        for node in tree.css(sel):
            node.decompose()
    root = tree.css_first(rules["keep"])
    return root.text(separator="\n", strip=True) if root else ""

A selector rule is deterministic, reviewable, and fails loudly when the site is redesigned — css_first returns nothing and the document extracts empty, which the character-per-page check from chapter 2 catches on the next run. A density heuristic fails quietly and differently on every page. The maintenance cost is real: roughly ten minutes per source to write, and an occasional afternoon when a source redesigns.

Choosing the selector is a design decision and not something the requirement forces. The requirement is that the chunk contain the article and not the template, and the density heuristic meets it more often than a stale selector does. The selector is chosen on a different axis: what happens when it is wrong. A selector that stops matching produces zero characters, which is a value no downstream check can mistake for success. A density score that drifts produces plausible text of the wrong extent, which every downstream check accepts. Preferring the failure you can see over the failure you cannot is the whole of the argument, and it is worth separating from a claim about which one is more accurate on a good day.

HTML has one silent hole to balance the noise, and it is the mirror image of a scanned PDF. A page rendered client-side returns a shell with an empty <div id="root"> and a bundle reference, so the fetch succeeds, the status is 200, and the extraction is 60 characters of <noscript> text. Tabbed reference pages and accordion sections are a subtler version: the markup is present but only the first panel is, with the remaining panels fetched on click. The detection is the same character-count check applied to fetched pages, and the remedy is a headless browser fetch — loading the page in a real browser engine running without a visible window, so its JavaScript executes and the content exists before you read it — for the sources that need it, which costs roughly 400ms to 2s per page against 40ms for a plain request — enough of a difference to be worth deciding per source rather than globally.

The status code is where a competent reader is most likely to believe something false here, so it is worth being explicit. A 200 does not mean the response contains the page; it means the server answered the request it was given. For a client-rendered site the server's honest answer is an empty shell, and it would return 200 for that shell whether the article behind it is a thousand words or has been deleted. A check on the status code therefore gives the same verdict in both of the cases it would need to tell apart, which makes it useless as evidence about content — and the same is true of the absence of an exception, since nothing exceptional occurred. The character count is the cheapest check that actually varies with the thing being tested, which is why it catches both this failure and the redesigned-selector failure above with one instrument.

The compensation for all this is that HTML makes chapter 4's table repair almost free. A <table> carries <th> cells, colspan and scope attributes, so the fully-qualified column labels that had to be reconstructed from geometry in a PDF can be read directly out of the markup. Any source that publishes its tables as HTML is a source whose tables you should be extracting from HTML rather than from the PDF rendering of the same content.

Two things DOCX carries that you did not ask for#

A .docx file is a zip archive of XML parts, and looking at the archive listing is the fastest way to understand both what you get and what you are missing.

$ unzip -l msa-northwind-v7.docx
   Length      Name
   -------     ----
    248910     word/document.xml
     19442     word/comments.xml
      8104     word/styles.xml
    412887     word/media/image3.png
     71204     word/embeddings/oleObject1.bin
      3106     word/footnotes.xml

document.xml holds paragraphs tagged with named styles, so headings are genuinely headings and there is no guessing from font size. Table cells are <w:tc> elements with real grid spans. That is the good half.

The presence of comments.xml is the first trap. Reviewer remarks are document content as far as any naive extractor is concerned, and a comment reading "not sure this is still accurate, check with Legal before we send" will be indexed, retrieved and quoted back to a customer as though it were the contract.

The second trap is tracked changes, and it is worse because the text looks legitimate. When a clause is edited with revision marking on, the old text does not leave the file; it is wrapped in <w:del> with its characters in <w:delText> elements, and the replacement sits in <w:ins>.

<w:p>
  <w:r><w:t xml:space="preserve">Service availability shall be at least </w:t></w:r>
  <w:del w:id="41" w:author="R. Kaur" w:date="2024-03-11T09:12:00Z">
    <w:r><w:delText>99.5%</w:delText></w:r>
  </w:del>
  <w:ins w:id="42" w:author="R. Kaur" w:date="2024-03-11T09:12:00Z">
    <w:r><w:t>99.9%</w:t></w:r>
  </w:ins>
  <w:r><w:t xml:space="preserve"> measured monthly.</w:t></w:r>
</w:p>

Which of 99.5% and 99.9% reaches your index depends entirely on which extractor you used, and the variation between them is not documented anywhere a user of the library would look. Stripping tags with a regular expression over document.xml — a thing people do, because it is four lines and appears to work — yields both figures in sequence, producing "at least 99.5%99.9% measured monthly". Some converters emit the deleted text and drop the insertion, which indexes the superseded number as current. Others walk only the direct run children of each paragraph and silently drop insertions, since runs inside <w:ins> are one level deeper in the tree.

That third behaviour is the one to be able to derive rather than memorise, because it explains why libraries disagree without any of them containing a bug. A paragraph's text is normally the concatenation of the <w:r> runs that are its direct children. In the fragment above there are two such runs: the one ending "at least " and the one beginning " measured monthly". The replacement text is a run too, but it is a child of <w:ins>, which is itself the child of the paragraph — one level further down. A library that collects direct children only is doing exactly what its own documentation says and will never see the insertion. A library that searches the whole subtree sees both the insertion and the deletion, and then has to choose, and its choice is what nobody wrote down. Neither is wrong about XML; they disagree about what a paragraph is.

The reliable move is to stop guessing and normalise before extraction, by having the document's revisions accepted programmatically.

# accept all revisions, then extract from the flattened copy
soffice --headless --convert-to docx:"MS Word 2007 XML" \
        --outdir ./flat/ msa-northwind-v7.docx

Normalising rather than teaching the extractor about <w:ins> is again a decision rather than a requirement. The requirement is that the index hold the current text of every clause. Accepting revisions meets it by removing the ambiguity from the file, so that every extractor downstream — including the four-line regular expression — now agrees, whichever one somebody reaches for next year. Handling <w:ins> in your own extraction code meets the same requirement, and leaves the ambiguity in place for every other tool that ever opens the file.

Where that is not available, the minimum is a check that refuses to ingest silently:

from docx import Document
from docx.oxml.ns import qn

def revision_report(path: str) -> dict[str, int]:
    doc = Document(path)
    body = doc.element.body
    return {
        "deletions": len(body.findall(f".//{qn('w:del')}")),
        "insertions": len(body.findall(f".//{qn('w:ins')}")),
        "comment_refs": len(body.findall(f".//{qn('w:commentReference')}")),
        "embedded_objects": len(body.findall(f".//{qn('o:OLEObject')}")),
    }

{'deletions': 23, 'insertions': 21, 'comment_refs': 7, 'embedded_objects': 2}

Twenty-three deletions means twenty-three places where the ingested text may be the old version of a clause, and it means this file should not have been in the corpus in this state at all. The embedded_objects count points at the other DOCX failure, which is silent rather than noisy: an OLE-embedded spreadsheet — OLE being the Windows mechanism that lets one application's document live inside another's file — or a pasted screenshot of a rate table contributes zero characters to the extraction. The document extracts to 8,400 clean words and the two artefacts everyone actually asks about are a .bin and a .png that no text extractor will ever open.

Slides put the substance where you are not looking#

A .pptx gives you shapes with text frames, and the order those shapes appear in is z-order — the order they were added to the slide — which has no relationship to how a human reads them.

from pptx import Presentation

prs = Presentation("qbr-fy24.pptx")
slide = prs.slides[11]

for i, shape in enumerate(slide.shapes):
    kind = shape.shape_type
    text = shape.text_frame.text[:34].replace("\n", " ") if shape.has_text_frame else ""
    print(f"{i}  top={shape.top//12700:4d} left={shape.left//12700:4d}  {kind}  {text!r}")

notes = slide.notes_slide.notes_text_frame.text if slide.has_notes_slide else ""
print("NOTES:", notes[:120])

0  top= 452 left=  60  TEXT_BOX      'Churn drivers'
1  top= 318 left= 396  TEXT_BOX      'Q4 actions'
2  top= 318 left=  60  TEXT_BOX      'Q3 findings'
3  top= 168 left=  60  PICTURE       ''
4  top=  40 left=  60  TEXT_BOX      'FY24 retention review'
NOTES: Three accounts drove 61% of the churn: Vantis (migrated to a
competitor in Aug), Orlex (acquired), and Byrne Foods (price)...

Reading shapes in file order gives "Churn drivers / Q4 actions / Q3 findings / FY24 retention review" — the title last, the two-column body reversed. Sorting by top then left fixes this deck and reintroduces exactly the interleaving problem from chapter 1 on any slide with genuine columns, so the ordering has to be column-aware there too. The picture at index 3 contributes nothing; on this deck it is a chart with its numbers baked into the image. Across a 60-deck sample from the same team, 34% of slides had at least one shape whose only content was text inside a raster image, and rasterising those slides and running OCR over them was the only way to recover it.

The coordinates are worth checking rather than taking on trust, because doing so produces the chapter's one genuine trap. Sorting the five shapes by top ascending puts the title at the smallest top first, then the picture, then the two shapes that share a top, and left breaks that tie with the left-hand one first. The resulting sequence is the file's own listing read backwards, end to start, exactly. Nothing about slide software makes that true in general; on this deck the author happened to build from the bottom up and add the title last. A reader who notices the reversal and writes "reverse the shape list" has a rule that is right on every slide of this deck and has no reason to be right on any other, because it produces one fixed permutation while the thing it is meant to correct — each author's build order — is different in every file. It is a defence that cannot vary with what it is defending against, and it will pass every test written against the deck it was derived from.

The notes field is the part that surprises people. The slide says "Churn drivers"; the note names three accounts, a percentage and three causes. Speaker notes routinely carry more retrievable substance than the slide body, and most slide extractors ignore them by default. They also carry things nobody intended to publish, which is a reason to route them deliberately rather than to switch them on globally.

The row worth reading across is "Cheap detection", because all three entries count something other than characters. That is not a stylistic preference. Each format's silent failure removes content that was never going to contribute characters in the first place — a rendered page's article, an OLE object's cells, a picture's baked-in text — so a character total has nothing to fall from and reports the same healthy figure either way. A count of dropped nodes, revision marks or picture shapes is a count of the opportunities for silent loss, and it moves when the loss does.

The failure mode of the fix#

Boilerplate removal is subtractive, and subtractive steps have their own defects. trafilatura with default settings drops tables it judges peripheral, drops <pre> blocks on some layouts, and on short pages occasionally returns nothing at all. A stripper tuned for a news site applied to an API reference removes the parameter tables that are the entire point of the page. The check that keeps this honest is a retention ratio logged per source — extracted characters divided by raw text characters — with the distribution watched rather than a single threshold enforced. On the documentation corpus the ratio sat between 0.28 and 0.44 for ordinary pages; the pages that came back below 0.10 were reference tables the extractor had discarded, and finding them took one sorted list.

The reason a sorted list was enough, rather than a threshold somebody had to argue about, is the gap between those figures. Ordinary pages occupy a band, the failures sit well below its floor, and nothing was reported in between. A distribution with an empty middle needs no cut-off chosen in advance: the outliers are visible as outliers, and whoever reads the list can see where the population stops. Where that gap closes — a source whose pages vary widely in how much template they carry — the same instrument stops separating anything, and the honest response is to log the ratio per source rather than per corpus, so that each source is compared against its own band.

The failure it catches is also the predictable consequence of the density argument made earlier. A reference table is short cells, many of them, with little prose between — which is to say it has low text density and, if the cells link to other pages, high link density. Those are precisely the two signals a boilerplate scorer reads as navigation. The extractor is not malfunctioning on those pages; it is applying a rule that was calibrated on documents where the body is prose, to a document where the body is a grid.

The common thread across all three formats is that the extraction is not what fails. It succeeds, on a file that contains more than the document, and it hands over the surplus with the same confidence it hands over the article — a cookie banner with the refund policy, a struck-out availability figure with the current one, a title read last because it was drawn last. That is why the detections in the table above count nodes, revision marks and picture shapes rather than characters: a check that measures how much text came out cannot see a failure whose signature is text that should not have come out, or content that was never text to begin with.

Everything so far has been about what happens at ingestion. The other direction is more common in practice: a specific question comes back with the wrong answer, or no answer, months after ingestion, and the parser is one of several suspects. Working backwards from a bad result to the parsed artefact is a different skill, and it starts with a test that settles whether retrieval was ever involved.