rag.core.text_quality

Heuristics to skip figure-dump and boilerplate PDF text.

Attributes

MIN_PROSE_CHARS
MIN_REAL_WORDS
MAX_SALAD_RATIO
MAX_GLUED_RATIO
MAX_SNAKE_RATIO
MAX_SINGLE_LETTER_RATIO
MIN_WORDS_IF_SPARSE
MIN_STUTTER_TOKEN_CHARS

Functions

is_prose_text(→ bool)

Return True when text looks like readable paper prose.

Module Contents

rag.core.text_quality.MIN_PROSE_CHARS: Final = 120

rag.core.text_quality.MIN_REAL_WORDS: Final = 8

rag.core.text_quality.MAX_SALAD_RATIO: Final = 0.1

rag.core.text_quality.MAX_GLUED_RATIO: Final = 0.04

rag.core.text_quality.MAX_SNAKE_RATIO: Final = 0.03

rag.core.text_quality.MAX_SINGLE_LETTER_RATIO: Final = 0.07

rag.core.text_quality.MIN_WORDS_IF_SPARSE: Final = 80

rag.core.text_quality.MIN_STUTTER_TOKEN_CHARS: Final = 8

rag.core.text_quality.is_prose_text(*, text: str) → bool

Return True when text looks like readable paper prose.

Drops short dedications, chart-axis dumps, pyLDAvis chrome, and PDF /uni00 figure encodings that pypdf extracts as glyph salad.

Parameters:

text (str) – Extracted page or chunk text.

Returns:

True when the text is worth indexing or citing.

Return type:

bool