rag.core.text_quality
=====================

.. py:module:: rag.core.text_quality

.. autoapi-nested-parse::

   Heuristics to skip figure-dump and boilerplate PDF text.



Attributes
----------

.. autoapisummary::

   rag.core.text_quality.MIN_PROSE_CHARS
   rag.core.text_quality.MIN_REAL_WORDS
   rag.core.text_quality.MAX_SALAD_RATIO
   rag.core.text_quality.MAX_GLUED_RATIO
   rag.core.text_quality.MAX_SNAKE_RATIO
   rag.core.text_quality.MAX_SINGLE_LETTER_RATIO
   rag.core.text_quality.MIN_WORDS_IF_SPARSE
   rag.core.text_quality.MIN_STUTTER_TOKEN_CHARS


Functions
---------

.. autoapisummary::

   rag.core.text_quality.is_prose_text


Module Contents
---------------

.. py:data:: MIN_PROSE_CHARS
   :type:  Final
   :value: 120


.. py:data:: MIN_REAL_WORDS
   :type:  Final
   :value: 8


.. py:data:: MAX_SALAD_RATIO
   :type:  Final
   :value: 0.1


.. py:data:: MAX_GLUED_RATIO
   :type:  Final
   :value: 0.04


.. py:data:: MAX_SNAKE_RATIO
   :type:  Final
   :value: 0.03


.. py:data:: MAX_SINGLE_LETTER_RATIO
   :type:  Final
   :value: 0.07


.. py:data:: MIN_WORDS_IF_SPARSE
   :type:  Final
   :value: 80


.. py:data:: MIN_STUTTER_TOKEN_CHARS
   :type:  Final
   :value: 8


.. py:function:: is_prose_text(*, text: str) -> bool

   Return True when text looks like readable paper prose.

   Drops short dedications, chart-axis dumps, pyLDAvis chrome, and PDF
   ``/uni00`` figure encodings that pypdf extracts as glyph salad.

   :param text: Extracted page or chunk text.
   :type text: str

   :returns: True when the text is worth indexing or citing.
   :rtype: bool