🧭 Overview

rag.core is the shared Papers RAG library. Apps and ingest CLIs import it; libs/ does not import apps/ or projects/.

Pipeline: pypdf pages → overlap chunks → local bge-small embeddings → Chroma → OpenAI. A prose filter and catalog routing (corpus / author questions skip vector search) sit on that path.

See RAG stack for cue lists, prompts, and the env table.

📦 Modules

Module

Responsibility

strategy

RagStrategy (llamaindex / langchain) + .data/indexes/{strategy}

config

RagConfig.from_env — paths, models, chunking, top-k

loaders

Non-recursive *.pdf; pypdf page extraction + prose skip

text_quality

is_prose_text (ingest and query)

passage

Author cues, acknowledgement / SPIE filters, page labels, QA_GROUNDING_RULES

catalog

catalog.json + QueryScope (paper vs corpus)

ingest

Dispatch + LlamaIndex chunk → embed → Chroma

index

Open persistent LlamaIndex collection; IndexMissingError

query

ask() routing, LlamaIndex engine, catalog synthesis

citations

Citation (file_name, page, score, snippet)

backends

RagBackend protocol; LlamaIndex and LangChain

Public API: RagConfig, RagStrategy, ingest_papers, ask, build_query_engine, Citation, QueryResult, IndexMissingError.

🎛️ Strategies

Do not share one Chroma folder across backends.

Strategy

Splitter

Query path

Default persist

llamaindex

SentenceSplitter

Query engine + ProseNodePostprocessor

.data/indexes/llamaindex/

langchain

RecursiveCharacterTextSplitter

Retriever → labeled context → ChatOpenAI

.data/indexes/langchain/

Shared: pypdf loader, is_prose_text, chunk_size=1024 / overlap=128, BAAI/bge-small-en-v1.5, collection name papers, overfetch top_k × 3.

CHROMA_DIR overrides persist for one strategy. --strategy all cannot take --chroma-dir.

🧭 ask() routing

  1. Author cues (who wrote, main author, authors of, … — not bare "author") → catalog.json + AUTHOR_SYNTHESIS_PROMPT.

  2. Corpus cues (all papers, common theme, …) → catalog + CORPUS_SYNTHESIS_PROMPT.

  3. Otherwise → vector retrieve, drop non-prose, generate with QA_GROUNDING_RULES.

Ingest writes catalog.json beside each strategy’s Chroma dir (filename + 220-char opening snippet). Missing JSON falls back to papers_dir filenames.

⚙️ Environment

Variable

Default

OPENAI_API_KEY

required for chat

OPENAI_MODEL

gpt-4o-mini

EMBED_MODEL

BAAI/bge-small-en-v1.5

RAG_STRATEGY

llamaindex

PAPERS_DIR

assets/pdf/papers

CHROMA_DIR

.data/indexes/{strategy}

CHROMA_COLLECTION

papers

CHUNK_SIZE / CHUNK_OVERLAP

1024 / 128

SIMILARITY_TOP_K

5

load_repo_dotenv() loads cwd .env then repo-root .env. Process env wins (override=False).

🆓 Defaults

  • Local embeddings — no embedding API bill.

  • Chroma on disk — no Docker for the local loop.

  • OpenAI only for generation.

  • Isolated strategy dirs so LlamaIndex and LangChain collections stay separate.