🧠 RAG stack

Papers RAG loads PDFs with pypdf, embeds locally, persists Chroma per backend, and generates answers with OpenAI plus citations. LlamaIndex and LangChain are in-process backends over the same corpus; each has its own Chroma directory.

Overlap chunking, a prose filter, and catalog routing (library-wide and authorship questions skip vector search) sit on top of retrieve-then-generate.

Leaf	Path	Role
rag-core	libs/rag-core → rag.core	Config, loaders, backends, ask
papers-ingest	projects/papers-ingest	CLI: PDF → Chroma + catalog.json
papers-rag	apps/papers-rag	Streamlit chat

Docs: rag-core, papers-rag, papers-ingest. Langflow is a separate studio: Langflow.

        flowchart TB
  PDFs["assets/pdf/papers/*.pdf"] --> Load["pypdf + is_prose_text"]
  Load --> Catalog["catalog.json"]
  Load --> Dispatch["papers-ingest --strategy"]
  Dispatch --> LI[".data/indexes/llamaindex/"]
  Dispatch --> LC[".data/indexes/langchain/"]
  LI --> App["papers-rag"]
  LC --> App
  Catalog --> Ask{"ask()"}
  App --> Ask
  Ask -->|corpus / author| Synth["OpenAI on catalog"]
  Ask -->|otherwise| Retrieve["overfetch → prose filter → top_k"]
  Retrieve --> GPT["OpenAI generation"]
  Synth --> UI["answer + citations"]
  GPT --> UI
    

        sequenceDiagram
  participant Dev as Developer
  participant Ingest as papers-ingest
  participant Embed as bge-small
  participant Chroma as Chroma
  participant App as papers-rag
  participant GPT as OpenAI

  Dev->>Ingest: poe ingest-papers --strategy llamaindex
  Ingest->>Embed: embed chunks
  Embed-->>Ingest: vectors
  Ingest->>Chroma: persist + catalog.json
  Dev->>App: poe run-papers-rag
  App->>App: classify question
  alt corpus or author
    App->>GPT: catalog plus question
  else paper
    App->>Chroma: retrieve top_k × 3
    Chroma-->>App: chunks
    App->>App: drop non-prose
    App->>GPT: labeled context plus question
  end
  GPT-->>App: answer
  App-->>Dev: answer plus citations
    

🎛️ Strategies

Strategy	Ingest	Query	Persist
llamaindex	SentenceSplitter → LlamaIndex VectorStoreIndex	Query engine + ProseNodePostprocessor	.data/indexes/llamaindex/
langchain	RecursiveCharacterTextSplitter → Chroma	Retriever → ChatOpenAI	.data/indexes/langchain/

poe ingest-papers --strategy llamaindex
poe ingest-papers --strategy langchain
poe ingest-papers --strategy all

--strategy all cannot take --chroma-dir. CHROMA_DIR overrides persist for one strategy. RAG_STRATEGY defaults to llamaindex; the Streamlit sidebar overrides it for the session.

📦 Code map (rag.core)

Module	Responsibility
strategy	RagStrategy and .data/indexes/{strategy}
config	RagConfig.from_env — paths, models, chunking, top-k
loaders	Non-recursive *.pdf listing; pypdf page extraction
text_quality	is_prose_text (ingest + query)
passage	Author-question cues, acknowledgement / SPIE filters, page labels
catalog	catalog.json + corpus vs paper QueryScope
ingest	Dispatch + LlamaIndex chunk → embed → Chroma
index	Open persistent LlamaIndex collection; IndexMissingError
query	ask() routing, LlamaIndex engine, catalog synthesis
citations	Citation (file_name, page, score, snippet)
backends	RagBackend protocol; LlamaIndex and LangChain

Public API: RagConfig, RagStrategy, ingest_papers, ask, build_query_engine, Citation, QueryResult, IndexMissingError.

📄 PDF loading

Shared loader (rag.core.loaders):

• Directory: PAPERS_DIR or assets/pdf/papers.

• Non-recursive *.pdf, sorted by path.

• Per-page pypdf extraction.

• Pages that fail is_prose_text are skipped (empty pages, dedications, figure dumps, pyLDAvis chrome, PDF /uni00 encodings).

• Metadata on kept pages: file_name, file_path, page (1-based), page_label.

• Image-only scans raise FileNotFoundError.

Rebuild with poe ingest-papers after loader or filter changes, then restart Streamlit (file watching is off).

🧹 Prose filter

is_prose_text runs at ingest and again at query. A page or chunk is kept when all of these hold:

Check	Threshold
Collapsed length	≥ 120 characters
Real words ([A-Za-z]{4,})	≥ 8
PDF /uni00 encodings	rejected
Letter stutter (aaaa in a long token)	rejected
Tokens mixing letters with unusual symbols	< 10% of tokens
Glued TitleCase / camelCase	< 4% of tokens
Snake_case tokens	< 3% of tokens
Single-letter tokens	reject if ≥ 7% and fewer than 80 real words

Query retrieves similarity_top_k × 3 (RETRIEVAL_OVERFETCH = 3), then keeps at most similarity_top_k prose chunks.

✂️ Chunking

Backend	Splitter	Defaults
LlamaIndex	SentenceSplitter	chunk_size=1024, chunk_overlap=128
LangChain	RecursiveCharacterTextSplitter	same sizes

Override with CHUNK_SIZE / CHUNK_OVERLAP. Rebuild the index after changing these.

🧮 Embeddings

LlamaIndex and LangChain both use the local model BAAI/bge-small-en-v1.5 (LlamaIndex: HuggingFaceEmbedding; LangChain: HuggingFaceEmbeddings). First ingest downloads weights from HuggingFace Hub. Override with EMBED_MODEL. Query-time embed must match ingest.

🗄️ Vector store

Chroma PersistentClient under .data/indexes/{strategy} (gitignored). Collection name defaults to papers (CHROMA_COLLECTION). Ingest defaults to --rebuild (delete and recreate). --no-rebuild appends.

Each ingest writes catalog.json next to that strategy’s Chroma dir:

{
  "papers": [
    {"file_name": "example.pdf", "title": "opening-page snippet…"}
  ]
}

Titles are the first extractable page, collapsed, truncated to 220 characters. If catalog.json is missing or invalid, ask() uses filenames from papers_dir.

🧭 Query routing

ask(question=…, config=…) classifies before vector search:

        flowchart TD
  Q[question] --> A{author cues?}
  A -->|yes| Cat[catalog.json → AUTHOR_SYNTHESIS_PROMPT]
  A -->|no| C{corpus cues?}
  C -->|yes| Corp[catalog.json → CORPUS_SYNTHESIS_PROMPT]
  C -->|no| R[vector retrieve → generate]
    

Author questions

Substring match on collapsed lowercase. Bare "author" is not a cue, so “what did the authors conclude?” still retrieves chunks:

• who wrote, who is the writer

• who is the author, who are the authors

• who is the main author, main author, first author

• corresponding author, list the authors, authors of

Answers come from the catalog (opening-page title and author lists), not from SPIE volume-editor footers (edited by … Proc. of SPIE) or acknowledgements.

Corpus questions

• all papers, all the papers, every paper

• entire corpus, the corpus, among all, across all

• common topic among, main topic among, common theme, unifying theme

• body of work, what do these papers, these papers have in common

• theme across, overall research

When files span domains but share ML methods, the prompt names applied machine learning as the umbrella, then lists domains from filenames.

Paper questions

Similarity search. The QA prompt treats excerpts as a retrieved subset: do not say “all papers” or “both papers” unless every filename used is listed.

🔎 Retrieval

1. Embed the question with the same local model.

2. Retrieve top_k × 3 nearest chunks from that strategy’s collection.

3. Drop non-prose chunks.

4. For leftover authorship-shaped questions: drop acknowledgement text and SPIE boilerplate (edited by, proc. of spie, ccc code, spiedigitallibrary, terms of use); sort remaining chunks by earlier page first.

5. Keep at most top_k (sidebar 1–12; env default 5).

6. LlamaIndex uses text_qa_template + refine_template. LangChain labels chunks as [file.pdf, p.N].

7. Both inject QA_GROUNDING_RULES: manuscript thanks are not authors; SPIE “edited by” names are volume editors; prefer title-page names; do not pick one main author for the library from a retrieval subset.

Empty collection raises IndexMissingError.

💬 LLM

Generation uses OpenAI (gpt-4o-mini by default, OPENAI_MODEL). OPENAI_API_KEY is required. Streamlit also reads Cloud secrets.toml. Process env wins over .env (override=False). Streamlit does not source .zshrc; load_repo_dotenv() loads the repo-root .env even when cwd is the app script directory.

📚 Citations

Citation objects render in the Streamlit Sources expander (collapsed):

Field	Retrieval	Catalog (corpus / author)
file_name	chunk metadata	catalog entry
page	1-based when present	None
score	retriever score when present	None
snippet	≤ 280 chars of chunk text	opening-page title snippet

Markdown: 1. **file.pdf**, p.3 (score=0.812) — snippet…

💬 Streamlit

poe run-papers-rag runs Streamlit with --server.fileWatcherType none so HuggingFace vision-module probes do not spam ModuleNotFoundError: torchvision. Restart after code changes.

Control	Meaning
Strategy	LlamaIndex or LangChain; switching clears chat
Similarity top-k	Chunks kept after overfetch + filter (1–12)
Clear chat	Reset session messages
Sources	Citations expander, collapsed

Sidebar shows LLM id, embed model, Chroma path, and whether the index is ready.

⚙️ Environment

Copy .env.example to .env. Do not commit real keys.

Variable	Default	Meaning
OPENAI_API_KEY	(required)	Chat completions
OPENAI_MODEL	gpt-4o-mini	Chat model id
EMBED_MODEL	BAAI/bge-small-en-v1.5	Local HuggingFace embed
RAG_STRATEGY	llamaindex	Default backend
PAPERS_DIR	assets/pdf/papers	PDF directory
CHROMA_DIR	.data/indexes/{strategy}	Persist dir (single strategy)
CHROMA_COLLECTION	papers	Collection name
CHUNK_SIZE	1024	Splitter chunk size
CHUNK_OVERLAP	128	Splitter overlap
SIMILARITY_TOP_K	5	Chunks after filter

CLI: papers-ingest --papers-dir / --chroma-dir / --strategy / --rebuild.

🆓 Cost

Layer	Choice	Cost
Orchestration	LlamaIndex or LangChain	OSS
Embeddings	BAAI/bge-small-en-v1.5 (local)	Free
Vector DB	Chroma (persistent dir)	Free
LLM chat	OpenAI API	Your key
Docs hosting	GitHub Pages	Free

🧯 Troubleshooting

Symptom	Fix
Empty index / IndexMissingError	poe ingest-papers --strategy <same as sidebar>
Missing OPENAI_API_KEY	Repo-root .env or export in the same terminal
ModuleNotFoundError: torchvision	Use poe run-papers-rag (file watcher off)
Chart axes / pyLDAvis / /uni00 in sources	Rebuild after the prose filter; restart Streamlit
Acknowledgements or SPIE editors named as authors	Author routing uses catalog.json; restart Streamlit
One-paper answer to an “all papers” question	Corpus cues + catalog; re-ingest to refresh catalog.json
Garbled UTF-8 snippets	Rebuild the index