Architecture Overview

A four-layer pipeline for scientific document intelligence: extraction, indexing, retrieval, and conversational RAG.

DocuStore processes scientific research documents through a four-layer pipeline. Each layer is event-driven, independently scalable, and orchestrated by Temporal durable workflows.

The Four Layers

1. Extraction

When a document (PDF, or a PPTX/DOCX file converted to PDF) is uploaded, DocuStore splits it into pages and runs multiple extraction pipelines in parallel:

Text extraction -- Docling parses the document (OCR, layout, tables, captions) and produces the text content of each page.
Chemical structure recognition (OCSR) -- The structflo-cser ML pipeline (D-FINE detector + RelationalMatcher + DECIMER) identifies chemical structure diagrams, pairs them with their labels, and extracts their SMILES representations.
Named entity recognition (NER) -- LLM-powered extraction via structflo-ner identifies compounds, targets, genes, diseases, assays, bioactivity values, and other domain-specific entities.
Document metadata -- Font-size analysis (PyMuPDF) finds the title; GLiNER2, with an LLM fallback, identifies authors and publication dates, starting from the first page.

2. Indexing

Extracted data is indexed into three Qdrant vector collections for retrieval:

Collection	Content	Embedding Model	Dimensions
`page_embeddings`	Text chunks with contextual headers	Configurable (default granite-embedding-english-r2)	768
`summary_embeddings`	Page + artifact summaries (unified)	Same as `page_embeddings`	768
`compound_embeddings`	Chemical structures as SMILES	ChemBERTa-77M-MTR	384

Text chunks are enriched with document context (title, tags, page summary) before embedding, so that isolated passages carry the semantic context of their parent document.

LLM summarization runs at two granularities: individual pages are summarized first, then a sliding-window chain (batch, synthesize, refine) produces a document-level summary from the page summaries.

3. Retrieval

Search queries flow through a multi-stage pipeline:

Query encoding -- The query is transformed into a dense semantic vector. When sparse encoding is enabled (SPARSE_ENCODING_ENABLED, off by default), a sparse term-frequency vector is produced as well.
Retrieval -- The dense vector queries Qdrant. With sparse encoding enabled, both representations query Qdrant and results are fused using Reciprocal Rank Fusion (RRF), which combines semantic similarity with exact-term matching.
Cross-encoder reranking -- A cross-encoder model (ms-marco-MiniLM-L-12-v2) rescores the top candidates by reading query and passage jointly, detecting subtle relevance signals that embedding similarity alone cannot capture.
Enrichment -- Final results are decorated with document titles, page numbers, tags, and compound mentions from MongoDB read models.

Hierarchical search queries the summary collection and the chunk collection and returns the two result sets separately (summary_hits and chunk_hits). Scores from the two collections are not merged into a single ranking.

4. Conversational RAG

The chat system is an agentic RAG pipeline with four operating modes:

Mode	Description
Quick	Single-shot retrieval, lightweight, fast responses
Thinking	Agentic iterative retrieval with LLM tool-calling loop (up to 5 iterations)
Deep Thinking	Thinking mode plus page image analysis for visual content
Literature	Thinking pipeline over published papers (Europe PMC) instead of the workspace corpus; available only when `LITERATURE_ENABLED=true`

All modes share the same grounding verification and answer formatting stages. Tokens stream to the client only during the final formatting stage, ensuring coherent, polished responses. Every claim in the response is verified against retrieved sources with inline [N] citations.

Infrastructure Stack

FastAPI            API layer + SSE streaming
EventStoreDB       Event sourcing (aggregates, CQRS)
MongoDB            Read models, projections, chat history
Qdrant             Vector search (3 collections)
Temporal           Workflow orchestration + retry
Kafka              External event streaming + plugin system
Duar               Authorization (workspace-scoped, entity-level ACL)
Langfuse           LLM observability + prompt management

Design Principles

Event sourcing -- All state changes are recorded as immutable domain events in EventStoreDB. Read models in MongoDB are projections derived from these events.
Port-based architecture -- The application layer depends on abstract ports (protocols). Infrastructure adapters implement these ports, making every external dependency swappable.
Idempotent workflows -- Temporal workflow IDs are derived from entity IDs (e.g., embedding-{page_id}), ensuring that duplicate triggers are safely ignored.
Eventually consistent -- Extraction results propagate through the event pipeline asynchronously. Each stage triggers the next via domain events, with no polling or scheduling.