Architecture Overview

DocuStore processes scientific research documents through a four-layer pipeline. Each layer is event-driven, independently scalable, and orchestrated by Temporal durable workflows.

The Four Layers

1. Extraction

When a PDF is uploaded, DocuStore splits it into pages and runs multiple extraction pipelines in parallel:

• Text extraction -- OCR and text parsing via PyMuPDF produces the raw text content of each page.
• Chemical structure recognition (OCSR) -- The structflo-cser ML pipeline (YOLO detector + neural matcher + DECIMER) identifies chemical structure diagrams and extracts their SMILES representations.
• Named entity recognition (NER) -- LLM-powered extraction via structflo-ner identifies compounds, targets, genes, diseases, assays, bioactivity values, and other domain-specific entities.
• Document metadata -- GLiNER2 + LLM extraction identifies authors, titles, and publication dates from the first page.

2. Indexing

Extracted data is indexed into three Qdrant vector collections for retrieval:

Text chunks are enriched with document context (title, tags, page summary) before embedding, so that isolated passages carry the semantic context of their parent document.

LLM summarization runs at two granularities: individual pages are summarized first, then a sliding-window chain (batch, synthesize, refine) produces a document-level summary from the page summaries.

3. Retrieval

Search queries flow through a multi-stage pipeline:

1. Query encoding -- The query is transformed into both a dense semantic vector and a sparse term-frequency vector.
2. Hybrid retrieval -- Both representations query Qdrant simultaneously. Results are fused using Reciprocal Rank Fusion (RRF), which combines semantic similarity with exact-term matching.
3. Cross-encoder reranking -- A cross-encoder model (ms-marco-MiniLM-L-12-v2) rescores the top candidates by reading query and passage jointly, detecting subtle relevance signals that embedding similarity alone cannot capture.
4. Enrichment -- Final results are decorated with document titles, page numbers, tags, and compound mentions from MongoDB read models.

Hierarchical search queries both the chunk collection and the summary collection in parallel, then merges results using two-level RRF to produce a unified document ranking.

4. Conversational RAG

The chat system is an agentic RAG pipeline with three operating modes:

All modes share the same grounding verification and answer formatting stages. Tokens stream to the client only during the final formatting stage, ensuring coherent, polished responses. Every claim in the response is verified against retrieved sources with inline [N] citations.

Infrastructure Stack

FastAPI            API layer + SSE streaming
EventStoreDB       Event sourcing (aggregates, CQRS)
MongoDB            Read models, projections, chat history
Qdrant             Vector search (3 collections)
Temporal           Workflow orchestration + retry
Kafka              External event streaming + plugin system
Sentinel           Authorization (workspace-scoped, entity-level ACL)
Langfuse           LLM observability + prompt management

Design Principles

• Event sourcing -- All state changes are recorded as immutable domain events in EventStoreDB. Read models in MongoDB are projections derived from these events.
• Port-based architecture -- The application layer depends on abstract ports (protocols). Infrastructure adapters implement these ports, making every external dependency swappable.
• Idempotent workflows -- Temporal workflow IDs are derived from entity IDs (e.g., embedding-{page_id}), ensuring that duplicate triggers are safely ignored.
• Eventually consistent -- Extraction results propagate through the event pipeline asynchronously. Each stage triggers the next via domain events, with no polling or scheduling.