Search Pipeline

Scientific documents contain dense domain-specific terminology -- compound codes like "SACC-111", gene names like "NadD", and quantitative measurements like "IC50=2.3uM" -- that generic search models handle poorly. DocuStore addresses this with a multi-stage search pipeline that combines seven complementary techniques.

Three Vector Collections

DocuStore maintains three separate Qdrant collections, each optimized for a different search modality:

Search Pipeline Stages

Stage 1: Query Encoding

The query text is transformed into a dense vector and, when sparse encoding is enabled, a sparse vector:

• Dense vector -- Captures semantic meaning via the text embedding model, with the model's query prefix where it has one (for example search_query: for nomic).
• Sparse vector (optional) -- Captures exact terms via scikit-learn's HashingVectorizer with a custom tokenizer that preserves scientific identifiers (hyphens, dots) and bigrams. Only produced when SPARSE_ENCODING_ENABLED=true; it is off by default.

Stage 2: Dense or Hybrid Retrieval

By default, retrieval is dense-only: the dense vector queries Qdrant and the results go straight to reranking. With sparse encoding enabled, both vectors query Qdrant simultaneously. Dense retrieval finds semantically similar passages; sparse retrieval guarantees exact-term recall for opaque identifiers. Results are fused server-side using Reciprocal Rank Fusion (RRF):

score = 1 / (k + dense_rank) + 1 / (k + sparse_rank)

Where k is Qdrant's built-in RRF constant (DocuStore does not override it). Results appearing in both channels receive the highest combined scores. The fusion runs entirely within Qdrant using native prefetch-and-fuse, eliminating application-level merge logic.

Stage 3: Cross-Encoder Reranking

The top candidates (approximately 3x the desired result count) are rescored by a cross-encoder model that reads query and passage jointly. This detects relevance signals that embedding similarity alone misses:

• Specificity -- Whether the passage answers the query or just discusses the topic
• Negation -- Whether the passage contradicts the query
• Quantitative relevance -- Whether numeric values match the query intent

The cross-encoder (ms-marco-MiniLM-L-12-v2) rescores 30 passages in under 200ms on CPU.

Stage 4: Enrichment

Final results are decorated with metadata from MongoDB read models: document titles, page numbers, tags, compound mentions, and text previews.

Contextual Chunk Embeddings

Text chunks lose their parent document context during segmentation. DocuStore addresses this by prepending a context header before embedding:

Document: NadD Inhibitor Screening Results | Tags: NadD, SACC-111, tuberculosis, MIC | Page 7 | Summary: This page presents IC50 and MIC data for the SACC series...

[actual chunk text follows]

The header carries the document title, up to 10 NER tags, the page number, and the first 200 characters of the page summary. It affects only the dense embedding -- the raw text is preserved for display, and sparse vectors (when enabled) use the raw text. Chunks are embedded once per document, after every page has its summary: BatchReEmbedArtifactPagesWorkflow builds the headers and encodes all chunks of the artifact in one batch.

Multi-Granularity Search

The hierarchical search endpoint (POST /search/hierarchical) encodes the query once, searches the summary collection, then the chunk collection, and returns the two result sets separately:

• Summary hits (summary_hits) -- Broad thematic retrieval from summary_embeddings, covering both page-level and artifact-level summaries.
• Chunk hits (chunk_hits) -- Fine-grained passage retrieval from page_embeddings, deduplicated to one result per page using Qdrant's native grouping, then reranked by the cross-encoder. Chunk search can be switched off per request with include_chunks: false.

The two lists are not merged into a single ranking. Cosine scores from different vector spaces have different distributions and cannot be compared directly, so the caller (the search UI, or the chat retrieval tools) decides how to use each list.

Metadata Filtering

Every vector carries structured metadata that supports filter-based search:

• Workspace ID -- Automatic multi-tenant isolation
• Tags -- NER-extracted entities (compounds, targets, genes, diseases), case-insensitive
• Tag match mode -- any (at least one tag matches) or all (every tag must match)
• Artifact scoping -- Restrict results to a specific document

Tags are stored in two independent payload fields -- tag_normalized (page-level NER) and artifact_tag_normalized (artifact-level metadata including authors). Filters use an OR condition across both fields so that author name filters match all pages belonging to that author's document.

Chemical Structural Similarity

The compound collection supports a fundamentally different search modality: given a SMILES string, find structurally similar compounds across all documents. ChemBERTa maps molecular structures to vectors where analogues in a medicinal chemistry series cluster together.

curl -X POST http://localhost:8000/search/compounds \
  -H "Content-Type: application/json" \
  -d '{"query_smiles": "CC(=O)Oc1ccccc1C(=O)O", "limit": 10}'

Configuration