Search Pipeline

Scientific documents contain dense domain-specific terminology -- compound codes like "SACC-111", gene names like "NadD", and quantitative measurements like "IC50=2.3uM" -- that generic search models handle poorly. DocuStore addresses this with a multi-stage search pipeline that combines seven complementary techniques.

Three Vector Collections

DocuStore maintains three separate Qdrant collections, each optimized for a different search modality:

Search Pipeline Stages

Stage 1: Query Encoding

The query text is transformed into two representations:

• Dense vector -- Captures semantic meaning via nomic-embed-text-v1.5 with the search_query: prefix for asymmetric retrieval.
• Sparse vector -- Captures exact terms via scikit-learn's HashingVectorizer with a custom tokenizer that preserves scientific identifiers (hyphens, dots) and bigrams.

Stage 2: Hybrid Retrieval

Both vectors query Qdrant simultaneously. Dense retrieval finds semantically similar passages; sparse retrieval guarantees exact-term recall for opaque identifiers. Results are fused server-side using Reciprocal Rank Fusion (RRF):

score = 1 / (k + dense_rank) + 1 / (k + sparse_rank)

Where k = 60. Results appearing in both channels receive the highest combined scores. The fusion runs entirely within Qdrant using native prefetch-and-fuse, eliminating application-level merge logic.

Stage 3: Cross-Encoder Reranking

The top candidates (approximately 3x the desired result count) are rescored by a cross-encoder model that reads query and passage jointly. This detects relevance signals that embedding similarity alone misses:

• Specificity -- Whether the passage answers the query or just discusses the topic
• Negation -- Whether the passage contradicts the query
• Quantitative relevance -- Whether numeric values match the query intent

The cross-encoder (ms-marco-MiniLM-L-12-v2) rescores 30 passages in under 200ms on CPU.

Stage 4: Enrichment

Final results are decorated with metadata from MongoDB read models: document titles, page numbers, tags, compound mentions, and text previews.

Contextual Chunk Embeddings

Text chunks lose their parent document context during segmentation. DocuStore addresses this by prepending a context header before embedding:

Document: NadD Inhibitor Screening Results
Tags: NadD, SACC-111, tuberculosis, MIC
Page: 7 of 12
Summary: This page presents IC50 and MIC data for the SACC series...

[actual chunk text follows]

This context header affects only the embedding computation -- the raw text is preserved for display. Context enrichment happens in two passes: an initial embedding immediately after text extraction, then a re-embedding when page summaries become available.

Multi-Granularity Search

The hierarchical search endpoint queries both collections in parallel:

• Chunk hits -- Fine-grained passage retrieval from page_embeddings, deduplicated to one result per page using Qdrant's native grouping.
• Summary hits -- Broad thematic retrieval from summary_embeddings, covering both page-level and artifact-level summaries.

Results are merged using two-level RRF across collections:

doc_score = chunk_weight / (k + best_chunk_rank)
          + summary_weight / (k + best_summary_rank)

This avoids the problem of naively comparing cosine scores from different vector spaces, which have fundamentally different distributions.

Metadata Filtering

Every vector carries structured metadata that supports filter-based search:

• Workspace ID -- Automatic multi-tenant isolation
• Tags -- NER-extracted entities (compounds, targets, genes, diseases), case-insensitive
• Tag match mode -- any (at least one tag matches) or all (every tag must match)
• Artifact scoping -- Restrict results to a specific document

Tags are stored in two independent payload fields -- tag_normalized (page-level NER) and artifact_tag_normalized (artifact-level metadata including authors). Filters use an OR condition across both fields so that author name filters match all pages belonging to that author's document.

Chemical Structural Similarity

The compound collection supports a fundamentally different search modality: given a SMILES string, find structurally similar compounds across all documents. ChemBERTa maps molecular structures to vectors where analogues in a medicinal chemistry series cluster together.

curl -X POST http://localhost:8000/search/compounds \
  -H "Content-Type: application/json" \
  -d '{"query_smiles": "CC(=O)Oc1ccccc1C(=O)O", "limit": 10}'

Configuration