DocuStore.io

Multi-Representation Indexing

Three vector collections capture text semantics, chemical structure, and document summaries.

/ The problem

A single embedding model cannot capture the full vocabulary of pharmaceutical research. General-purpose dense embeddings fail on compound identifiers, IUPAC names, and scientific abbreviations. Without sparse indexing, exact identifier matches are lost in the noise of semantic similarity.

/ How DocuStore solves it

DocuStore indexes every document across three Qdrant vector collections, each optimized for a different retrieval pattern. Text chunks get both dense embeddings (nomic-embed-text-v1.5, 768-dim) and sparse term-frequency vectors with custom tokenization that preserves scientific identifiers. Chemical structures are embedded separately via ChemBERTa (384-dim) for structure-similarity search. Page and artifact summaries live in a unified summary collection for hierarchical retrieval.

dense · 768dsparse · termsBMS-986158chem · ChemBERTa

/ Pipeline

text_embeddings
Dense (768-dim) + Sparse vectors per text chunk
Payload: page_id, artifact_id, entities, tags
compound_embeddings
ChemBERTa (384-dim) per SMILES string
Payload: canonical_smiles, compound_name, source_page
summary_embeddings
Dense (384-dim) for page + artifact summaries
Payload: type (page|artifact), parent_id, entities

/ Key capabilities

  • Dense neural embeddings via nomic-embed-text-v1.5 (768 dimensions)
  • Sparse term-frequency vectors with scientific identifier-preserving tokenization
  • Chemical structure embeddings via ChemBERTa (384 dimensions)
  • Unified summary collection for hierarchical page + document retrieval
  • Entity metadata stored as filterable Qdrant payload on every vector
  • Event-driven indexing: embeddings auto-trigger after extraction completes
Read the docsNext: Entity-Aware Search