Plugin System | DocuStore

Kafka-driven extensibility framework for adding external enrichment services without modifying core code.

DocuStore includes a plugin system that lets you add external data enrichment capabilities -- such as PubChem lookups, ChEMBL cross-references, or UniProt annotations -- without touching core application code.

When to Use Plugins

Use Case	Approach
External API enrichment (PubChem, ChEMBL)	Plugin system
Supplementary data not on domain aggregates	Plugin system
Core domain data (compounds, NER, summaries)	Traditional service (`ADDING_SERVICES.md`)
Data stored on domain aggregates	Traditional service

Architecture

Plugins react to domain events delivered via Kafka. The core pipeline publishes enriched events with sub-types, and plugins subscribe to the specific sub-types they care about.

EventStoreDB → pipeline_worker → Use Cases → Kafka
                                               ├─ plugin_pubchem (consumer group)
                                               ├─ plugin_chembl  (consumer group)
                                               └─ ...

Event Flow

A core use case modifies a domain aggregate (e.g., compounds extracted from a page).
The use case publishes to Kafka with a sub-type: ExternalEventPublisher.notify_page_updated(result, sub_type="CompoundMentionsUpdated").
The plugin consumer process matches the sub-type against plugin manifests.
The matching plugin handler starts a Temporal workflow on the plugin's dedicated task queue.

Kafka Message Format

{
  "event_type": "PageUpdated",
  "sub_type": "CompoundMentionsUpdated",
  "data": {
    "page_id": "...",
    "artifact_id": "...",
    "compound_mentions": [...]
  }
}

The data field contains the full DTO, so plugins receive rich context without needing to query read models.

Available Sub-Types

Sub-Type	Triggered When
`CompoundMentionsUpdated`	Compounds extracted or manually added
`TextMentionUpdated`	Page text extracted from PDF
`TagMentionsUpdated`	NER entities extracted
`SummaryCandidateUpdated`	LLM summary generated
`PagesAdded` / `PagesRemoved`	Pages linked/unlinked from artifact
`TitleMentionUpdated`	Artifact title extracted
`TagsUpdated`	Artifact tags aggregated

Plugin Contract

Every plugin implements a standard protocol and declares its requirements via a PluginManifest:

MY_MANIFEST = PluginManifest(
    name="pubchem_enrichment",
    version="1.0.0",
    description="Enriches compounds with PubChem data.",
    subscribed_events=["CompoundMentionsUpdated"],
    consumer_group="plugin_pubchem_enrichment",
    mongo_collections=["pubchem_enrichments"],
    temporal_task_queue="plugin_pubchem_enrichment",
    has_api_routes=True,
    api_prefix="/plugins/pubchem_enrichment",
)

The plugin class provides factory methods for all its components:

Method	Returns
`manifest()`	`PluginManifest` declaration
`create_event_handler(context)`	Kafka event handler that starts Temporal workflows
`create_workflows()`	List of Temporal workflow classes
`create_activities(context)`	List of Temporal activity callables
`create_router(context)`	FastAPI router for plugin-specific API endpoints
`health_check()`	Plugin health status
`backfill(context)`	Re-process historical documents

Plugin Isolation

Plugins operate within strict boundaries:

Plugins receive:

page_read_model / artifact_read_model -- MongoDB read-only queries
mongo_db -- Access to plugin-owned collections only
temporal_client -- For plugin-owned workflows only
embedding_generator -- Shared embedding infrastructure
plugin_config -- Plugin-specific settings

Plugins cannot:

Load or save domain aggregates (no PageRepository or ArtifactRepository)
Write to core MongoDB collections (page_read_models, artifact_read_models)
Modify Qdrant collection payloads
Access the core WorkflowOrchestrator

Plugin Directory Structure

plugins/my_plugin/
  __init__.py          # Exports `plugin` attribute
  manifest.py          # PluginManifest
  config.py            # Plugin-specific pydantic Settings
  plugin.py            # Main Plugin class
  use_cases/           # Business logic
  infrastructure/      # External API clients
  storage/             # MongoDB adapters
  temporal/            # Workflows + activities
  api/                 # FastAPI routes

Enabling Plugins

Add plugin names to the ENABLED_PLUGINS environment variable:

ENABLED_PLUGINS=pubchem_enrichment,chembl_enrichment

Per-plugin configuration uses a PLUGIN_{NAME}_ prefix:

PLUGIN_PUBCHEM_API_BASE_URL=https://pubchem.ncbi.nlm.nih.gov/rest/pug
PLUGIN_PUBCHEM_RATE_LIMIT_PER_SECOND=5.0
PLUGIN_PUBCHEM_BATCH_SIZE=10

Backfill

When a plugin is installed after documents have already been processed:

Automatic (Kafka replay) -- New consumer groups start from auto.offset.reset=earliest, replaying all events within Kafka's retention window.
Manual -- Call the plugin's backfill() method, which iterates MongoDB read models and runs enrichment logic directly.

Frontend Integration

Plugin data is fetched separately from core API responses to keep the core API stable:

# Discover enabled plugins
curl http://localhost:8000/plugins

# Fetch plugin data for a page
curl http://localhost:8000/plugins/pubchem_enrichment/pages/{page_id}/enrichments

Adding or removing a plugin never changes core response shapes.