LLM Providers

DocuStore uses LLMs for multiple purposes: page and document summarization, named entity recognition, conversational RAG, and evaluation. The LLM integration supports three providers and allows separate configuration for batch processing and interactive chat.

Supported Providers

Batch LLM (Summarization, NER)

The batch LLM handles automated pipeline tasks: page summarization, artifact summarization, and (optionally) NER extraction. These tasks prioritize consistency over creativity, so a low temperature is used.

# Ollama (default)
LLM_PROVIDER=ollama
LLM_MODEL_NAME=gemma3:27b
LLM_BASE_URL=http://localhost:11434
LLM_TEMPERATURE=0.1

# OpenAI
LLM_PROVIDER=openai
LLM_MODEL_NAME=gpt-4o
LLM_API_KEY=sk-...
LLM_TEMPERATURE=0.1

# Gemini
LLM_PROVIDER=gemini
LLM_MODEL_NAME=gemini-pro
LLM_API_KEY=AIza...
LLM_TEMPERATURE=0.1

Chat LLM (Conversational RAG)

The chat system can use a different model and provider than the batch pipeline. This is useful when you want a larger, more capable model for interactive conversations while using a smaller model for batch processing.

If chat-specific variables are not set, they fall back to the batch LLM configuration.

# Use a different model for chat
CHAT_LLM_PROVIDER=openai
CHAT_LLM_MODEL_NAME=gpt-4o
CHAT_LLM_API_KEY=sk-...
CHAT_LLM_TEMPERATURE=0.3

# Or use Ollama with a larger model
CHAT_LLM_PROVIDER=ollama
CHAT_LLM_MODEL_NAME=gemma3:27b
CHAT_LLM_BASE_URL=http://localhost:11434
CHAT_LLM_TEMPERATURE=0.3

Embedding Models

Embeddings use separate, non-LLM models that run locally via sentence-transformers:

EMBEDDING_MODEL_PROVIDER=sentence-transformers
EMBEDDING_MODEL_NAME=nomic-ai/nomic-embed-text-v1.5
EMBEDDING_DIMENSIONS=768
EMBEDDING_DEVICE=cpu  # or cuda, mps

SMILES_EMBEDDING_MODEL_NAME=DeepChem/ChemBERTa-77M-MTR
SMILES_EMBEDDING_DEVICE=cpu

RERANKER_MODEL_NAME=cross-encoder/ms-marco-MiniLM-L-12-v2
RERANKER_DEVICE=cpu
RERANKER_ENABLED=true

NER Models

Named entity recognition uses two models:

NER_MAX_CHAR_BUFFER=5000
GLINER2_MODEL_NAME=fastino/gliner2-large-v1

Evaluation Judge

For automated evaluation of search and chat quality, a separate LLM-as-judge configuration is available:

EVAL_JUDGE_PROVIDER=openai
EVAL_JUDGE_MODEL=gpt-4o
EVAL_JUDGE_API_KEY=sk-...
EVAL_JUDGE_TEMPERATURE=0.0

Tool Calling Modes

The chat pipeline's agentic retrieval uses LLM tool calling. The mode is auto-detected based on provider:

Override with CHAT_AGENT_TOOL_CALLING_MODE=auto|native|react.

Ollama Setup

For local development, install Ollama and pull the required model:

# Install Ollama (macOS)
brew install ollama

# Start the server
ollama serve

# Pull the default model
ollama pull gemma3:27b

For memory-constrained environments, reduce concurrent LLM activities:

TEMPORAL_MAX_CONCURRENT_LLM_ACTIVITIES=1