LLM Providers

DocuStore uses LLMs for multiple purposes: page and document summarization, named entity recognition, conversational RAG, and evaluation. The LLM integration supports four providers and allows separate configuration for batch processing, entity extraction, and interactive chat.

Supported Providers

Each provider's own key variable takes precedence; LLM_API_KEY is the fallback for all of them. Set ALLOW_CLOUD_LLM=false to refuse every cloud provider (air-gapped deployments: Ollama only).

Batch LLM (Summarization, NER)

The batch LLM handles automated pipeline tasks: page summarization, artifact summarization, and (optionally) NER extraction. These tasks prioritize consistency over creativity, so a low temperature is used.

# Ollama (default)
LLM_PROVIDER=ollama
LLM_MODEL_NAME=gemma4:31b
LLM_BASE_URL=http://localhost:11434
LLM_TEMPERATURE=0.1

# OpenAI
LLM_PROVIDER=openai
LLM_MODEL_NAME=gpt-5-mini
OPENAI_API_KEY=sk-...

# Anthropic
LLM_PROVIDER=anthropic
LLM_MODEL_NAME=claude-sonnet-5-5
ANTHROPIC_API_KEY=sk-ant-...

# Gemini
LLM_PROVIDER=gemini
LLM_MODEL_NAME=gemini-2.5-flash
GOOGLE_API_KEY=AIza...

Chat LLM (Conversational RAG)

The chat system can use a different model and provider than the batch pipeline. This is useful when you want a larger, more capable model for interactive conversations while using a smaller model for batch processing.

If chat-specific variables are not set, they fall back to the batch LLM configuration.

# Use a different model for chat
CHAT_LLM_PROVIDER=openai
CHAT_LLM_MODEL_NAME=gpt-5
CHAT_LLM_API_KEY=sk-...
CHAT_LLM_TEMPERATURE=0.3

# Or use Ollama with a larger model
CHAT_LLM_PROVIDER=ollama
CHAT_LLM_MODEL_NAME=gemma4:31b
CHAT_LLM_BASE_URL=http://localhost:11434
CHAT_LLM_TEMPERATURE=0.3

Embedding and Reranking Models

Text embeddings (documents and search queries) are covered on their own page: Embedding Models, which explains the local default, API options, and how to switch.

The chemistry embedding model and the reranker always run locally via sentence-transformers:

SMILES_EMBEDDING_MODEL_NAME=DeepChem/ChemBERTa-77M-MTR
SMILES_EMBEDDING_DEVICE=cpu

RERANKER_MODEL_NAME=cross-encoder/ms-marco-MiniLM-L-12-v2
RERANKER_DEVICE=cpu
RERANKER_ENABLED=true

NER Models

Named entity recognition uses two models:

Entity extraction can run on its own provider and model. Each NER_LLM_* variable falls back to the matching LLM_* one. Only ollama, openai, and gemini can run LLM entity extraction; with anthropic it falls back to dictionary-only matching.

NER_LLM_PROVIDER=openai          # ollama | openai | gemini
NER_LLM_MODEL_NAME=gpt-5-mini
NER_LLM_API_KEY=sk-...           # else the provider's key, then LLM_API_KEY
NER_LLM_BASE_URL=                # Ollama only
NER_MAX_CHAR_BUFFER=5000
GLINER2_MODEL_NAME=fastino/gliner2-large-v1

Evaluation Judge

For automated evaluation of search and chat quality, a separate LLM-as-judge configuration is available:

EVAL_JUDGE_PROVIDER=openai
EVAL_JUDGE_MODEL=gpt-4o
EVAL_JUDGE_API_KEY=sk-...
EVAL_JUDGE_TEMPERATURE=0.0

Tool Calling Modes

The chat pipeline's agentic retrieval uses LLM tool calling. CHAT_AGENT_TOOL_CALLING_MODE (default auto) uses each provider's native tool calling, including Ollama's. Set it to react for a model without native tool support; the agent then drives tools through ReAct-style prompting.

Ollama Setup

For local development, install Ollama and pull the required model:

# Install Ollama (macOS)
brew install ollama

# Start the server
ollama serve

# Pull the default model
ollama pull gemma4:31b

For memory-constrained environments, reduce concurrent LLM activities:

TEMPORAL_MAX_CONCURRENT_LLM_ACTIVITIES=1

Workspace Keys (Self-Serve)

On a shared, self-serve deployment you usually don't want one server-wide key paying for everyone. With workspace keys enabled, each workspace runs on the AI provider its own admins connect:

USER_LLM_KEYS_ENABLED=true
# Fernet key that encrypts stored provider keys. Needed by the API and the workers.
USER_LLM_KEYS_SECRET=...   # python -c 'from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())'

How it behaves:

• Admins connect the keys. Under Settings → AI Provider, a workspace admin connects OpenAI or Google Gemini with an API key, or OpenRouter through its sign-in flow, and picks the ingestion and chat models. A workspace can keep several providers and switch between them; one is active.
• Every member runs on them. Uploads, entity extraction, summaries, chat, and the workspace's embedding model all use the workspace's active provider. Members see which provider is in use but never the key. Cap each member's monthly usage under Settings → Token Settings.
• Fails closed. Until a workspace has a provider, uploads and chat return HTTP 428: admins are sent to set one up, members are told to ask an admin. If the server also has its own LLM_* key configured, workspaces without a key fall back to it; leave the server keys unset for a public deployment.
• Keys are write-only. They are encrypted at rest and never returned by the API; admins see the last four characters.

Rotating the encryption key

USER_LLM_KEYS_SECRET accepts several comma-separated keys. The first encrypts; all of them decrypt.

1. Generate a new key and set USER_LLM_KEYS_SECRET=NEW_KEY,OLD_KEY for the API and the workers, then restart. On startup the API re-encrypts every stored key with NEW_KEY.
2. Set USER_LLM_KEYS_SECRET=NEW_KEY and restart. The old key is retired.