Observability
Configuring Langfuse for LLM tracing and prompt management, plus Umami for web analytics.
DocuStore integrates Langfuse for LLM observability and prompt management, and Umami for web analytics. Both are self-hosted and included in the Docker Compose development stack.
Langfuse
Langfuse provides two capabilities:
- LLM tracing -- Every LLM call (summarization, NER, chat) is traced with input/output tokens, latency, model parameters, and prompt versions.
- Prompt management -- Production prompts are stored and versioned in Langfuse, allowing iteration without code changes.
Prompt Management
DocuStore prompts are managed in Langfuse rather than hardcoded. The PromptRepository port loads prompts by name, and the Langfuse adapter fetches them from the Langfuse API with caching.
Default prompts live in services/infrastructure/llm/default_prompts/ and are seeded into Langfuse by make docker-up (make langfuse-seed). Key prompts:
| Prompt Name | Used By |
|---|---|
page_summarization_hybrid | Page summarization (text + page image) |
page_summarization_text_only | Page summarization (text only) |
page_summarization_image_only | Page summarization (page image only) |
artifact_batch_summary | Artifact summarization (batch stage) |
artifact_synthesis | Artifact summarization (synthesis stage) |
artifact_refinement | Artifact summarization (refinement stage) |
document_metadata_extraction | Document metadata extraction |
chat_question_analysis | Chat question analysis |
chat_query_planning | Chat query planning |
chat_agentic_retrieval, chat_agentic_retrieval_react | Chat agentic retrieval |
chat_system, chat_answer_synthesis | Chat answer synthesis |
chat_answer_synthesis_v2, chat_answer_plan | Chat adaptive synthesis |
chat_system_factual | Chat synthesis (factual queries) |
chat_system_comparative | Chat synthesis (comparative queries) |
chat_system_exploratory | Chat synthesis (exploratory queries) |
chat_system_compound | Chat synthesis (compound queries) |
chat_system_followup | Chat synthesis (follow-up queries) |
chat_grounding_verification | Chat grounding verification |
chat_answer_formatting | Chat answer formatting |
chat_conversation_summary | Chat history summarization |
YAML Fallback
If Langfuse is unavailable, set PROMPT_REPOSITORY_TYPE=yaml to load prompts from local YAML files instead. This is useful for offline development or environments without Langfuse.
# Use Langfuse (default)
PROMPT_REPOSITORY_TYPE=langfuse
# Use local YAML files
PROMPT_REPOSITORY_TYPE=yamlConfiguration
| Variable | Default | Description |
|---|---|---|
PROMPT_REPOSITORY_TYPE | langfuse | Prompt source: langfuse or yaml |
LANGFUSE_HOST | http://localhost:3000 | Langfuse server URL |
LANGFUSE_PUBLIC_KEY | (none) | Langfuse public key |
LANGFUSE_SECRET_KEY | (none) | Langfuse secret key |
Development Setup
Langfuse runs as part of the Docker Compose stack with pre-seeded API keys:
LANGFUSE_HOST=http://localhost:3100
LANGFUSE_PUBLIC_KEY=pk-lf-docu-store-dev
LANGFUSE_SECRET_KEY=sk-lf-docu-store-devAccess the Langfuse dashboard at http://localhost:3100 to:
- View LLM call traces with input/output/tokens/latency
- Edit and version prompts
- Compare prompt versions side by side
- Monitor cost and usage over time
LLM Tracing
Every LLM interaction is automatically traced. The chat pipeline wraps entire conversations in a TokenCounter context manager that captures prompt and completion token counts from LLM API responses.
Traces include:
| Field | Description |
|---|---|
| Model name | Which model was used |
| Input tokens | Prompt token count |
| Output tokens | Completion token count |
| Latency | Wall-clock time for the LLM call |
| Prompt version | Which Langfuse prompt version was used |
| Metadata | Query type, mode, conversation ID |
Umami (Web Analytics)
Umami provides privacy-focused web analytics for the DocuStore frontend portal. It tracks page views, sessions, and user behavior without cookies and is fully GDPR compliant.
What Umami Tracks
| Metric | Description |
|---|---|
| Page views | Which pages users visit |
| Sessions | Unique visitor sessions |
| Referrers | How users arrive at the portal |
| Browsers/OS | Client environment distribution |
| Custom events | Feature usage (search, chat, upload) |
Setup
Umami runs as part of the Docker Compose stack alongside Langfuse:
# Both are started by:
make docker-up
# This runs docker-compose.yml + docker-compose-langfuse.yml + docker-compose-umami.ymlThe Umami dashboard is at http://localhost:3101 in the dev stack. The frontend portal includes the Umami tracking script, which sends analytics data to the self-hosted instance.
Privacy
Umami sets no cookies and shares no data with third parties, and all analytics data stays on your own infrastructure. Note that the portal does call Umami's identify() with the signed-in user_id and workspace_id (see AnalyticsProvider.tsx), so events are attributable to a specific account rather than anonymous — treat them as personal data under GDPR and disclose them in your privacy notice. Remove that call if you want analytics to stay fully anonymous.
Request Timing
DocuStore includes built-in request timing for API performance monitoring:
ENABLE_REQUEST_TIMING=true
SLOW_REQUEST_THRESHOLD_MS=1000When enabled, every API request is logged with its duration. Requests exceeding the threshold are flagged as slow in the logs, making it easy to identify performance bottlenecks.
Structured Logging
DocuStore uses structlog for structured logging throughout the backend. Log entries include:
- Request ID and correlation IDs
- Search latency metrics (
search.latencyevents) - Chat pipeline duration and step counts
- Workflow execution events
LOG_LEVEL=INFO # DEBUG, INFO, WARNING, ERROR
LOG_DIR=./logs # Directory for log filesWith APP_ENV=development, logs use structlog's human-readable console renderer. In staging and production they are structured JSON, ready for log aggregation systems (ELK, Loki, Datadog).