Observability
Configuring Langfuse for LLM tracing and prompt management, plus Umami for web analytics.
DocuStore integrates Langfuse for LLM observability and prompt management, and Umami for web analytics. Both are self-hosted and included in the Docker Compose development stack.
Langfuse
Langfuse provides two capabilities:
- LLM tracing -- Every LLM call (summarization, NER, chat) is traced with input/output tokens, latency, model parameters, and prompt versions.
- Prompt management -- Production prompts are stored and versioned in Langfuse, allowing iteration without code changes.
Prompt Management
DocuStore prompts are managed in Langfuse rather than hardcoded. The PromptRepository port loads prompts by name, and the Langfuse adapter fetches them from the Langfuse API with caching.
Key prompts managed in Langfuse:
| Prompt Name | Used By |
|---|---|
page_summary | Page summarization |
artifact_batch_summary | Artifact summarization (batch stage) |
artifact_synthesis | Artifact summarization (synthesis stage) |
artifact_refinement | Artifact summarization (refinement stage) |
chat_system_factual | Chat synthesis (factual queries) |
chat_system_comparative | Chat synthesis (comparative queries) |
chat_system_exploratory | Chat synthesis (exploratory queries) |
chat_system_compound | Chat synthesis (compound queries) |
chat_system_followup | Chat synthesis (follow-up queries) |
YAML Fallback
If Langfuse is unavailable, set PROMPT_REPOSITORY_TYPE=yaml to load prompts from local YAML files instead. This is useful for offline development or environments without Langfuse.
# Use Langfuse (default)
PROMPT_REPOSITORY_TYPE=langfuse
# Use local YAML files
PROMPT_REPOSITORY_TYPE=yamlConfiguration
| Variable | Default | Description |
|---|---|---|
PROMPT_REPOSITORY_TYPE | langfuse | Prompt source: langfuse or yaml |
LANGFUSE_HOST | http://localhost:3000 | Langfuse server URL |
LANGFUSE_PUBLIC_KEY | (none) | Langfuse public key |
LANGFUSE_SECRET_KEY | (none) | Langfuse secret key |
Development Setup
Langfuse runs as part of the Docker Compose stack with pre-seeded API keys:
LANGFUSE_HOST=http://localhost:3000
LANGFUSE_PUBLIC_KEY=pk-lf-docu-store-dev
LANGFUSE_SECRET_KEY=sk-lf-docu-store-devAccess the Langfuse dashboard at http://localhost:3000 to:
- View LLM call traces with input/output/tokens/latency
- Edit and version prompts
- Compare prompt versions side by side
- Monitor cost and usage over time
LLM Tracing
Every LLM interaction is automatically traced. The chat pipeline wraps entire conversations in a TokenCounter context manager that captures prompt and completion token counts from LLM API responses.
Traces include:
| Field | Description |
|---|---|
| Model name | Which model was used |
| Input tokens | Prompt token count |
| Output tokens | Completion token count |
| Latency | Wall-clock time for the LLM call |
| Prompt version | Which Langfuse prompt version was used |
| Metadata | Query type, mode, conversation ID |
Umami (Web Analytics)
Umami provides privacy-focused web analytics for the DocuStore frontend portal. It tracks page views, sessions, and user behavior without cookies and is fully GDPR compliant.
What Umami Tracks
| Metric | Description |
|---|---|
| Page views | Which pages users visit |
| Sessions | Unique visitor sessions |
| Referrers | How users arrive at the portal |
| Browsers/OS | Client environment distribution |
| Custom events | Feature usage (search, chat, upload) |
Setup
Umami runs as part of the Docker Compose stack alongside Langfuse:
# Both are started by:
make docker-up
# This runs docker-compose.yml + docker-compose-langfuse.ymlThe Umami dashboard is accessible at its configured port. The frontend portal includes the Umami tracking script, which sends analytics data to the self-hosted instance.
Privacy
Umami does not use cookies, does not track personal information, and does not share data with third parties. All analytics data stays on your infrastructure. This makes it suitable for research environments with strict data handling requirements.
Request Timing
DocuStore includes built-in request timing for API performance monitoring:
ENABLE_REQUEST_TIMING=true
SLOW_REQUEST_THRESHOLD_MS=1000When enabled, every API request is logged with its duration. Requests exceeding the threshold are flagged as slow in the logs, making it easy to identify performance bottlenecks.
Structured Logging
DocuStore uses structlog for structured JSON logging throughout the backend. Log entries include:
- Request ID and correlation IDs
- Search latency metrics (
search.latencyevents) - Chat pipeline duration and step counts
- Workflow execution events
LOG_LEVEL=INFO # DEBUG, INFO, WARNING, ERROR
LOG_DIR=./logs # Directory for log filesAll log output is structured JSON, making it easy to ingest into log aggregation systems (ELK, Loki, Datadog) for production monitoring.