Blob Storage

DocuStore stores uploaded documents (PDFs) and rendered page images in blob storage. The BlobStore port is implemented with fsspec, so it works with the local filesystem (the default), S3-compatible object storage and Azure Blob Storage. The drivers for both object stores (s3fs and adlfs) ship with DocuStore and are in the published image.

Storage Layout

When a document is uploaded, the blob store organizes files in a predictable structure:

{blob_root}/
  artifacts/
    {artifact_id}/
      source.pdf            # Original upload (extension matches the file)
      pages/
        0.png               # Full-resolution page image (page 0)
        0_thumb.jpg          # Thumbnail (page 0)
        1.png               # Full-resolution page image (page 1)
        1_thumb.jpg          # Thumbnail (page 1)
        ...

Page images are generated during the PDF processing workflow. Full-resolution PNGs are used for deep thinking mode (page image analysis), while JPEG thumbnails (400px on the longest edge) are served for list views and table previews.

Local Filesystem (Development)

The default configuration stores blobs on the local filesystem. This is the simplest option and requires no additional infrastructure.

# Default: stores in services/blobs/ directory
BLOB_BASE_URL=file:///path/to/docu-store/services/blobs

The file:// URL scheme tells DocuStore to use the local filesystem adapter. The path must be absolute.

S3-Compatible Storage

Point the blob root at a bucket (AWS S3, MinIO, Cloudflare R2, etc.). DocuStore writes under the prefix after the bucket name:

BLOB_BASE_URL=s3://my-docustore-bucket/docu-store

Credentials come from the standard AWS environment variables. When they are unset, the instance's IAM role is used instead:

AWS_ACCESS_KEY_ID=AKIA...
AWS_SECRET_ACCESS_KEY=...
AWS_DEFAULT_REGION=us-east-1

For S3-compatible services (MinIO, R2), also set the endpoint:

AWS_ENDPOINT_URL=https://minio.example.com:9000

The credentials must be allowed to list the bucket and to read, write and delete objects under the prefix (s3:ListBucket, s3:GetObject, s3:PutObject, s3:DeleteObject).

Azure Blob Storage

Point the blob root at a container in a storage account:

BLOB_BASE_URL=az://my-container/docu-store

Then authenticate in one of three ways:

# 1. Account name and key
AZURE_STORAGE_ACCOUNT_NAME=mystorageaccount
AZURE_STORAGE_ACCOUNT_KEY=...

# 2. Connection string
AZURE_STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=https;AccountName=mystorageaccount;AccountKey=...;EndpointSuffix=core.windows.net"

# 3. Managed identity: the account name alone
AZURE_STORAGE_ACCOUNT_NAME=mystorageaccount

With only the account name, DocuStore signs in through Azure's DefaultAzureCredential: a managed identity, workload identity, the AZURE_CLIENT_ID / AZURE_CLIENT_SECRET / AZURE_TENANT_ID variables of a service principal, or az login on a developer machine. Give that identity the Storage Blob Data Contributor role on the storage account or the container. A user-assigned managed identity also needs AZURE_CLIENT_ID set to its client ID.

Storage Options

Anything else fsspec accepts goes in BLOB_STORAGE_OPTIONS as JSON, and values there take precedence over the environment variables above:

# S3-compatible endpoint with explicit keys
BLOB_STORAGE_OPTIONS='{"key": "...", "secret": "...", "endpoint_url": "http://localhost:9000"}'

# Azure connection string
BLOB_STORAGE_OPTIONS='{"connection_string": "DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...;EndpointSuffix=core.windows.net"}'

When you run the services outside Docker (make run-all), credentials must go here, or be exported in the shell. The services read services/.env into their own settings only, so the AWS and Azure SDKs never see AWS_* or AZURE_* variables written in that file. Docker's env_file does export them.

Checking the Connection

Settings → Status shows the Blob Storage check. It is healthy once the bucket or container is reachable, even before the first upload. Bucket or container not found means it has to be created first. Any other failure shows only the error type (for example PermissionError for rejected S3 credentials), because the full message names buckets and accounts. The full message is in the API log as health.blob_storage_failed.

API Endpoints

The API provides streaming endpoints for serving blob content to the frontend:

Stream PDF

GET /artifacts/{artifact_id}/pdf

Returns the raw PDF binary as application/pdf with an inline content disposition. The browser can render this directly in an iframe or PDF viewer.

Stream Page Image

GET /artifacts/{artifact_id}/pages/{page_index}/image

Returns the full-resolution PNG page image. Add ?size=thumb for the lightweight JPEG thumbnail.

# Full resolution (for detail views, deep thinking)
GET /artifacts/{artifact_id}/pages/0/image

# Thumbnail (for list views, previews)
GET /artifacts/{artifact_id}/pages/0/image?size=thumb

The thumbnail endpoint falls back to the full PNG if a thumbnail has not been generated.

How Blobs Are Created

The blob store is used at two points in the processing pipeline:

1. Document Upload

When a document is uploaded via POST /artifacts/upload, the ArtifactUploadSaga orchestrates:

1. Store the file bytes in blob storage at artifacts/{artifact_id}/source.pdf.
2. Create the Artifact aggregate with the storage_location pointing to the stored file.
3. The Artifact.Created event triggers the processing pipeline.

2. PDF Processing

The ParseArtifactWorkflow reads the PDF from blob storage (Office formats are first converted to a derived PDF) and:

1. Parses each page with Docling.
2. Renders each page to a PNG image (Docling images_scale=4.0, about 288 DPI).
3. Generates a JPEG thumbnail for each page.
4. Stores images and thumbnails back to blob storage.
5. Creates Page aggregates with references to the artifact.

Configuration

Considerations

• Blob storage is append-mostly. Documents are uploaded once and read many times. Updates are rare (only when re-processing overwrites page images).
• Page images can be large. At ~288 DPI a page PNG is typically 0.5-1 MB, so a 12-page PDF produces roughly 6-12 MB of PNG images. Thumbnails are about 20 KB each.
• Local storage needs a shared volume. The API and the workers run as separate containers and all read the same blobs, so with file:// every one of them must mount the same volume (or network share) at the same path. The sample docker-compose.prod.yml shares a blobs volume between the API and both Temporal workers. A named volume is local to one machine, so on a multi-node swarm, mount shared storage such as NFS there instead. Object storage (S3, Azure) avoids this and also suits multi-instance deployments.
• Blobs are referenced by storage location, not copied. The artifact aggregate stores the blob path (e.g., artifacts/{id}/source.pdf), and all access goes through the BlobStore port. Migrating storage backends requires updating the blob root URL, not the aggregate data. Copy the existing tree under the new root as it is, e.g. aws s3 sync /data/blobs s3://my-docustore-bucket/docu-store or azcopy copy '/data/blobs/*' 'https://<account>.blob.core.windows.net/<container>/docu-store' --recursive.