Architecture

Data Hub has four components that coordinate through S3, a REST API, and a shared Python library. This page is the conceptual map; the deployment mechanics live in the first-time deployment guide in the data-hub developer docs.

System overview

Components

Data flow

Automatic upload (auto mode)

1. A lab instrument writes output files to a watched directory.
2. The watcher detects new stable files via filesystem events.
3. It groups files into runs and reports each run to the API.
4. It uploads raw files to S3 at the key {instrument_id}/{run_id}/{filename}.
5. The S3 upload triggers the Lambda.
6. Lambda downloads the file and dispatches to the instrument's processor for preprocessing.
7. Lambda creates/updates the run and files via the API, which sends a Slack notification once per new run.
8. Users view the run in the web dashboard.

Manual upload (manual mode)

Steps 1–3 are the same, but the watcher doesn't upload immediately:

4. The server adds files to an upload queue.
5. On each heartbeat tick the watcher polls the queue and uploads requested files.
6. Steps 5–8 from auto mode follow.

Key design decisions

• S3 is the integration boundary. The watcher and Lambda never talk directly; S3 is the durable hand-off: the watcher writes, Lambda reads.
• API-driven coordination. The watcher registers, syncs its YAML config, and heartbeats, which is what powers dashboard health and the upload queue.
• Pre-signed URLs from the API. The web app mints pre-signed S3 upload/download URLs so bytes move directly to/from S3 without routing through the API. On Vercel the app assumes an AWS Identity and Access Management (IAM) role via OpenID Connect (OIDC), with no long-lived AWS keys.
• Lambda-built run archives. “Download all” delegates to the Lambda, which zips files into a separate archives bucket via S3 multipart upload; the web app then 302s the browser to a short-lived presigned URL. Bytes never traverse Vercel.
• Shared library for contracts. Instrument IDs, S3 utilities, and environment config live in data-hub-shared so Lambda and the watcher stay consistent.
• MCP for AI access. The web app serves an MCP endpoint at /api/v1/mcp exposing read tools, resources, and prompts to AI clients via a personal access token. See the API reference.

Branch and environment strategy

Feature branches target staging via pull requests (PRs). Continuous integration (CI) runs on every PR and on merges to both branches; merges to staging and production deploy each component to its environment automatically. Staging and production are independent deployments with separate databases.