DataFlow architecture
A visual map of the web, API, durable runtime, data stores, and connectors.
DataFlow keeps the user experience, durable orchestration, connector runtime, and data stores as clear operational layers. The API and workers are separate processes, but share one Go codebase and versioned contracts.
Runtime at a glance
Process boundaries
One Go module builds three independently deployable processes:
- API — identity, tenants, connector catalog, pipeline versions, run control, monitoring, lineage, analytics, and audit events.
- Workflow worker — deterministic Temporal DAG orchestration. It performs no network or database I/O.
- Activity worker — connector I/O, payload handling, checkpoints, sink writes, and execution bookkeeping.
Processes share code and contracts, not in-memory state. Temporal task queues separate Integration and Production execution.
Data ownership
| Store | Responsibility |
|---|---|
| PostgreSQL | System of record for tenants, users, versions, executions, checkpoints, entitlements, audit events, and outboxes. |
| Temporal | Workflow history, retries, timers, signals, and orchestration state. |
| Redis | Rate-limit counters and a bounded event stream; it is not a system of record. |
| ClickHouse | Rebuildable analytics data; it is never authoritative. |
Large payloads use encrypted object storage. Small payloads can remain inline only within configured size and retention limits.
Scaling model
API, workflow workers, and activity workers scale independently. Connector workers can be isolated by Temporal task queue without splitting the product into microservices. For production, scale the stateless processes horizontally and use managed or highly available data services appropriate to your SLOs.