Self-host DataFlow

Choose the deployment that matches your environment. All three paths use the published repository configuration; you do not need to edit application code.

Docker Compose

Best for a local evaluation on one machine. Allow at least 8 GB of memory and install Docker Desktop or Docker Engine with Compose.

git clone https://github.com/Cohestra/cohestra-dataflow.git
cd cohestra-dataflow
cp .env.example .env
node scripts/gen-worker-keypair.js
docker compose up -d

Open http://localhost:3002. Check the deployment with:

docker compose ps
curl --fail http://localhost:3002/api/health

Stop the stack without deleting its volumes:

docker compose down

Local Kubernetes

Best when you want to validate the Helm deployment on your workstation. Install Docker, Kind, kubectl, and Helm, then run the repository bootstrap:

git clone https://github.com/Cohestra/cohestra-dataflow.git
cd cohestra-dataflow
./scripts/bootstrap.sh
./scripts/smoke-test.sh

Bootstrap creates the local configuration, builds the images, creates a Kind cluster, and installs the DataFlow Helm chart. Open http://localhost:3002.

To remove the local cluster and its data:

kind delete cluster --name dataflow

Existing Kubernetes cluster

Use the chart in deploy/helm/dataflow. Supply your image references, public application URL, storage classes, and secrets through your normal Helm values and secret-management workflow.

helm upgrade --install dataflow deploy/helm/dataflow \
  --namespace dataflow \
  --create-namespace \
  --values my-dataflow-values.yaml

kubectl rollout status deployment/api -n dataflow
kubectl rollout status deployment/web -n dataflow

For production, use persistent storage and highly available PostgreSQL, Temporal, Redis, and ClickHouse services sized for your workload. Keep database, worker, and internal service ports private; expose only the web ingress.

GCP with Terraform

The repository includes a Terraform starting point in infra/. Use it when GCP is the desired host, review the plan in your own project, and keep secrets out of Terraform state.

cd infra
terraform init
terraform plan \
  -var='project_id=YOUR_PROJECT' \
  -var='admin_cidr=YOUR_PUBLIC_IP/32'
terraform apply \
  -var='project_id=YOUR_PROJECT' \
  -var='admin_cidr=YOUR_PUBLIC_IP/32'

Treat this as a reference deployment, not a universal production blueprint. Adapt networking, managed services, backups, availability, and ingress to your organization's standards. Store runtime secrets in GCP Secret Manager or your existing secret manager and inject them during deployment.

After deployment

1. Open the DataFlow URL and create the first workspace owner.
2. Add connector credentials from Connectors in the application.
3. Build and run a small source-to-sink pipeline.
4. Confirm run history, logs, lineage, and destination records.
5. Configure backups, monitoring, TLS, and credential rotation before using production data.