Manual Install (Helm)

This page covers installing Cardinal Data Lake yourself with Helm. A POC via the Helm chart is supported for a local or throwaway cluster. To have an AI agent do the install, give it the install skill.

Quick start (local cluster)

The fastest manual path — deploy Cardinal Data Lake on a local Kubernetes cluster with no cloud account required.

kubectl create namespace lakerunner

helm install lakerunner oci://public.ecr.aws/cardinalhq.io/lakerunner \
  --values values.yaml \
  --namespace lakerunner

kubectl get pods -n lakerunner -w

kubectl port-forward -n lakerunner svc/lakerunner-grafana 3000:3000

Before you install

These apply to any install beyond the local quick start.

Kubernetes version

Use Kubernetes 1.33 or later. The chart’s README lists 1.28 as its minimum, but these docs only cover 1.33 and later.

Cluster size

At minimum replicas, the chart’s pods request about 16.3 vCPU and 29.5 GiB of memory, including Grafana and 2 pubsub-http pods. Add room for system overhead and for scale-up:

• Each worker role can scale to 10 replicas. On a small cluster, lower process.<signal>.autoscaling.maxReplicas.
• The query worker defaults to a 100Gi emptyDir scratch limit, which is close to the whole disk on nodes with about 120 GiB. Lower queryWorker.temporaryStorage.size to fit your nodes.

PostgreSQL connections

Set max_connections to at least 200. A minimum-replica install held 66 to 98 connections while idle, and scale-up adds more, so the common default of 100 runs out. When it does, control-plane crash-loops with:

remaining connection slots are reserved for roles with the SUPERUSER attribute (SQLSTATE 53300)

S3-compatible storage

Cardinal Data Lake only processes an object after it gets an event about it. Without events, the bucket fills, ingest stays idle, queries return nothing, and no error appears. Outside AWS, set the wizard’s S3 Endpoint, choose HTTP for notifications, and use one of these routes.

Collector gateway notifications (recommended)

The store needs no event support, because the Cardinal collector’s awss3 exporter POSTs after each successful upload. It needs collector v1.7.0 or later. Add this to the gateway’s exporter config, using your Helm release name and namespace:

exporters:
  awss3:
    s3uploader:
      notifications:
        endpoint: http://<release>-pubsub-http.<namespace>.svc.cluster.local:8080/

This has limits:

• The queue is in memory, and each delivery is tried 3 times by default.
• A dropped notification leaves that object in the bucket unprocessed.
• Anything else that writes to the bucket produces no event.

Bucket events

The store POSTs to the webhook itself, so it has to support webhook events. Set pubsub.HTTP.enabled: true and make the Service reachable from the store, because it defaults to ClusterIP (pubsub.HTTP.service.type).

The webhook:

• Listens on port 8080 and accepts POST on any path.
• Accepts the S3 Records JSON format.
• Has no authentication, so restrict access by network if you expose it. On a LoadBalancer, use pubsub.HTTP.service.loadBalancerSourceRanges.

Configuration wizard

Use the interactive wizard below to generate a values.yaml for your deployment. Select your installation type and fill in the required details.

For the complete list of configuration options, see the full values.yaml reference on GitHub .

Post-installation

Verify the installation

After installation, verify all pods are running:

kubectl get pods -n lakerunner -l app.kubernetes.io/name=lakerunner

Access Grafana

Configure an Ingress or use port-forwarding to access Grafana:

kubectl port-forward -n lakerunner svc/lakerunner-grafana 3000:3000

Deploying on OpenShift

The chart renders cleanly under the restricted-v2 SCC with two adjustments. Add the following to the values.yaml produced by the wizard above:

1. Let the SCC assign UIDs

OpenShift’s restricted-v2 SCC rejects pods whose runAsUser/runAsGroup/fsGroup fall outside the namespace’s assigned UID range — it wants to inject them from the range itself. Null the fields so the chart emits only runAsNonRoot: true:

global:
  podSecurityContext:
    runAsNonRoot: true
    runAsUser: null
    runAsGroup: null
    fsGroup: null
grafana:
  podSecurityContext:
    runAsNonRoot: true
    runAsUser: null
    runAsGroup: null
    fsGroup: null

The SCC fills in runAsUser, runAsGroup, and fsGroup. All other hardening from global.containerSecurityContext (no-privilege-escalation, drop ALL, RuntimeDefault seccomp, read-only rootfs) stays in effect.

2. Grafana needs an SCC that permits UID 472

The upstream grafana/grafana image expects UID 472 to own /var/lib/grafana. Because that UID almost never falls inside a namespace’s restricted-v2 range, Grafana needs either a custom SCC or an OpenShift-compatible image. The simplest path is:

oc adm policy add-scc-to-user nonroot-v2 -z <release>-lakerunner -n <namespace>

…and then keep Grafana’s defaults (UID 472). If you prefer to avoid the SCC grant, swap grafana.image.repository for an image that supports random UIDs.

3. Perch needs elevated RBAC

The Perch component ships with a ClusterRole that includes patch on apps/deployments cluster-wide — this is its legitimate function (cross-namespace deployment management), but it counts as a privileged grant. On clusters with strict RBAC review you may need admin approval or an oc adm policy add-role-to-user edit against the chart’s ServiceAccount.

4. Ingress / Routes

The chart uses standard networking.k8s.io/v1 Ingress resources. They work with the OpenShift HAProxy router out of the box; no nginx-specific annotations are emitted.

Upgrading

To upgrade an existing installation:

helm upgrade lakerunner oci://public.ecr.aws/cardinalhq.io/lakerunner \
  --values values.yaml \
  --namespace lakerunner

Uninstalling

To remove Cardinal Data Lake:

helm uninstall lakerunner --namespace lakerunner

Note: This does not delete your data in the cloud storage bucket.