PostgreSQL Storage Backend

By default Ark stores resources as Kubernetes CRDs in etcd. Ark also supports a PostgreSQL-backed mode where resources live in a Postgres database and are served via a Kubernetes aggregated API server . This page covers when to choose it, the database requirements, how to install it, and how to operate it.

For the architectural background, see Core Architecture.

When to use PostgreSQL mode

Prefer PostgreSQL when any of these apply:

• Resource scale beyond etcd’s comfort zone. Large fleets of Agents, Models, Queries, MCPServers can push etcd object-count limits and slow the API server.
• Resource-size pressure. etcd’s 1.5 MiB per-object limit is a hard ceiling; Postgres rows are not.
• Persistence and operational tooling. Standard SQL backups, point-in-time recovery, CDC, and BI tooling become available.
• Multi-region or external DB strategy. Managed Postgres (RDS, Cloud SQL, Aiven) is easier to share across clusters than etcd.

Stay on etcd if:

• You want zero database operational burden.
• You don’t need the scale above and prefer the simpler single-binary controller.

Architecture in PostgreSQL mode

Two Helm releases work together:

• ark-controller — runs the reconciler. CRDs for ark.mckinsey.com/* are not installed in this mode.
• ark-apiserver — registers as an aggregated API server. The Kubernetes API server proxies all ark.mckinsey.com requests to it; it persists resources to Postgres.

When a user runs kubectl apply -f agent.yaml, the request flow is:

kubectl → kube-apiserver → APIService (v1alpha1.ark.mckinsey.com)
       → ark-apiserver → PostgreSQL (resources table)

The controller observes resources through the same K8s API path; it doesn’t talk to Postgres directly.

PostgreSQL requirements

The apiserver creates a logical replication slot to drive its watch stream, so the database must allow logical replication:

wal_level             = logical
max_replication_slots >= 1
max_wal_senders       >= 1

Minimum version

The watch stream decodes the WAL with the built-in pgoutput plugin and a CREATE PUBLICATION, introduced in PostgreSQL 10. The hard minimum is PostgreSQL 13: slot health checks read pg_replication_slots.wal_status and list pagination calls pg_current_snapshot(), neither of which exists in earlier majors. Run a currently-supported major (14 or newer) in production.

A user/role with permission to:

• CREATE TABLE, CREATE INDEX, CREATE PUBLICATION on the target database
• the REPLICATION attribute (ALTER ROLE ark REPLICATION). Postgres permits replication slots only to roles that hold this attribute, and no predefined role grants it. Managed services often withhold ALTER ROLE and expose an equivalent grant instead (for example rds_replication on AWS RDS) — check your provider’s documentation.

For managed services:

The connection settings the chart accepts are listed in ark/dist/chart-apiserver/values.yaml.

Connection pooling and PgBouncer

The apiserver holds a dedicated logical replication connection for its watch stream in addition to its regular query pool. Route only the query traffic through an external pooler, and keep the replication connection direct:

• PgBouncer in transaction or statement pooling mode is incompatible with logical replication. The replication connection must reach Postgres directly (or through a session-pooling/session mode pooler — for PgBouncer this requires version 1.21.0 or newer, which added replication-connection passthrough; older versions reject the replication protocol even in session mode). Point ARK_POSTGRES_HOST at a direct endpoint, or run a separate session-mode pooler for the apiserver.
• Managed poolers (RDS Proxy, Cloud SQL connectors) have the same constraint — replication slots require a session-scoped connection.

The apiserver already pools its own query connections (MaxOpenConns defaults to 40); an external transaction-pooling layer in front of the query path is optional and must not carry the replication connection.

Schema and replication slot

On first start, ark-apiserver creates:

• A single table, resources, with one row per Ark resource (Agent, Model, Query, Team, …). Columns include kind, namespace, name, uid, resource_version, JSONB columns for spec, status, labels, annotations, finalizers, owner_references, plus timestamps and a soft-delete flag (deleted_at).
• Indexes on (kind, namespace), (kind, namespace, name), a GIN index on labels, and a unique partial index on active (non-deleted) rows.
• A publication and a logical replication slot, both named ark_cdc. The slot is what powers kubectl get -w and controller informers.

The slot is persistent: it survives apiserver restarts. It is dropped automatically on helm uninstall — see Uninstall and cleanup below.

WAL retention

ark_cdc publishes only the resources table, but WAL is cluster-wide: while the slot holds a position, Postgres keeps every segment from that point, including changes for unrelated tables and databases on the same instance. The consumer therefore acknowledges the server’s reported WAL position on each keepalive, not just its own last write, so an idle Ark install does not pin WAL on a busy instance.

Check that the slot is keeping up:

SELECT slot_name, active,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained
FROM pg_replication_slots WHERE slot_name = 'ark_cdc';

retained should stay small. Sustained growth means no replica is consuming the slot — check that a pod holds the ark-apiserver-leader lease and that active is true. Setting max_slot_wal_keep_size caps the damage: Postgres invalidates the slot instead of filling the disk, and the apiserver recreates it on the next restart.

Installing PostgreSQL mode

1. Prepare PostgreSQL

Provision a database with logical replication enabled, create the Ark database and user, and obtain the password.

2. Create the Kubernetes password secret

The chart references the password by secret name; you create it once:

kubectl create namespace ark-system
kubectl create secret generic ark-db-password \
  -n ark-system \
  --from-literal=password='<your-password>'

3. Configure .arkrc.yaml

The CLI reads the backend choice and connection details from .arkrc.yaml. You can place this file in either:

• ~/.arkrc.yaml (user-level, applies to all projects)
• ./.arkrc.yaml (project-level, takes precedence)

# .arkrc.yaml
storage:
  backend: postgresql
  postgresql:
    host: ark-storage.example.com
    port: 5432
    database: ark
    user: ark
    passwordSecretName: ark-db-password
    passwordSecretKey: password
    sslMode: require

sslMode accepts the standard libpq values: disable, require, verify-ca, verify-full. The default is require, so the controller→database connection is encrypted out of the box; set disable explicitly for a non-TLS database (e.g. local dev).

require encrypts the connection but does not verify the server’s identity, so it only protects against a passive eavesdropper on the wire. An active attacker who can present any TLS certificate — a rogue pod, DNS misroute, or MITM — can still impersonate the database. For production, use verify-full, which validates the server certificate and hostname; verify-ca validates the certificate but not the hostname. Against a publicly-trusted managed database (RDS, Cloud SQL) they work with no extra configuration because the controller image trusts the system CA bundle. For a private CA, self-signed certificate, or mutual TLS, mount the PEM files via a Secret on the apiserver chart:

kubectl create secret generic ark-pg-tls \
  --namespace ark-system \
  --from-file=ca.crt=ca.crt \
  --from-file=tls.crt=client.crt \
  --from-file=tls.key=client.key
 
helm upgrade --install ark-apiserver \
  oci://ghcr.io/mckinsey/agents-at-scale-ark/charts/ark-apiserver \
  --namespace ark-system \
  --set postgresql.host=ark-storage.example.com \
  --set postgresql.user=ark \
  --set postgresql.passwordSecretName=ark-db-password \
  --set postgresql.sslMode=verify-full \
  --set postgresql.sslSecretName=ark-pg-tls \
  --set postgresql.sslRootCertKey=ca.crt \
  --set postgresql.sslClientCertKey=tls.crt \
  --set postgresql.sslClientKeyKey=tls.key

The Secret is mounted read-only at /etc/ark/postgres-tls and the named keys are passed to libpq as sslrootcert/sslcert/sslkey. Only set the client cert/key keys if the database requires client-certificate authentication; sslRootCertKey alone is enough for verify-ca/verify-full against a private CA.

The --backend CLI flag and ARK_STORAGE_BACKEND env var override the config value, useful for testing the same code against a different backend without editing the file.

4. Install via the CLI

ark install

The CLI installs ark-controller with storage.backend=postgresql (which disables CRD installation) and ark-apiserver with the connection values from the config. cert-manager and Gateway API CRDs are installed as dependencies just as in etcd mode.

Install via raw Helm

If you prefer to skip the CLI, install the two charts directly with --set flags from the values you would have put in .arkrc.yaml:

helm upgrade --install ark-controller \
  oci://ghcr.io/mckinsey/agents-at-scale-ark/charts/ark-controller \
  --namespace ark-system --create-namespace \
  --set rbac.enable=true \
  --set storage.backend=postgresql
 
helm upgrade --install ark-apiserver \
  oci://ghcr.io/mckinsey/agents-at-scale-ark/charts/ark-apiserver \
  --namespace ark-system \
  --set postgresql.host=ark-storage.example.com \
  --set postgresql.user=ark \
  --set postgresql.passwordSecretName=ark-db-password \
  --set postgresql.sslMode=require

The apiserver chart’s postgresql.host, postgresql.user, and postgresql.passwordSecretName are required values — helm install will fail at template time if they are missing.

Verifying the install

# No Ark CRDs in postgresql mode.
kubectl get crd | grep ark.mckinsey.com
# (no output)
 
# Both APIServices should report Available=True.
kubectl get apiservice v1alpha1.ark.mckinsey.com v1prealpha1.ark.mckinsey.com
 
# kubectl operates on Ark resources transparently.
kubectl get agents,models,queries -A

Create a smoke-test Agent to confirm the round trip lands in Postgres:

kubectl apply -f - <<EOF
apiVersion: ark.mckinsey.com/v1alpha1
kind: Agent
metadata:
  name: smoke
  namespace: default
spec:
  description: smoke test
  prompt: "You are a helpful assistant."
EOF

Then connect to Postgres and confirm the row exists:

SELECT kind, namespace, name, uid FROM resources WHERE kind = 'Agent';

Security model

The aggregated apiserver enforces the same access control as the rest of the cluster:

• Delegated authentication and authorization. Every request is authenticated against the kube-apiserver (TokenReview and requestheader/front-proxy identity) and authorized via SubjectAccessReview, so Kubernetes RBAC on ark.mckinsey.com resources applies to direct service access as well as to the kubectl path. Health endpoints (/healthz, /readyz, /livez) stay unauthenticated. Set ARK_APISERVER_AUTH_MODE=off (for example via extraEnv) only for local development outside a cluster.
• Verified serving TLS. With certManager.enabled (the default), cert-manager issues the serving certificate for the ark-apiserver service, mounts it into the pod, and injects the CA into both APIServices — the kube-apiserver verifies the aggregated apiserver’s identity instead of insecureSkipTLSVerify. The certificate rotates through cert-manager and the server reloads it without a restart. Setting certManager.enabled=false restores the previous behavior (ephemeral self-signed certificate, unverified proxy channel) for clusters without cert-manager.
• Optional NetworkPolicy. networkPolicy.enabled=true restricts ingress to the serving and health ports; use networkPolicy.extraIngressFrom to pin the allowed sources for the serving port (the origin of kube-apiserver traffic depends on your CNI, which is why this is off by default).
• Audit log. audit.enabled (default true) makes the aggregated apiserver emit its own Kubernetes audit trail as JSON on stdout, at Metadata level by default. Because the records are produced in-process they cover the direct service path too. A policy file is mandatory when audit is enabled — without one the upstream backend records nothing, so the apiserver refuses to start rather than report audit as active while emitting nothing; the chart always mounts one. Tune with audit.level or supply a full audit.policy. Note that Request/RequestResponse levels put request and response bodies — including Query .spec.input — into your log pipeline.
• Policy enforcement. The host kube-apiserver runs no webhook chain on aggregated resources, so this apiserver enforces policy itself, two independent ways. A native ValidatingAdmissionPolicy (CEL) is evaluated in-process (policy.cel.enabled, default true) and requires the host cluster at k8s ≥1.30. The webhook admission plugins can also be run here (policy.thirdPartyWebhooks.enabled, default false), which makes Kyverno/Gatekeeper webhook configurations fire on Ark resources; it is off by default because it puts a synchronous webhook call on every write. Each mechanism is best-effort by default: on an older host, if the startup discovery probe never succeeds, or if the ServiceAccount lacks the watch RBAC, the apiserver logs the reason and serves with that mechanism off (Ark’s own in-process validation and audit still apply). Set policy.cel.required=true or policy.thirdPartyWebhooks.required=true to make that a startup failure instead. Policies that use paramKind need read access to the parameter resource via policy.extraParamRules unless the parameters live in ConfigMaps or Secrets, which are already granted.

Availability and multi-replica behaviour

The chart defaults to a single replica. Understand what that means before running it in production: when the aggregated apiserver is unavailable, the ark.mckinsey.com APIService goes unavailable, which degrades kube-apiserver API discovery and the garbage-collection and namespace controllers cluster-wide — a standard Kubernetes aggregation-layer failure mode, not something specific to Ark. A single replica is therefore a cluster-wide single point of failure during pod restarts, node drains, and crashes.

For production, run two replicas and enable the disruption budget:

--set replicas=2 --set podDisruptionBudget.enabled=true

How multiple replicas behave:

• API serving is active-active. Every replica serves reads and writes directly against Postgres; the Service load-balances across them. Killing any one pod leaves the APIService available.
• The watch stream is single-consumer by design. Only the replica holding the ark-apiserver-leader lease runs the WAL consumer that drives real-time watch events (the logical replication slot admits one connection; the slot’s active flag is a backstop even without the lease). Non-leader replicas do not touch the slot.
• Watches served by non-leader replicas are relist-bound. Without the WAL stream, a non-leader refreshes its watchers on a periodic relist (up to ~120 seconds stale). Controllers tolerate this — informers resync — but if consistently low watch latency matters, keep a single replica and accept the SPOF trade-off, or route watch-heavy clients through the kubectl path.
• Failover: if the leader dies, a surviving replica acquires the lease (roughly the 15s lease duration) and starts the WAL consumer; the slot position is persistent, so no events are lost across the handover.

Backups and restore

Treat the resources table like any other application table:

• Use your provider’s automated backups or pg_dump for ad-hoc snapshots.
• A point-in-time restore restores Ark state to that moment. Take care to also drop and recreate the ark_cdc replication slot after a restore so the apiserver starts a fresh watch stream.
• Cluster-side state (Pods, Deployments owned by Ark) is not restored by a Postgres restore — only the declarative resources are.

Uninstall and cleanup

helm uninstall runs a post-delete hook Job that drops the ark_cdc replication slot and publication. Without this cleanup, an orphaned slot would pin WAL retention and can fill the disk. The Job connects with the same credentials as the apiserver, so the password secret must still exist when you uninstall.

Confirm in the database that the slot is gone rather than trusting the Job’s log line — both queries should return no rows:

SELECT slot_name, active, wal_status FROM pg_replication_slots WHERE slot_name = 'ark_cdc';
SELECT pubname FROM pg_publication WHERE pubname = 'ark_cdc';

If the hook Job could not run (for example the database was unreachable, you removed the release with --no-hooks, or the password secret was deleted first), drop the slot manually:

SELECT pg_drop_replication_slot('ark_cdc');
DROP PUBLICATION IF EXISTS ark_cdc;

If the slot was invalidated (wal_status = 'lost', typically after max_slot_wal_keep_size was exceeded), the apiserver drops and recreates it automatically on startup.

The resources table itself is yours — drop it manually if you are decommissioning the database, or keep it for forensic queries.

Troubleshooting

helm install fails with postgresql.host is required. You ran the apiserver chart without supplying connection details. Set storage.postgresql in .arkrc.yaml (the CLI passes these through) or use --set postgresql.host=… --set postgresql.user=… --set postgresql.passwordSecretName=… for raw helm.

ark install fails with missing 'storage.postgresql' block. You set storage.backend: postgresql in .arkrc.yaml but didn’t add the storage.postgresql block, or it’s missing host/user/passwordSecretName. Fill in the required fields.

Apiserver pod is CreateContainerConfigError. The pod is referencing a secret that doesn’t exist. Confirm the secret named in postgresql.passwordSecretName exists in the same namespace as the release.

Apiserver crashes with failed to connect to database: dial tcp … connect: connection refused. The host/port is wrong, the database isn’t up yet, or a NetworkPolicy is blocking egress. The pod will restart and retry.

Apiserver logs error retrieving resource lock. Leader election can’t reach the Kubernetes API. Usually a transient startup issue; if it persists, check the ServiceAccount, RBAC bindings, and any egress restrictions.

APIService stuck at False (FailedDiscoveryCheck). The aggregator can’t reach the apiserver Service. Check kubectl get svc -n ark-system ark-apiserver, verify pods are 1/1, and confirm no NetworkPolicy blocks port 6443 from the kube-apiserver.

kubectl get agents returns “the server doesn’t have a resource type …”. The APIService is not registered or not Available. Look at kubectl get apiservice v1alpha1.ark.mckinsey.com -o yaml.

Resources don’t appear after kubectl apply, but no error. Check kubectl get events -A. Confirm the apiserver pod is running and the replication slot exists in Postgres (SELECT slot_name, active, wal_status FROM pg_replication_slots).