PostgreSQL Storage Backend

By default Ark stores resources as Kubernetes CRDs in etcd. Ark also supports a PostgreSQL-backed mode where resources live in a Postgres database and are served via a Kubernetes aggregated API server . This page covers when to choose it, the database requirements, how to install it, and how to operate it.

For the architectural background, see Core Architecture.

When to use PostgreSQL mode

Prefer PostgreSQL when any of these apply:

• Resource scale beyond etcd’s comfort zone. Large fleets of Agents, Models, Queries, MCPServers can push etcd object-count limits and slow the API server.
• Resource-size pressure. etcd’s 1.5 MiB per-object limit is a hard ceiling; Postgres rows are not.
• Persistence and operational tooling. Standard SQL backups, point-in-time recovery, CDC, and BI tooling become available.
• Multi-region or external DB strategy. Managed Postgres (RDS, Cloud SQL, Aiven) is easier to share across clusters than etcd.

Stay on etcd if:

• You want zero database operational burden.
• You don’t need the scale above and prefer the simpler single-binary controller.

Architecture in PostgreSQL mode

Two Helm releases work together:

• ark-controller — runs the reconciler. CRDs for ark.mckinsey.com/* are not installed in this mode.
• ark-apiserver — registers as an aggregated API server. The Kubernetes API server proxies all ark.mckinsey.com requests to it; it persists resources to Postgres.

When a user runs kubectl apply -f agent.yaml, the request flow is:

kubectl → kube-apiserver → APIService (v1alpha1.ark.mckinsey.com)
       → ark-apiserver → PostgreSQL (resources table)

The controller observes resources through the same K8s API path; it doesn’t talk to Postgres directly.

PostgreSQL requirements

The apiserver creates a logical replication slot to drive its watch stream, so the database must allow logical replication:

wal_level             = logical
max_replication_slots >= 1
max_wal_senders       >= 1

Minimum version

The hard minimum is PostgreSQL 15 — the apiserver checks server_version_num at startup and refuses to run against anything older. The floor tracks community support: 13 is already end-of-life, 14 follows in November 2026, and 15 is supported until November 2027. The storage integration suite runs against PostgreSQL 15 in CI to keep the floor honest; bump it together with the startup check when raising the minimum.

A user/role with permission to:

• CREATE TABLE, CREATE INDEX, CREATE PUBLICATION on the target database
• the REPLICATION attribute (ALTER ROLE ark REPLICATION). Postgres permits replication slots only to roles that hold this attribute, and no predefined role grants it. Managed services often withhold ALTER ROLE and expose an equivalent grant instead (for example rds_replication on AWS RDS) — check your provider’s documentation.

For managed services:

The connection settings the chart accepts are listed in ark/dist/chart-apiserver/values.yaml.

Connection pooling and PgBouncer

The apiserver holds a dedicated logical replication connection for its watch stream in addition to its regular query pool. Route only the query traffic through an external pooler, and keep the replication connection direct:

• PgBouncer in transaction or statement pooling mode is incompatible with logical replication. The replication connection must reach Postgres directly (or through a session-pooling/session mode pooler — for PgBouncer this requires version 1.21.0 or newer, which added replication-connection passthrough; older versions reject the replication protocol even in session mode). Point ARK_POSTGRES_HOST at a direct endpoint, or run a separate session-mode pooler for the apiserver.
• Managed poolers (RDS Proxy, Cloud SQL connectors) have the same constraint — replication slots require a session-scoped connection.

The apiserver already pools its own query connections (MaxOpenConns defaults to 40); an external transaction-pooling layer in front of the query path is optional and must not carry the replication connection.

Schema and replication slot

On first start, ark-apiserver creates:

• A single table, resources, with one row per Ark resource (Agent, Model, Query, Team, …). Columns include kind, namespace, name, uid, resource_version, JSONB columns for spec, status, labels, annotations, finalizers, owner_references, plus timestamps and a soft-delete flag (deleted_at).
• Indexes on (kind, namespace), (kind, namespace, name), a GIN index on labels, and a unique partial index on active (non-deleted) rows.
• Indexes on (kind, resource_version) and (resource_version), which serve the watch path: watches read changes as a resource-version range within a kind, and the bookmark refresh reads the highest resource version in the table.
• A publication and a logical replication slot, both named ark_cdc. The slot is what powers kubectl get -w and controller informers.

The slot is persistent: it survives apiserver restarts. It is dropped automatically on helm uninstall — see Uninstall and cleanup below.

WAL retention

ark_cdc publishes only the resources table, but WAL is cluster-wide: while the slot holds a position, Postgres keeps every segment from that point, including changes for unrelated tables and databases on the same instance. The consumer therefore acknowledges the server’s reported WAL position on each keepalive, not just its own last write, so an idle Ark install does not pin WAL on a busy instance.

Check that the slot is keeping up:

SELECT slot_name, active,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained
FROM pg_replication_slots WHERE slot_name = 'ark_cdc';

retained should stay small. Sustained growth means no replica is consuming the slot — check that a pod holds the ark-apiserver-leader lease and that active is true. Setting max_slot_wal_keep_size caps the damage: Postgres invalidates the slot instead of filling the disk, and the apiserver recreates it on the next restart.

Upgrading an existing deployment creates any index the schema is missing on the next apiserver start. Each build holds a write lock on resources until it completes — under half a second for a table of 100k rows, proportionally longer for a larger one.

Installing PostgreSQL mode

1. Prepare PostgreSQL

Provision a database with logical replication enabled, create the Ark database and user, and obtain the password.

2. Create the Kubernetes password secret

The chart references the password by secret name; you create it once:

kubectl create namespace ark-system
kubectl create secret generic ark-db-password \
  -n ark-system \
  --from-literal=password='<your-password>'

3. Configure .arkrc.yaml

The CLI reads the backend choice and connection details from .arkrc.yaml. You can place this file in either:

• ~/.arkrc.yaml (user-level, applies to all projects)
• ./.arkrc.yaml (project-level, takes precedence)

# .arkrc.yaml
storage:
  backend: postgresql
  postgresql:
    host: ark-storage.example.com
    port: 5432
    database: ark
    user: ark
    passwordSecretName: ark-db-password
    passwordSecretKey: password
    sslMode: require

sslMode accepts the standard libpq values: disable, require, verify-ca, verify-full. The default is require, so the controller→database connection is encrypted out of the box; set disable explicitly for a non-TLS database (e.g. local dev).

require encrypts the connection but does not verify the server’s identity, so it only protects against a passive eavesdropper on the wire. An active attacker who can present any TLS certificate — a rogue pod, DNS misroute, or MITM — can still impersonate the database. For production, use verify-full, which validates the server certificate and hostname; verify-ca validates the certificate but not the hostname. Against a publicly-trusted managed database (RDS, Cloud SQL) they work with no extra configuration because the controller image trusts the system CA bundle. For a private CA, self-signed certificate, or mutual TLS, mount the PEM files via a Secret on the apiserver chart:

kubectl create secret generic ark-pg-tls \
  --namespace ark-system \
  --from-file=ca.crt=ca.crt \
  --from-file=tls.crt=client.crt \
  --from-file=tls.key=client.key
 
helm upgrade --install ark-apiserver \
  oci://ghcr.io/mckinsey/agents-at-scale-ark/charts/ark-apiserver \
  --namespace ark-system \
  --set postgresql.host=ark-storage.example.com \
  --set postgresql.user=ark \
  --set postgresql.passwordSecretName=ark-db-password \
  --set postgresql.sslMode=verify-full \
  --set postgresql.sslSecretName=ark-pg-tls \
  --set postgresql.sslRootCertKey=ca.crt \
  --set postgresql.sslClientCertKey=tls.crt \
  --set postgresql.sslClientKeyKey=tls.key

The Secret is mounted read-only at /etc/ark/postgres-tls and the named keys are passed to libpq as sslrootcert/sslcert/sslkey. Only set the client cert/key keys if the database requires client-certificate authentication; sslRootCertKey alone is enough for verify-ca/verify-full against a private CA.

The --backend CLI flag and ARK_STORAGE_BACKEND env var override the config value, useful for testing the same code against a different backend without editing the file.

4. Install via the CLI

ark install

The CLI installs ark-controller with storage.backend=postgresql (which disables CRD installation) and ark-apiserver with the connection values from the config. cert-manager and Gateway API CRDs are installed as dependencies just as in etcd mode.

Install via raw Helm

If you prefer to skip the CLI, install the two charts directly with --set flags from the values you would have put in .arkrc.yaml:

helm upgrade --install ark-controller \
  oci://ghcr.io/mckinsey/agents-at-scale-ark/charts/ark-controller \
  --namespace ark-system --create-namespace \
  --set rbac.enable=true \
  --set storage.backend=postgresql
 
helm upgrade --install ark-apiserver \
  oci://ghcr.io/mckinsey/agents-at-scale-ark/charts/ark-apiserver \
  --namespace ark-system \
  --set postgresql.host=ark-storage.example.com \
  --set postgresql.user=ark \
  --set postgresql.passwordSecretName=ark-db-password \
  --set postgresql.sslMode=require

The apiserver chart’s postgresql.host, postgresql.user, and postgresql.passwordSecretName are required values — helm install will fail at template time if they are missing.

Connecting to the database

Several checks on this page are SQL. Connect with the values you configured above, reading the password from the secret you created in step 2:

export PGHOST=ark-storage.example.com   # postgresql.host
export PGPORT=5432                      # postgresql.port
export PGUSER=ark                       # postgresql.user
export PGDATABASE=ark                   # postgresql.database, default "ark"
export PGSSLMODE=require                # postgresql.sslMode
export PGPASSWORD=$(kubectl get secret ark-db-password -n ark-system \
  -o jsonpath='{.data.password}' | base64 -d)
 
psql -c 'SELECT count(*) FROM resources;'

If Postgres runs inside the cluster and is not reachable from your workstation, forward the port first and set PGHOST=127.0.0.1:

kubectl port-forward -n ark-system svc/<postgres-service> 5432:5432

Verifying the install

# No Ark CRDs in postgresql mode.
kubectl get crd | grep ark.mckinsey.com
# (no output)
 
# Both APIServices should report Available=True.
kubectl get apiservice v1alpha1.ark.mckinsey.com v1prealpha1.ark.mckinsey.com
 
# kubectl operates on Ark resources transparently.
kubectl get agents,models,queries -A

Create a smoke-test Agent to confirm the round trip lands in Postgres:

kubectl apply -f - <<EOF
apiVersion: ark.mckinsey.com/v1alpha1
kind: Agent
metadata:
  name: smoke
  namespace: default
spec:
  description: smoke test
  prompt: "You are a helpful assistant."
EOF

Then connect to Postgres (Connecting to the database) and confirm the row exists:

SELECT kind, namespace, name, uid FROM resources WHERE kind = 'Agent';

Migrating between storage backends

Switching backends does not move data. etcd and PostgreSQL hold separate copies of the ark.mckinsey.com resources, and neither backend can see the other’s. Export first, switch, then import.

Both directions use the same four steps: export, switch, import, verify. Two of them depend on the direction — the switch in step 2, and the backend check in step 4.

Before you start

• Ark CLI v0.1.68 or newer (first tagged v0.1.68-rc). Install it with npm install -g @agents-at-scale/ark (see Ark CLI) and check with ark --version. Older versions export server-managed fields that the destination rejects or stores as stale, and they include MCPServer-generated Tools and A2AServer-generated Agents that should not be imported. ark import --upsert, which makes the procedure re-runnable, arrived in v0.1.67.
• kubectl pointed at the right cluster. Step 1 reads every namespace and step 2 can delete every Ark CRD, so confirm the context before you begin.
• psql access to the database for the check in step 4. See Connecting to the database.
• jq, yq, and bc, used by the export and count commands below.
• Time to finish steps 1 to 3 in one go. Resources created between the export and the switch are not migrated.

What moves

Only ark.mckinsey.com resources are backend-specific. Secrets, ConfigMaps, Deployments, and other core Kubernetes objects stay in etcd in both modes. Leave them alone.

Tools generated by an MCPServer and Agents generated by an A2AServer are left out of the export. The destination controller recreates them once you import the parent resource.

Queries and A2ATasks do not transfer, and the loss is permanent. Their status and resource versions belong to the backend that produced them, so the new backend starts with no query history. The old copies are not deleted — they remain in etcd behind the retained CRDs, or in the PostgreSQL resources table — but nothing reads them while the other backend is serving. If you need the history, snapshot it before you switch: kubectl get queries,a2atasks -A -o yaml > query-history.yaml, or pg_dump the resources table.

1. Export before switching

Run this while the source backend is still serving. ark export reads one namespace at a time (the current context namespace unless you pass -n) and overwrites its output file, so give each namespace its own file in a directory named for the backend you are leaving. The reverse migration writes its own set and must not clobber this one.

Use an empty directory. mkdir without -p fails if one is already there, which is what you want: a leftover directory from an earlier attempt still holds files for namespaces that no longer exist, and step 3 imports everything it finds.

# Confirm you are on the intended cluster. Step 2 can delete every Ark CRD.
kubectl config current-context
 
outdir=ark-export-etcd    # ark-export-postgres when migrating the other way
mkdir "$outdir"
 
for ns in $(kubectl get agents,models,teams,tools,mcpservers,a2aservers -A \
  -o jsonpath='{.items[*].metadata.namespace}' | tr ' ' '\n' | sort -u); do
  ark export -n "$ns" -t tools,models,agents,teams,mcpservers,a2aservers \
    -o "$outdir/$ns.yaml"
done

-t omits secrets. Secrets are core Kubernetes objects served from etcd in both modes, so they never move. Because ark import uses kubectl create, importing them back onto the cluster they already live on fails the import with AlreadyExists.

ark export cannot handle ExecutionEngines and Memories, so remove the server-managed fields yourself. A plain kubectl get -o yaml includes uid, resourceVersion, and status, which the destination backend either rejects or stores as stale:

kubectl get executionengines,memories -A -o yaml \
  | yq 'del(
      .items[].metadata.uid,
      .items[].metadata.resourceVersion,
      .items[].metadata.generation,
      .items[].metadata.creationTimestamp,
      .items[].metadata.managedFields,
      .items[].metadata.ownerReferences,
      .items[].metadata.finalizers,
      .items[].metadata.annotations."kubectl.kubernetes.io/last-applied-configuration",
      .items[].status
    )' > "$outdir/extras.yaml"

This del list mirrors the fields ark export removes (SERVER_MANAGED_METADATA_FIELDS in tools/ark-cli/src/commands/export/index.ts). If an import later rejects a field, compare the two lists.

Check the export before running step 2, which can be irreversible. ark export reports failures on stderr but still exits 0, and it writes no file for a namespace that holds nothing — so a failed run looks like a clean one. Compare three counts:

outdir=ark-export-etcd    # same directory as above, if you started a new shell
 
# 1. Everything on the cluster, generated resources included. Step 4 reuses this.
kubectl get agents,models,teams,tools,mcpservers,a2aservers -A --no-headers | wc -l
 
# 2. What the export should contain. ark export drops only Tools generated by an
# MCPServer and Agents generated by an A2AServer, matched on the generated label
# or the ownerReference. Every other kind is exported in full.
{
  kubectl get models,teams,mcpservers,a2aservers -A --no-headers | wc -l
  kubectl get tools -A -o json | jq '[.items[] | select(
      .metadata.labels["mcp/server"] == null and
      ([.metadata.ownerReferences[]? | select(.kind == "MCPServer")] | length == 0)
    )] | length'
  kubectl get agents -A -o json | jq '[.items[] | select(
      .metadata.labels["a2a/server"] == null and
      ([.metadata.ownerReferences[]? | select(.kind == "A2AServer")] | length == 0)
    )] | length'
} | paste -sd+ - | bc
 
# 3. What the files contain: one 'kind:' line per resource, plus one for the
# List wrapper in extras.yaml. Items inside that wrapper are indented, so
# '^kind:' does not count them twice.
cat "$outdir"/*.yaml | grep -c '^kind:'

Count 3 should equal count 2 plus one. Do not compare it against count 1: generated Tools and Agents are excluded on purpose, so on a cluster running an MCPServer the export is smaller than the cluster inventory. Note count 1 down for step 4, which runs after the controller has recreated them.

If count 3 does not match, stop and fix the export before switching.

Count 3 is too low. The export missed resources. Common causes, most likely first:

• A namespace produced no file. ark export writes nothing and warns no resources found to export when a namespace comes back empty. Compare ls "$outdir" with the namespaces the loop covered.
• No files at all. Count 3 reports 0, and some shells also print no matches found for the unmatched glob. The 0 is the real answer: check that the namespace loop ran.
• The export failed part-way. It prints export failed: on stderr and still exits 0, so re-run the loop and read every line. The file is written only after all kinds are fetched, so a failure loses the whole namespace, not one kind.
• Missing permissions. ark export calls kubectl get. A kind your credentials cannot list fails that namespace instead of producing a partial file.

Fix the cause, then re-run the export for the affected namespaces. Each run overwrites its own file, so repeating is safe.

Count 3 is too high. The export kept resources that count 2 excluded. This usually means an old CLI: the generated-resource exclusion arrived in #3105 , first tagged v0.1.68-rc. Earlier versions write MCPServer-generated Tools and A2AServer-generated Agents into the files. Upgrade the CLI, then re-run step 1 into a new directory. Importing generated resources creates duplicates: they arrive without the ownerReference that links them to their parent, so the destination controller adds its own copies alongside them and nothing cleans up either set.

For every run, ark export prints found N <kind>, excluded N controller-managed resources, and exported N resources to <file>. Read those lines — they record what the command actually did.

Keep the export directory until the migration is verified; it is your rollback. The files hold desired state only, so they apply cleanly to either backend.

2. Switch the backend

etcd → PostgreSQL. Follow Installing PostgreSQL mode. What happens to the etcd copies depends on how you switch:

• ark install and raw Helm keep them. The controller chart templates the CRDs out when storage.backend=postgresql, but they carry helm.sh/resource-policy: keep (crd.keep, default true), so Helm skips the deletion. The etcd copies stay intact but unreachable: the v1alpha1.ark.mckinsey.com APIService now claims that group-version path, and the aggregation layer  proxies every request to ark-apiserver.
• DevSpace deletes them. Both the dev and deploy pipelines run kubectl delete on every ark.mckinsey.com CRD and clear their finalizers before installing the apiserver (ark/devspace.yaml). That destroys the etcd copies with no recovery path. Never point either at a cluster whose resources you have not exported.

Deleting a CRD deletes every custom resource of that kind. If you set crd.keep=false, the ark install path becomes destructive in the same way.

PostgreSQL → etcd. The switch is the command sequence in Rolling back: deregister the APIServices, then reinstall the controller with storage.backend=etcd. Do it before you import. While the APIServices are registered, the aggregation layer proxies every write to ark-apiserver, so the import would land back in Postgres and step 4 would verify the wrong backend.

3. Import

Import every file in the directory from step 1: one per namespace, plus extras.yaml.

outdir=ark-export-etcd    # the directory step 1 wrote
for f in "$outdir"/*.yaml; do
  ark import "$f" || { echo "import failed: $f"; break; }
done

ark import exits non-zero on the first resource it cannot import, but a plain loop moves straight on to the next file. Stop on failure, as above. Otherwise a namespace that failed to import is easy to miss, and step 4 is the only thing left to catch it.

ark import uses kubectl create, so it fails if a resource already exists. To resume a partial import or re-run the migration, add --upsert, which switches to kubectl apply:

for f in "$outdir"/*.yaml; do
  ark import "$f" --upsert || { echo "import failed: $f"; break; }
done

--upsert overwrites the spec of resources already in the destination. Imports can still fail on immutable-field changes and admission-webhook rejections; the command reports how many resources it created, configured, and left unchanged before the error.

One class of rejection resolves itself on a second pass. Ark validates that referenced resources exist, so a Team whose member is another Team is rejected if that member has not been created yet — ark export orders Agents before Teams, but not Teams among themselves. Re-run the loop with --upsert: the first pass creates the members, the second pass accepts the Teams that referenced them.

kubectl apply does not prune, so --upsert only creates or updates. A resource deleted after the export was taken, or deleted in the destination while the other backend was serving, comes back. Delete those explicitly once the import is done.

4. Verify

These counts apply in both directions — they go through the Kubernetes API, whichever backend is serving it.

# Compare against count 1 from step 1: the full inventory, generated resources
# included. Give the controller time to reconcile before trusting a shortfall.
kubectl get agents,models,teams,tools,mcpservers,a2aservers -A --no-headers | wc -l
 
# MCPServer-generated Tools and A2AServer-generated Agents reappear
# once the controller reconciles their parents.
kubectl get tools,agents -A

Going etcd → PostgreSQL, this should equal count 1. Going PostgreSQL → etcd it can be higher, and that is expected: if the CRDs were kept, etcd still holds everything from before the original switch, and you have just imported the PostgreSQL-era resources on top. Only a number below count 1 means something is missing.

The backend check differs by direction.

etcd → PostgreSQL. Confirm the rows landed in Postgres — see Connecting to the database for the connection:

SELECT kind, count(*) FROM resources WHERE deleted_at IS NULL GROUP BY kind;

PostgreSQL → etcd. Do not run that query. helm uninstall ark-apiserver leaves the resources table in place (see Uninstall and cleanup), so it still returns the pre-rollback rows and reads as a pass even if nothing reached etcd. Confirm etcd is serving instead:

# The aggregation layer no longer claims the group-version path.
kubectl get apiservice v1alpha1.ark.mckinsey.com v1prealpha1.ark.mckinsey.com
# (NotFound)
 
# The CRDs are back, so the counts above came from etcd.
kubectl get crd -o name | grep ark.mckinsey.com

Finally, run a query against a migrated agent. This is the check that matters: it proves the controller can resolve the agent’s model, tools, and secret references in the new backend, which counting rows does not.

kubectl get agents -A                      # pick a migrated agent
ark query agent/<name> "Reply with OK."

A response means the migration works end to end. An error naming a missing model, tool, or secret means that reference did not survive — re-check the export for that kind.

Rolling back

Roll back through the same four steps. Run step 1 against PostgreSQL first, into ark-export-postgres/. Anything created while it was serving exists only there, and the ark-export-etcd/ directory holds the pre-switch state you must not overwrite. Then run the switch below, and import ark-export-postgres/ in step 3.

# The chart owns both APIServices, so this deregisters them.
helm uninstall ark-apiserver -n ark-system
kubectl get apiservice v1alpha1.ark.mckinsey.com v1prealpha1.ark.mckinsey.com
# (NotFound)
 
# Re-apply the rest of your controller values alongside the backend flip.
helm upgrade --install ark-controller \
  oci://ghcr.io/mckinsey/agents-at-scale-ark/charts/ark-controller \
  --namespace ark-system \
  --set rbac.enable=true \
  --set storage.backend=etcd

Removing the APIServices releases the group-version path. If the CRDs were kept, the original etcd resources are reachable again as they were before the switch; if they were deleted, the etcd-mode controller chart reinstalls the CRDs empty. Either way, import ark-export-postgres/ to bring across everything created while PostgreSQL was serving. Use --upsert when the CRDs were kept, since the pre-switch resources are already there. Re-create any ArkConfig by hand: the aggregated apiserver never served it, so it is not in the export.

helm uninstall ark-apiserver drops the ark_cdc replication slot but leaves the resources table, so the PostgreSQL copy survives a rollback and the migration can be retried in either direction. See Uninstall and cleanup.

Security model

The aggregated apiserver enforces the same access control as the rest of the cluster:

• Delegated authentication and authorization. Every request is authenticated against the kube-apiserver (TokenReview and requestheader/front-proxy identity) and authorized via SubjectAccessReview, so Kubernetes RBAC on ark.mckinsey.com resources applies to direct service access as well as to the kubectl path. Health endpoints (/healthz, /readyz, /livez) stay unauthenticated. Set ARK_APISERVER_AUTH_MODE=off (for example via extraEnv) only for local development outside a cluster.
• Verified serving TLS. With certManager.enabled (the default), cert-manager issues the serving certificate for the ark-apiserver service, mounts it into the pod, and injects the CA into both APIServices — the kube-apiserver verifies the aggregated apiserver’s identity instead of insecureSkipTLSVerify. The certificate rotates through cert-manager and the server reloads it without a restart. Setting certManager.enabled=false restores the previous behavior (ephemeral self-signed certificate, unverified proxy channel) for clusters without cert-manager.
• Optional NetworkPolicy. networkPolicy.enabled=true restricts ingress to the serving and health ports; use networkPolicy.extraIngressFrom to pin the allowed sources for the serving port (the origin of kube-apiserver traffic depends on your CNI, which is why this is off by default).
• Audit log. audit.enabled (default true) makes the aggregated apiserver emit its own Kubernetes audit trail as JSON on stdout, at Metadata level by default. Because the records are produced in-process they cover the direct service path too. A policy file is mandatory when audit is enabled — without one the upstream backend records nothing, so the apiserver refuses to start rather than report audit as active while emitting nothing; the chart always mounts one. Tune with audit.level or supply a full audit.policy. Note that Request/RequestResponse levels put request and response bodies — including Query .spec.input — into your log pipeline.
• Policy enforcement. The host kube-apiserver runs no webhook chain on aggregated resources, so this apiserver enforces policy itself, two independent ways. A native ValidatingAdmissionPolicy (CEL) is evaluated in-process (policy.cel.enabled, default true) and requires the host cluster at k8s ≥1.30. The webhook admission plugins can also be run here (policy.thirdPartyWebhooks.enabled, default false), which makes Kyverno/Gatekeeper webhook configurations fire on Ark resources; it is off by default because it puts a synchronous webhook call on every write. Each mechanism is best-effort by default: on an older host, if the startup discovery probe never succeeds, or if the ServiceAccount lacks the watch RBAC, the apiserver logs the reason and serves with that mechanism off (Ark’s own in-process validation and audit still apply). Set policy.cel.required=true or policy.thirdPartyWebhooks.required=true to make that a startup failure instead. Policies that use paramKind need read access to the parameter resource via policy.extraParamRules unless the parameters live in ConfigMaps or Secrets, which are already granted.

Availability and multi-replica behaviour

The chart defaults to a single replica. Understand what that means before running it in production: when the aggregated apiserver is unavailable, the ark.mckinsey.com APIService goes unavailable, which degrades kube-apiserver API discovery and the garbage-collection and namespace controllers cluster-wide — a standard Kubernetes aggregation-layer failure mode, not something specific to Ark. A single replica is therefore a cluster-wide single point of failure during pod restarts, node drains, and crashes.

For production, run two replicas and enable the disruption budget:

--set replicas=2 --set podDisruptionBudget.enabled=true

How multiple replicas behave:

• API serving is active-active. Every replica serves reads and writes directly against Postgres; the Service load-balances across them. Killing any one pod leaves the APIService available.
• The watch stream is single-consumer by design. Only the replica holding the ark-apiserver-leader lease runs the WAL consumer that drives real-time watch events (the logical replication slot admits one connection; the slot’s active flag is a backstop even without the lease). Non-leader replicas do not touch the slot.
• Watches served by non-leader replicas are relist-bound. Without the WAL stream, a non-leader refreshes its watchers on a periodic relist (up to ~120 seconds stale). Controllers tolerate this — informers resync — but if consistently low watch latency matters, keep a single replica and accept the SPOF trade-off, or route watch-heavy clients through the kubectl path.
• Failover: if the leader dies, a surviving replica acquires the lease (roughly the 15s lease duration) and starts the WAL consumer; the slot position is persistent, so no events are lost across the handover.

Backups and restore

Treat the resources table like any other application table:

• Use your provider’s automated backups or pg_dump for ad-hoc snapshots.
• A point-in-time restore restores Ark state to that moment. Take care to also drop and recreate the ark_cdc replication slot after a restore so the apiserver starts a fresh watch stream.
• Cluster-side state (Pods, Deployments owned by Ark) is not restored by a Postgres restore — only the declarative resources are.

Uninstall and cleanup

helm uninstall runs a post-delete hook Job that drops the ark_cdc replication slot and publication. Without this cleanup, an orphaned slot would pin WAL retention and can fill the disk. The Job connects with the same credentials as the apiserver, so the password secret must still exist when you uninstall.

Confirm in the database that the slot is gone rather than trusting the Job’s log line — both queries should return no rows:

SELECT slot_name, active, wal_status FROM pg_replication_slots WHERE slot_name = 'ark_cdc';
SELECT pubname FROM pg_publication WHERE pubname = 'ark_cdc';

If the hook Job could not run (for example the database was unreachable, you removed the release with --no-hooks, or the password secret was deleted first), drop the slot manually:

SELECT pg_drop_replication_slot('ark_cdc');
DROP PUBLICATION IF EXISTS ark_cdc;

If the slot was invalidated (wal_status = 'lost', typically after max_slot_wal_keep_size was exceeded), the apiserver drops and recreates it automatically on startup.

The resources table itself is yours — drop it manually if you are decommissioning the database, or keep it for forensic queries.

Troubleshooting

helm install fails with postgresql.host is required. You ran the apiserver chart without supplying connection details. Set storage.postgresql in .arkrc.yaml (the CLI passes these through) or use --set postgresql.host=… --set postgresql.user=… --set postgresql.passwordSecretName=… for raw helm.

ark install fails with missing 'storage.postgresql' block. You set storage.backend: postgresql in .arkrc.yaml but didn’t add the storage.postgresql block, or it’s missing host/user/passwordSecretName. Fill in the required fields.

Apiserver pod is CreateContainerConfigError. The pod is referencing a secret that doesn’t exist. Confirm the secret named in postgresql.passwordSecretName exists in the same namespace as the release.

Apiserver crashes with failed to connect to database: dial tcp … connect: connection refused. The host/port is wrong, the database isn’t up yet, or a NetworkPolicy is blocking egress. The pod will restart and retry.

Apiserver logs error retrieving resource lock. Leader election can’t reach the Kubernetes API. Usually a transient startup issue; if it persists, check the ServiceAccount, RBAC bindings, and any egress restrictions.

APIService stuck at False (FailedDiscoveryCheck). The aggregator can’t reach the apiserver Service. Check kubectl get svc -n ark-system ark-apiserver, verify pods are 1/1, and confirm no NetworkPolicy blocks port 6443 from the kube-apiserver.

kubectl get agents returns “the server doesn’t have a resource type …”. The APIService is not registered or not Available. Look at kubectl get apiservice v1alpha1.ark.mckinsey.com -o yaml.

Resources don’t appear after kubectl apply, but no error. Check kubectl get events -A. Confirm the apiserver pod is running and the replication slot exists in Postgres (SELECT slot_name, active, wal_status FROM pg_replication_slots).