Ark Broker

In-memory storage service with Server-Sent Events (SSE) streaming support for Ark queries.

• In-memory message storage by default, optionally persisted to disk / persistent volume, or backed by Postgres for the messages store
• Completion chunks held in memory by default, optionally backed by Redis Streams for live streaming across broker replicas
• Implements the Memory API
• Supports streaming queries

This service provides a basic but complete memory system for an Ark cluster, as well as offering the up-coming streaming APIs which enable real-time streaming responses to queries.

Installation

Install via helm:

helm install ark-broker \
    ghcr.io/mckinsey/agents-at-scale-ark/charts/ark-broker

Or for local development:

cd services/ark-broker
 
# Deploy the service...
devspace deploy
 
# ...or run in dev mode with live-reload.
devspace dev

Configuration

All configuration options are documented in the Helm chart. Key configuration is:

Environment variables:

The Helm chart can optionally configure a persistent volume for data storage by setting persistence.enabled=true.

Message backend (Postgres)

By default the broker keeps all stores in memory. The messages store can opt in to Postgres so messages survive pod restarts. Only the messages store is affected — chunks, traces, events, and sessions remain in memory (or file-based, as above).

This is separate from the controller’s PostgreSQL storage backend, which stores Ark resources (CRDs) via the aggregated API server. The two use different databases and serve different purposes.

Enable it in the Helm chart:

backends:
  message: postgres
 
database:
  url: "postgres://user:password@host:5432/ark_broker"

When enabled, the chart renders a migrate/migrate init container (image ark-broker-migrate) that applies pending schema migrations before the broker starts. The broker image itself carries no migration tooling and never runs DDL.

Key environment variables:

Messages carry a TTL-based expiry: each message is written with an expires_at snapshot and filtered out once it elapses. The expiry aligns with the originating query’s TTL (spec.ttl, defaulting to ArkConfig.spec.queryTTL with a 720h fallback), and falls back to MESSAGE_VISIBILITY_TTL_SECONDS when no per-message TTL is supplied.

Messages are also removed when their originating Query is deleted: the controller issues DELETE /queries/{queryId}/messages, which hard-deletes every message for that query across all conversations. On the Postgres backend this is a single indexed delete. This complements the TTL above — TTL is time-based, this is triggered by the Query’s removal. See Query → Deletion and cleanup.

Operation events are cleaned up the same way, but through a separate path: events aren’t part of the Memory API, so the controller calls DELETE /events/{queryId} directly on the broker endpoint discovered via the ark-config-broker ConfigMap, rather than going through a Memory resource. Events are keyed by the Query’s UID rather than its name, so {queryId} here is the UID.

See the service README  for local development with devspace, running migrations, and integration tests.

Securing the Postgres connection

In-transit encryption (sslmode=require)

No certificates to mount. Append ?sslmode=require to the connection URL:

database:
  url: "postgres://user:password@host:5432/ark_broker?sslmode=require"

CA verification (sslmode=verify-full) — recommended for production

Create a Kubernetes Secret containing the Postgres CA certificate:

kubectl create secret generic pg-tls \
  --from-file=ca.crt=/path/to/postgres-ca.crt

Then enable the TLS mount:

database:
  url: "postgres://user:password@host:5432/ark_broker?sslmode=verify-full"
  tls:
    enabled: true
    secretName: pg-tls
    mountPath: /etc/pg-ssl

When tls.enabled=true, the chart automatically sets DATABASE_SSL_ROOT_CERT_PATH={mountPath}/ca.crt in the broker container. The broker reads the CA certificate from that path and passes it to the Postgres driver for server certificate verification.

Do not add sslrootcert= to DATABASE_URL. The postgres.js driver does not read certificate files from URL parameters — it forwards unknown query parameters to PostgreSQL as SET commands, which the server rejects. Certificate loading happens through DATABASE_SSL_ROOT_CERT_PATH instead.

database.migrateUrl

The migrate init container uses golang-migrate, which reads sslrootcert from the URL as a client-side file path (different from postgres.js). When using sslmode=verify-full, set database.migrateUrl separately so the migrate container can verify the server certificate:

database:
  url: "postgres://user:password@host:5432/ark_broker?sslmode=verify-full"
  migrateUrl: "postgres://user:password@host:5432/ark_broker?sslmode=verify-full&sslrootcert=/etc/pg-ssl/ca.crt"
  tls:
    enabled: true
    secretName: pg-tls
    mountPath: /etc/pg-ssl

When migrateUrl is not set, the migrate container falls back to database.url.

Hiding the connection URL from Helm values

database.url embeds the password in plain text, which is visible in kubectl get deployment -o yaml. For production, store the full URL in a Secret and reference it instead:

kubectl create secret generic broker-db-url \
  --from-literal=database-url="postgres://user:password@host:5432/ark_broker?sslmode=require"

database:
  urlSecretRef:
    name: broker-db-url
    key: database-url   # default key; can be omitted

urlSecretRef takes priority over database.url and applies to both the broker container and the migrate init container.

Chunk backend (Redis)

By default the broker holds completion chunks in process memory, served from a single replica. The chunks store can opt in to Redis Streams so that chunks produced by one broker replica can be streamed live to a client whose SSE connection landed on another replica. Only the chunks store is affected — messages (unless on Postgres), traces, events, and sessions remain in memory (or file-based, as above).

Each query gets its own Redis Stream key ({prefix}:chunks:{queryId}), plus a capped global key ({prefix}:chunks:all) backing the paginated and watch endpoints. Live tail uses XREAD BLOCK so chunks arrive cross-replica.

Enable it in the Helm chart:

backends:
  chunk: redis
 
redis:
  url: "redis://:password@redis-host:6379"

Key environment variables:

See the service README  for local development with devspace and integration tests.

Securing the Redis connection

In-transit encryption (rediss://)

Use the rediss:// scheme in the connection URL to connect over TLS. For a Redis endpoint with a CA trusted by the system store, no certificates need to be mounted.

CA verification (self-signed certs) — recommended for production

Create a Kubernetes Secret containing the Redis CA certificate:

kubectl create secret generic redis-tls \
  --from-file=ca.crt=/path/to/redis-ca.crt

Then enable the TLS mount:

redis:
  url: "rediss://:password@redis-host:6380"
  tls:
    enabled: true
    secretName: redis-tls

When tls.enabled=true, the chart mounts the secret and sets REDIS_TLS_CA_CERT_PATH in the broker container automatically. The secret must contain a ca.crt key.

Hiding the connection URL from Helm values

redis.url embeds the password in plain text, which is visible in kubectl get deployment -o yaml. For production, store the full URL in a Secret and reference it instead:

kubectl create secret generic broker-redis-url \
  --from-literal=redis-url="rediss://:password@redis-host:6380"

redis:
  urlSecretRef:
    name: broker-redis-url
    key: redis-url   # default key; can be omitted

The password can also be supplied separately from the URL via redis.passwordSecretRef, sourced into REDIS_PASSWORD.

Sessions Endpoint

The /sessions endpoint provides a live, event-sourced index of active sessions and their queries. It materializes state from the existing event and message streams — no additional data ingestion is needed.

Set SESSIONS_FILE_PATH to enable persistence to disk.