Ark Broker

In-memory storage service with Server-Sent Events (SSE) streaming support for Ark queries.

• In-memory message storage by default, optionally persisted to disk / persistent volume, or backed by Postgres for the messages, operation events, and sessions stores
• Completion chunks held in memory by default, optionally backed by Redis Streams for live streaming across broker replicas
• Implements the Memory API
• Supports streaming queries

This service provides a basic but complete memory system for an Ark cluster, as well as offering the up-coming streaming APIs which enable real-time streaming responses to queries.

Installation

Install via helm:

helm install ark-broker \
    ghcr.io/mckinsey/agents-at-scale-ark/charts/ark-broker

Or for local development:

cd services/ark-broker
 
# Deploy the service...
devspace deploy
 
# ...or run in dev mode with live-reload.
devspace dev

Configuration

All configuration options are documented in the Helm chart. Key configuration is:

Environment variables:

Backend selection (MESSAGE_BACKEND, EVENT_BACKEND, CHUNK_BACKEND) and its Postgres and Redis settings are documented in the backend sections below. The request body limit is fixed at 10MB and is not configurable.

The Helm chart can optionally configure a persistent volume for data storage by setting persistence.enabled=true.

Heap sizing

V8 sizes its default heap from the host’s memory rather than the container limit, so on a large node it may pick a cap unrelated to the memory the pod was actually given. Where that cap lands below the limit, the broker exhausts its heap and exits with FATAL ERROR: Ineffective mark-compacts near heap limit while the container still has headroom — a Node-initiated exit, not a Linux OOMKill.

The chart sets NODE_OPTIONS=--max-old-space-size=... from app.resources.limits.memory, taking the lesser of 85% and limit - 128Mi — 870MB at the 1024Mi default. The margin is deliberate: the flag caps only the old space, V8’s total heap ceiling lands roughly 24MB above it (measured under a container memory limit; without one V8 sizes the young generation from host memory and the delta is closer to 96MB — not a case that arises here, since the flag is only emitted when limits.memory is set), and non-heap memory (native buffers, sockets, protobuf parsing) sits above that, so a larger value trades a clean V8 heap limit for a kernel OOMKill. Because that 24MB is a fixed addition rather than a proportion, the limit - 128Mi term is what keeps small limits safe; it dominates below roughly 853Mi. Under a 256Mi limit that term yields a heap too small for the process to start, so NODE_OPTIONS is omitted entirely rather than rendered at an unusable size — 256Mi is the smallest limit the broker is sized for. Raising the limit raises the heap with it:

helm upgrade --install ark-broker services/ark-broker/chart \
  --set app.resources.limits.memory=2Gi

Set app.nodeOptions to override the computed value. If app.resources.limits.memory is unset, NODE_OPTIONS is not emitted and V8 falls back to its own default.

Heap pressure scales with the in-memory stores, so limits.maxMessages, limits.maxChunks, limits.maxSpans and limits.maxEvents are the other lever. Moving messages or sessions to Postgres, or chunks to Redis, takes those stores off the heap; traces stay in memory regardless. On the default in-memory backend the sessions index has no size limit — it grows with the queries it has seen until DELETE /sessions purges it, which is the one heap consumer with no cap.

Metrics

The broker exposes Prometheus metrics at GET /metrics on the service port:

A gauge is registered only for stores actually held in the heap — with MESSAGE_BACKEND=postgres, broker_messages_count is absent rather than reported as zero, and the two session gauges are absent under SESSIONS_BACKEND=postgres. Traces have no backend to move to, so broker_spans_count is always present.

Session heap use tracks queries rather than sessions: each session holds every query it has seen, plus derived conversation and participant summaries. A small broker_sessions_count alongside a large broker_session_queries_count means a few long-lived sessions are holding the memory.

Standard Node process metrics are included, notably process_resident_memory_bytes and nodejs_heap_size_used_bytes. Comparing the cache gauges against heap usage shows which store is consuming the heap before it fills.

Nothing scrapes the endpoint by default. Set prometheus.enable=true to render a ServiceMonitor for it, which requires the Prometheus Operator in the cluster:

helm upgrade --install ark-broker services/ark-broker/chart \
  --set prometheus.enable=true

Adjust the scrape cadence with prometheus.interval (default 30s). Without the operator, reach the endpoint directly with kubectl port-forward svc/ark-broker 8080:8080 and curl localhost:8080/metrics.

Message backend (Postgres)

By default the broker keeps all stores in memory. The messages store can opt in to Postgres so messages survive pod restarts. This section covers the messages store; operation events and sessions have their own Postgres opt-ins (see Event backend (Postgres) and Sessions backend (Postgres)) and completion chunks can opt in to Redis (see Chunk backend (Redis)), while traces remain in memory (or file-based, as above).

This is separate from the controller’s PostgreSQL storage backend, which stores Ark resources (CRDs) via the aggregated API server. The two use different databases and serve different purposes.

Enable it in the Helm chart:

backends:
  message: postgres
 
database:
  url: "postgres://user:password@host:5432/ark_broker"

When enabled, the chart renders a migrate/migrate init container (image ark-broker-migrate) that applies pending schema migrations before the broker starts. The broker image itself carries no migration tooling and never runs DDL.

Key environment variables:

Messages carry a TTL-based expiry: each message is written with an expires_at snapshot and filtered out once it elapses. The expiry aligns with the originating query’s TTL (spec.ttl, defaulting to ArkConfig.spec.queryTTL with a 720h fallback), and falls back to MESSAGE_VISIBILITY_TTL_SECONDS when no per-message TTL is supplied.

Messages are also removed when their originating Query is deleted: the controller issues DELETE /queries/{queryId}/messages, which hard-deletes every message for that query across all conversations. On the Postgres backend this is a single indexed delete. This complements the TTL above — TTL is time-based, this is triggered by the Query’s removal. See Query → Deletion and cleanup.

See the service README  for local development with devspace, running migrations, and integration tests.

Securing the Postgres connection

In-transit encryption (sslmode=require)

No certificates to mount. Append ?sslmode=require to the connection URL:

database:
  url: "postgres://user:password@host:5432/ark_broker?sslmode=require"

CA verification (sslmode=verify-full) — recommended for production

Create a Kubernetes Secret containing the Postgres CA certificate:

kubectl create secret generic pg-tls \
  --from-file=ca.crt=/path/to/postgres-ca.crt

Then enable the TLS mount:

database:
  url: "postgres://user:password@host:5432/ark_broker?sslmode=verify-full"
  tls:
    enabled: true
    secretName: pg-tls
    mountPath: /etc/pg-ssl

When tls.enabled=true, the chart automatically sets DATABASE_SSL_ROOT_CERT_PATH={mountPath}/ca.crt in the broker container. The broker reads the CA certificate from that path and passes it to the Postgres driver for server certificate verification.

Do not add sslrootcert= to DATABASE_URL. The postgres.js driver does not read certificate files from URL parameters — it forwards unknown query parameters to PostgreSQL as SET commands, which the server rejects. Certificate loading happens through DATABASE_SSL_ROOT_CERT_PATH instead.

database.migrateUrl

The migrate init container uses golang-migrate, which reads sslrootcert from the URL as a client-side file path (different from postgres.js). When using sslmode=verify-full, set database.migrateUrl separately so the migrate container can verify the server certificate:

database:
  url: "postgres://user:password@host:5432/ark_broker?sslmode=verify-full"
  migrateUrl: "postgres://user:password@host:5432/ark_broker?sslmode=verify-full&sslrootcert=/etc/pg-ssl/ca.crt"
  tls:
    enabled: true
    secretName: pg-tls
    mountPath: /etc/pg-ssl

When migrateUrl is not set, the migrate container falls back to database.url.

Hiding the connection URL from Helm values

database.url embeds the password in plain text, which is visible in kubectl get deployment -o yaml. For production, store the full URL in a Secret and reference it instead:

kubectl create secret generic broker-db-url \
  --from-literal=database-url="postgres://user:password@host:5432/ark_broker?sslmode=require"

database:
  urlSecretRef:
    name: broker-db-url
    key: database-url   # default key; can be omitted

urlSecretRef takes priority over database.url and applies to both the broker container and the migrate init container.

Event backend (Postgres)

Operation events — the controller lifecycle events emitted as a query executes (QueryExecutionStart, LLMCallComplete, and so on) — are kept in memory by default. The events store can opt in to Postgres so they survive pod restarts, independently of the messages store.

The event backend shares the same DATABASE_URL, connection pool, and migrate/migrate init container as the Message backend: the events and messages tables live in one schema. You can enable either backend on its own or both together.

Enable it in the Helm chart:

backends:
  event: postgres        # combine with `message: postgres` to persist both
 
database:
  url: "postgres://user:password@host:5432/ark_broker"

Key environment variables:

The other DATABASE_* variables (DATABASE_POOL_MAX, DATABASE_CONNECT_TIMEOUT_MS, DATABASE_STATEMENT_TIMEOUT_MS, DATABASE_DEBUG_QUERIES, DATABASE_SSL_ROOT_CERT_PATH) are shared with the message backend — see the table above. Securing the connection (TLS, migrateUrl, hiding the URL in a Secret) works identically; see Securing the Postgres connection.

Events carry a TTL-based expiry: each event is written with an expires_at snapshot and filtered out once it elapses. The expiry aligns with the originating query’s TTL (spec.ttl, defaulting to ArkConfig.spec.queryTTL with a 720h fallback), and falls back to EVENT_VISIBILITY_TTL_SECONDS when no per-query TTL is supplied.

Events are also removed when their originating Query is deleted, through a path separate from messages: events aren’t part of the Memory API, so the controller calls DELETE /events/{queryId} directly on the broker endpoint discovered via the ark-config-broker ConfigMap, rather than going through a Memory resource. Events are keyed by the Query’s UID rather than its name, so {queryId} here is the UID. See Query → Deletion and cleanup.

Sessions backend (Postgres)

Sessions — the materialized index of participants, conversations, status, and per-query progress that backs the dashboard’s session list and live session panel — are kept in memory by default. The sessions store can opt in to Postgres so it survives pod restarts and is shared by every broker replica.

Unlike the messages and events stores, this backend cannot be enabled on its own: a session is materialized from both of those streams, so backends.sessions: postgres also requires backends.message: postgres and backends.event: postgres. The broker validates this at startup and exits with an explanatory error otherwise. The reason is sequence numbering: the watermarks that stop an out-of-order write from regressing a session are the sequence numbers of the messages and events themselves, and an in-memory stream numbers its items per process starting from 1 — two replicas would feed the same sequences into one shared watermark and each would discard the other’s writes.

The sessions backend shares the same DATABASE_URL, connection pool, and migrate/migrate init container as the Message backend and Event backend. It adds two tables to that schema: sessions, one header row per session, and session_queries, one row per query, foreign-keyed to its session with ON DELETE CASCADE.

Enable it in the Helm chart:

backends:
  message: postgres
  event: postgres
  sessions: postgres
 
database:
  url: "postgres://user:password@host:5432/ark_broker"

Key environment variables:

The other DATABASE_* variables are shared with the message and event backends — see the Message backend table. Securing the connection (TLS, migrateUrl, hiding the URL in a Secret) works identically; see Securing the Postgres connection.

Session expiry is a sliding window rather than the write-time snapshot used for messages and events: every event or message on a session pushes expires_at out by SESSIONS_VISIBILITY_TTL_SECONDS, so a session stays visible for as long as it is in use, and the value does not track the originating query’s spec.ttl. The minimum enforced at startup exists because a session is the index into its own messages and events — hiding it earlier than they expire would strand data that is still retained. Expiry hides a session from reads; it does not delete the row, and a later event or message on an expired session makes it visible again.

Deleting a Query does not remove it from the sessions read model. Messages and events have per-query cleanup paths (see Query → Deletion and cleanup); sessions do not, so the query’s row stays in session_queries and its session keeps listing it. DELETE /sessions purges every session and its queries.

Cross-replica session updates

GET /sessions?watch=true opens an SSE stream of session updates. With the in-memory store that stream only carries updates produced by the replica holding the connection, so a dashboard connected to one replica never sees a session updated through another.

On Postgres the writing replica issues a NOTIFY once its transaction commits, and every replica holds a LISTEN on the same channel, so each one pushes the update to its own SSE clients regardless of where the write landed. This is for sessions what the Chunk backend (Redis) is for streaming chunks.

Chunk backend (Redis)

By default the broker holds completion chunks in process memory, served from a single replica. The chunks store can opt in to Redis Streams so that chunks produced by one broker replica can be streamed live to a client whose SSE connection landed on another replica. Only the chunks store is affected — messages, operation events, and sessions stay wherever their own backends put them, and traces remain in memory (or file-based, as above).

Each query gets its own Redis Stream key ({prefix}:chunks:{queryId}), plus a capped global key ({prefix}:chunks:all) backing the paginated and watch endpoints. Live tail uses XREAD BLOCK so chunks arrive cross-replica.

Enable it in the Helm chart:

backends:
  chunk: redis
 
redis:
  url: "redis://:password@redis-host:6379"

Key environment variables:

See the service README  for local development with devspace and integration tests.

Securing the Redis connection

In-transit encryption (rediss://)

Use the rediss:// scheme in the connection URL to connect over TLS. For a Redis endpoint with a CA trusted by the system store, no certificates need to be mounted.

CA verification (self-signed certs) — recommended for production

Create a Kubernetes Secret containing the Redis CA certificate:

kubectl create secret generic redis-tls \
  --from-file=ca.crt=/path/to/redis-ca.crt

Then enable the TLS mount:

redis:
  url: "rediss://:password@redis-host:6380"
  tls:
    enabled: true
    secretName: redis-tls

When tls.enabled=true, the chart mounts the secret and sets REDIS_TLS_CA_CERT_PATH in the broker container automatically. The secret must contain a ca.crt key.

Hiding the connection URL from Helm values

redis.url embeds the password in plain text, which is visible in kubectl get deployment -o yaml. For production, store the full URL in a Secret and reference it instead:

kubectl create secret generic broker-redis-url \
  --from-literal=redis-url="rediss://:password@redis-host:6380"

redis:
  urlSecretRef:
    name: broker-redis-url
    key: redis-url   # default key; can be omitted

The password can also be supplied separately from the URL via redis.passwordSecretRef, sourced into REDIS_PASSWORD.

Sessions Endpoint

The /sessions endpoint provides a live, event-sourced index of active sessions and their queries. It materializes state from the existing event and message streams — no additional data ingestion is needed.

Set SESSIONS_FILE_PATH to enable persistence to disk, or enable the Sessions backend (Postgres) to persist sessions in Postgres and share them across broker replicas.