Scalability

This page describes how Ark scales, what to monitor, and how to plan capacity for production deployments. It complements the Core Architecture and Monitoring guides.

Ark’s control plane runs on Kubernetes and inherits its scaling primitives. The controller coordinates reconciliation via leader election. The API and dashboard are stateless and can be horizontally scaled. The broker holds data in memory, so it runs as a single replica by default.

Capacity management

Controller

The controller reconciles all Ark custom resources across the cluster. It runs with leader election enabled by default, so multiple replicas can be deployed for availability — only the leader actively reconciles.

The controller’s default resource allocation is intentionally conservative. Production deployments with a high volume of concurrent queries or large numbers of resources should increase CPU and memory limits based on observed usage.

Query concurrency tuning

The Query controller exposes two knobs via Helm values under controllerManager:

• maxConcurrentQueries (default 32) — caps the number of Query executions running concurrently as in-process goroutines. When the cap is reached, the reconciler requeues so the backlog stays in the workqueue instead of the controller heap. This bounds memory under burst load. Set to 0 to disable the cap (not recommended — susceptible to OOM under burst load).
• maxConcurrentReconciles (default 4) — caps the number of Query reconciles running in parallel. The workqueue dedupes per-key, so this only enables concurrency across different Query objects, not faster repeats of the same one. Set to 0 to use the controller-runtime default (1).

The two knobs are independent and both effective at the defaults. maxConcurrentReconciles controls how fast queued queries are picked up; maxConcurrentQueries bounds how many of them can be in-flight at once. Raise both together if the controller has spare CPU/memory headroom and queue depth is growing. Lower maxConcurrentQueries if the controller pod is near its memory limit during bursts.

API and dashboard

The API and dashboard are stateless and can be scaled horizontally by increasing replica counts in their Helm values.

Broker

The broker stores messages, streaming chunks, traces, events, and sessions in memory by default. It is not stateless — each replica maintains its own state, so it runs as a single replica by default. If scaling beyond a single replica is needed, consider enabling file-based persistence or a sticky session strategy.

The messages, operation events, and sessions stores can each opt in to a Postgres backend (backends.message: postgres, backends.event: postgres, and backends.sessions: postgres, sharing one database) so that data survives pod restarts, and the chunks store can opt in to Redis (backends.chunk: redis). Sessions on Postgres require the message and event backends to be on Postgres too, and they also fan ?watch=true updates out across replicas through LISTEN/NOTIFY, as chunks on Redis do for live streaming. These add durability; they do not by themselves make the broker horizontally scalable, since traces still remain in-memory per replica.

Storage backend

When using the default etcd-backed storage, scalability is bounded by the cluster’s etcd capacity. For deployments with a high volume of resources, the PostgreSQL storage backend provides more headroom. PostgreSQL connection pooling is configured with sensible defaults (max open connections, idle connection management, connection lifetime limits) and can be tuned for higher throughput.

The PostgreSQL backend’s advantage grows with resource count — it is most pronounced on LIST operations. Benchmarks at 10,000 Ark resources:

On PostgreSQL, production capacity is roughly ~230 ops/sec burst / ~180 ops/sec sustained, 200+ concurrent watchers at ~80ms P50, with optimal concurrency around 50–100 workers. See Core Architecture → Storage backends for the trade-off and PostgreSQL Storage Backend for setup.

Executors

All executors — including the default completions executor — run as separate deployments and communicate with the controller via A2A. The default completions executor uses an in-memory A2A task manager that holds conversation history, active tasks, and streaming subscribers, so it runs as a single replica by default. Scaling it to multiple replicas would require sticky sessions or a shared task store. Marketplace and custom executors can define their own HPA, resource limits, and replica strategy depending on their state management approach.

Resource monitoring

Metrics

The controller exposes Prometheus metrics over a TLS-protected endpoint. A ServiceMonitor template is included in the Helm chart (disabled by default) for integration with the Prometheus Operator. Standard controller-runtime metrics cover reconciliation rates, queue depths, and error counts.

Ark also integrates with OpenTelemetry for distributed tracing across the query lifecycle — from the controller through A2A dispatch to executors. See the Monitoring guide for setup instructions.

Health probes

All services expose liveness and readiness probes. The controller uses conservative initial delays and failure thresholds to avoid premature restarts during startup. Probe timing is configurable through Helm values.

Key indicators to watch

• Reconciler queue depth and latency — Rising queue depth or increasing reconciliation time indicates the controller is falling behind. Consider increasing resource limits or investigating slow external calls (LLM providers, executors).
• Broker memory usage — The broker holds data in memory with configurable item limits. Monitor memory consumption against the configured limits to avoid OOM conditions.
• API response times — Increased latency at the API layer may indicate Kubernetes API server pressure, especially under high query volume.
• Executor pod scaling — If executors are autoscaled, monitor HPA status and pending pod counts to ensure the cluster has sufficient capacity to schedule new replicas.

Capacity planning

Sizing considerations

Ark’s resource footprint depends primarily on the number of concurrent queries, the complexity of agent configurations (tool count, team depth), and the volume of stored resources. The default Helm values are sized for development and small-scale deployments.

For production planning, consider:

• Query throughput — Each active query involves a controller reconciliation, an A2A dispatch to an executor, and one or more LLM provider calls. The controller and executor are the primary bottlenecks.
• Resource count — A large number of Agent, Tool, or Team definitions increases the controller’s reconciliation workload and etcd/PostgreSQL storage requirements.
• Conversation history — If the broker has persistence enabled, or if you use an external memory backend, storage grows with conversation volume. Plan retention and cleanup accordingly. With the Postgres message backend, message storage is bounded by the TTL-based expiry rather than item-count limits.

Cluster-level capacity

Ark does not ship PodDisruptionBudgets by default. Production deployments should add PDBs for the controller and API to ensure availability during node maintenance. See the Kubernetes PDB documentation  for guidance.

Node capacity should account for executor workloads, which are typically the most resource-intensive components. LLM-backed executors may require significant memory for request/response buffering, especially with long-context models.

Scaling path

1. Start with defaults — Deploy with the default Helm values and a single replica per service.
2. Monitor under load — Use Prometheus metrics and pod resource usage to identify bottlenecks.
3. Scale horizontally — Increase replica counts for the API, dashboard, and executors as needed.
4. Tune the controller — Increase CPU and memory limits based on observed reconciliation load.
5. Consider PostgreSQL — Switch from etcd to PostgreSQL if resource volume or query history exceeds etcd’s practical limits.
6. Add autoscaling — Configure HPA for executors and stateless services based on CPU or custom metrics.