Skip to Content
ReferenceSecure Software Architecture

Secure Software Architecture

This page describes the runtime security architecture of Ark. It complements the Secure Software Development Lifecycle, which covers the engineering process, and the Data Flow and Encryption guide, which details per-surface encryption status and log sanitization.

Ark runs inside customer-controlled Kubernetes clusters. Many of the controls described here depend on correct cluster and cloud configuration — see the Disclaimer.

Access control

Ark supports OIDC-based SSO, API key authentication, and a hybrid mode combining both. When user impersonation is enabled, Kubernetes RBAC is enforced against the authenticated user’s identity rather than the Ark service account. Ark ships viewer / editor / admin ClusterRoles for each resource type that can be combined or extended.

See the Authentication guide for full details on authentication modes, impersonation, RBAC roles, and API key management.

Segregation of duties

Each Ark service runs under its own Kubernetes service account with permissions scoped to what that service needs. All service accounts set automountServiceAccountToken: false — tokens are mounted explicitly only where required.

Controller

The controller manages the full lifecycle of Ark custom resources across all namespaces. Its ClusterRole grants create, read, update, and delete access to all ark.mckinsey.com resources, plus read access to Secrets and ConfigMaps it needs to resolve references. When RBAC impersonation is enabled, the controller can impersonate service accounts to execute queries under tenant-specific identities.

A separate leader-election Role, scoped to the controller’s own namespace, manages ConfigMap and Lease resources for HA leader election.

API

The Ark API service has a namespace-scoped Role granting access to Ark resources, Secrets, and supporting Kubernetes objects within its deployment namespace. A separate ClusterRole covers cluster-scoped resources (ArkConfig) and, when impersonation is enabled, the ability to impersonate users and groups.

Dashboard and broker

The dashboard and broker are stateless services with no Kubernetes API permissions beyond their own service account identity. They do not read or write Ark custom resources directly. When the broker’s Postgres message, event, or sessions backend is enabled, the broker additionally consumes a database credential (DATABASE_URL, typically supplied from a Secret; the three backends share it); it still requires no extra Kubernetes API permissions, and it never runs schema migrations itself — those run in a separate init container as the nobody user. Likewise, when the Redis chunks backend is enabled it consumes a Redis credential (REDIS_URL or a separate REDIS_PASSWORD, typically from a Secret) with no extra Kubernetes API permissions. None of the messages, events, sessions, or session_queries tables, nor the chunk streams, have a tenant_id column or row-level security, so tenant isolation for broker storage relies on each per-tenant broker using a dedicated database and Redis instance — see Broker storage isolation.

Input validation and error handling

Admission webhooks

Ark registers validating and mutating webhooks for its core resource types. All webhooks are configured with failurePolicy: Fail — invalid resources are rejected, never silently accepted.

Validating webhooks check referential integrity, structural correctness, and security constraints (e.g., HTTPS enforcement and private IP blocking for model URLs). Mutating webhooks apply defaults and migrate deprecated configurations, annotating resources with warnings so users can update their manifests.

This describes the default (etcd/CRD) deployment. In aggregated apiserver mode (the opt-in PostgreSQL backend), Ark resources are served by Ark’s own aggregated apiserver rather than as CRDs on the main kube-apiserver, which changes admission behaviour:

  • The controller-side validating/mutating webhooks do not run (ENABLE_WEBHOOKS=false). Ark’s defaulting and validation are instead applied in-process in the apiserver’s storage path (same internal/validation package), so referential integrity, HTTPS enforcement, and structural checks still apply.

  • Third-party admission webhooks do not fire by default, and there are two enforcement points available. The main kube-apiserver does not run its mutating/validating webhook chain on aggregated (proxied) resources, so ValidatingWebhookConfiguration/MutatingWebhookConfiguration registered by Kyverno, OPA/Gatekeeper, etc. never see ark.mckinsey.com resources via the host. Ark’s aggregated apiserver can run that chain itself:

    • CEL ValidatingAdmissionPolicy (default on) — evaluated in-process, no network hop, no dependency on another pod. Requires the host cluster at k8s ≥1.30, where ValidatingAdmissionPolicy is GA.
    • Third-party webhooks (policy.thirdPartyWebhooks.enabled, default off) — runs the webhook admission plugins, so Kyverno/Gatekeeper policies apply to Ark resources on both the proxied and the direct service path. Off by default because it changes the write path: every matching webhook becomes a synchronous call, and one with failurePolicy: Fail couples Ark writes to that engine’s availability. It also requires the controller chart at storage.backend=postgresql, which stops rendering Ark’s own webhook configurations — otherwise Ark’s validation would run twice, once in-process and once over the webhook.

    The two are independent; either can run without the other.

  • CEL enforcement is best-effort by default, and that is a deliberate posture worth understanding. If the host cannot serve ValidatingAdmissionPolicy, the startup discovery probe never succeeds, or the ServiceAccount lacks permission to watch the policy objects (checked with a SelfSubjectAccessReview before the plugin is wired), the apiserver logs the reason and continues to serve with CEL enforcement off (Ark’s in-process validation and audit still apply). Enforcement can also be switched off outright with policy.cel.enabled=false, which additionally removes the cluster-wide policy and namespace watches and their RBAC. The same posture holds at runtime, not just at startup: if the policy informers stall after the process is up (revoked RBAC, egress blocked to the kube-apiserver, a swapped ServiceAccount), a required=false apiserver admits without CEL evaluation rather than failing writes, while a required=true apiserver keeps failing closed. Because a pod in that state still looks healthy to the kubelet — the readiness probe watches controller-runtime’s health server, not the aggregated apiserver’s /readyz — enforcement is exported as ark_apiserver_admission_enforcement_active, one series per mechanism, which is the signal to alert on. A deployment that treats policy as a control should set that mechanism’s required=true — policy.cel.required for CEL, policy.thirdPartyWebhooks.required for an external engine such as Kyverno — which turns each of those cases into a startup failure rather than a silently unenforced apiserver. The two are separate flags because the mechanisms are wired and fail independently, so a cluster that mandates an external engine on Ark resources but uses no CEL policy can require the one it depends on without turning the other into a startup precondition. Policy enforcement covers create, update and delete of Ark resources; writes to the /status subresource bypass both CEL policy and Ark’s own validation, as elsewhere in Kubernetes they are treated as controller-owned.

  • The aggregated apiserver emits its own audit log (JSON to stdout by default), covering resource operations including the direct-to-service path that never transits the main kube-apiserver. Audit is independent of the host’s Kubernetes version. Because an audit backend that records nothing is worse than none at all, the apiserver requires a policy file whenever audit is enabled and refuses to start otherwise. The default policy records one Metadata-level record per Ark API request and excludes health, metrics and discovery traffic.

API-level validation

The Ark API (FastAPI) validates inbound requests using Pydantic models with field constraints. Path segments are validated against traversal attacks (.., /, \), and URLs are constructed with proper encoding.

Error handling

Webhook validation errors are returned to the user with context: field paths, array indices, and specific violation descriptions. Errors are never swallowed — the failurePolicy: Fail configuration ensures the API server rejects requests when webhooks are unreachable. Mutation webhooks return non-blocking warnings for deprecated configurations, allowing the request to proceed while surfacing the migration path.

Cryptographic practices

TLS

Webhook and metrics endpoints are served over TLS. The default Helm chart uses cert-manager  with a self-signed Issuer, but production deployments can substitute any certificate provisioning mechanism. For inter-service communication beyond these endpoints, Ark relies on cluster-level controls — see Data Flow and Encryption for the TLS status of each hop.

Secret storage

Model credentials and other sensitive values are stored as Kubernetes Secrets. Secrets are base64-encoded but not encrypted by default in etcd — encryption at rest requires configuring the Kubernetes API server’s EncryptionConfiguration. See the Kubernetes encryption-at-rest documentation  for setup instructions.

API key hashing

API key secret components are hashed using bcrypt with an automatically generated salt before storage. The plaintext secret is shown only once at creation time and is never stored or returned again. Verification uses bcrypt’s constant-time comparison.

OIDC token validation

JWT tokens are validated by fetching the issuer’s JWKS endpoint via OIDC discovery (RFC 8414). The Ark API verifies the token signature, expiration, audience, and issuer. RSA and EC key types are supported. No signing keys are stored locally — they are fetched from the identity provider.

OAuth for MCP authorization

When MCP servers require OAuth, Ark implements the authorization code flow with PKCE (RFC 7636, S256 method). The code verifier is generated with 64 cryptographically random characters, and the challenge is derived via SHA-256. Callback URLs enforce HTTPS for non-loopback hosts per RFC 8252.

Last updated on
Secure Software Development LifecycleScalability