Troubleshooting

Troubleshooting guide for challenges when installing or using Ark.

Ark Installation

This section covers common issues and solutions when installing Ark on various platforms.

Prerequisites

Before installing Ark, ensure you have:

Node.js (v18 or later)
A Kubernetes cluster (e.g., Docker Desktop , Minikube , Kind , or Orbstack ).
kubectl and Helm

The easiest way to install dependencies on macOS:


brew install node kubectl helm
 
# Install a cluster such as:
# brew install minikube
# brew install kind
# brew install orbstack

Verify your cluster context:


kubectl config get-contexts

Common Installation Issues

Installing Ark on Windows

Ark requires a Kubernetes cluster to run. There are a number of potential issues that can arise from permissions, organizational policies and so on, particularly around running Hypervisors (which is needed to run services like Docker and local Kubernetes clusters).

WSL and Docker

In most cases, you should install and enable Windows Subsystem for Linux with Ubuntu or another Linux distribution.

You should then install Docker Desktop for Windows . Hyper-V must be enabled. Enable Hyper-V and restart. Verify in “Task Manager” → “Performance” → “CPU” that “Virtualization” shows as “Enabled”.

Kubernetes Cluster

Docker Desktop for Windows has Kubernetes support, go to wDocker Desktop Settings → Kubernetes → Enable Kubernetes.

Another option is Minikube for Windows. Install and start with minikube start.

Windows Installation Troubleshooting

Many Windows installation issues stem from enterprise security policies and restricted environments. Potential issues are:

Virtualization disabled - Hyper-V is often disabled by policy for security reasons
Restricted user permissions - Standard users may not be able to modify system settings
Security software - Endpoint protection may flag container operations as suspicious

Node.js Installation Hangs

Has been seen on some Windows installs. When the hang happens, terminate the installer, you can typically run npm install -g @agents-at-scale/ark and then install as normal, if not retry the Node.js installation.

Missing Dependencies

If ark status shows missing kubectl or helm then you local cluster is not setup, follow the guides above. The easiest way to install dependencies is via Chocolatey:


# Check system status
ark status


# Install Chocolatey first (requires PowerShell as Administrator)
Set-ExecutionPolicy Bypass -Scope Process -Force
[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072
iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))
 
# Install the core dependencies.
choco install docker
choco install helm
 
# Install minikube if you want to use this as your cluster - this'll also install
# the kubectl binary.
choco install minikube
 
# Other cluster options are Kind and so on.
# choco install kind

Minikube fails with “Hyper-V PowerShell Module is not available”

Enable Hyper-V (requires restart):


Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-Tools-All -All

If the error persists after restart, check if your organization has a specific tool or process for enabling Hyper-V. May require BIOS-level virtualization to be enabled.

Minikube “IP not found” Errors

Update WSL, delete and recreate the minikube cluster:


wsl --update
minikube delete
minikube start

If using Docker Desktop, ensure it’s running with proper permissions

Security Software Blocking Operations

If you are unable to install Hyper-V, WSL2, Docker or Kubernetes you may need to work with your organizations IT department to request access to services.

Download Failures

Failed downloads, network errors, disconnects and so on for packages or installations can sometimes be due to corporate VPN issues, which may be able to be temporarily disabled.

Avoiding Kubernetes on Windows

If Kubernetes or Docker cannot be run in your environment, the best solution would be to run a remote Kubernetes cluster, on AWS/GCP or similar. Check the Operations Guide on how to do this.

Docker Desktop Issues

If you are using Docker Desktop to launch your kubernetes cluster, you may encounter an issue where pods fail to start. When describing said pods, you may see an error like this.


Warning    Failed    pod/ark-controller-xxxxxxxxx-xxxx Error: container has runAsNonRoot and image will run as root

This is due to an the Docker Desktop Kubernetes cluster trying to run containers as root, when they shouldn’t be.

To fix this, within your Docker Desktop Kubernetes set up:

Stop the Kubernetes cluster
Start the Kubernetes cluster again, but change the cluster type from Kubeadm to kind

Webhook Call Fails on Kind Cluster

See issue #393 for tracking this bug.

Error Message

When running Ark locally on a Kind cluster, model creation or installation fails with a timeout error when attempting to reach the validating webhook. The same setup works without issues on Minikube.


ark-api dev:ark-api logs
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook \"vmodel-v1.kb.io\": failed to call webhook: Post \"https://ark-webhook-service.ark-system.svc:443/validate-ark-mckinsey-com-v1alpha1-model?timeout=10s\": context deadline exceeded","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook \"vmodel-v1.kb.io\": failed to call webhook: Post \"https://ark-webhook-service.ark-system.svc:443/validate-ark-mckinsey-com-v1alpha1-model?timeout=10s\": context deadline exceeded"}]},"code":500}

Cause

This failure occurs because the webhook service (ark-webhook-service) cannot be reached within the Kind cluster. While the specific root cause is still under investigation, it appears to be related to networking differences between Kind and Minikube.

Workaround

Use Minikube as a sanity check and use it for local.

Verification

After switching to Minikube, confirm that webhook validation succeeds by creating a model:


ark models create default
✓ Model 'default' created successfully

ark-api-a2a Not Installed Automatically

See issue #392 for tracking this issue.

Error or Symptom


 The installation command completes successfully and reports "✓ Installation complete", but the ark-api-a2a component is not actually deployed in the cluster.
'helm list -A' would not list ark-api-a2a

Cause

Investigation in progress

Workaround

Until the issue is fixed, run the ‘make ark-api-a2a-install’ command from agents-at-scale-ark repo root

Access Denied When Installing agents-at-scale

Error Message


Release "agents-at-scale" does not exist. Installing it now.
Error: GET "https://ghcr.io/v2/mck-private/qb-fm-labs-legacyx/charts/legacyx/tags/list": response status code 403: denied: permission_denied: read_package

Cause

The issue occurs when ark install attempts to pull the agents-at-scale Helm chart from the GitHub Container Registry (ghcr.io) under the private mck-private organization. If your GitHub token is missing the read:packages scope or is not authorized for SSO access to that organization, the registry request fails with a 403 Permission Denied error.

Workaround

Work in progress to fix the issue. Until it is fixed, you can install agents-at-scale manually by following the setup steps in its repository ‘agent-at-scale-user’ This ensures the required charts are locally available for Ark to reference.

Agents

Agents without a model show as ‘Unavailable’

Agents created prior to version v0.1.34 that do not have a model set may show as “Unavailable” or have events such as ModelNotFound.

To resolve, set the agent modelRef to default. See v0.1.34 Upgrade Guide for more details.