Compatible Models

Models are the foundation of the Agents at Scale platform, providing the AI capabilities that power agents. They define the connection to various AI model providers and handle authentication and configuration.

Models wrap various AI providers (OpenAI, Azure OpenAI, AWS Bedrock) and provide a consistent interface for agents to interact with them. The model resource handles API keys, base URLs, and provider-specific configurations.

Specification


apiVersion: ark.mckinsey.com/v1alpha1
kind: Model
metadata:
  name: gpt-4-model
spec:
  type: azure  # Required: "openai", "azure", or "bedrock"
  model:
    value: gpt-4.1-mini
  config:
    azure:
      baseUrl:
        value: "https://lxo.openai.azure.com"
      apiKey:
        valueFrom:
          secretKeyRef:
            name: azure-openai-secret
            key: token
      apiVersion:
        value: "2024-12-01-preview"

Model Properties

All model providers support a flexible properties system that allows you to customize model behavior by setting parameters like temperature, max tokens, and other OpenAI ChatCompletion parameters.

Basic Properties Example


apiVersion: ark.mckinsey.com/v1alpha1
kind: Model
metadata:
  name: gpt-4-custom
spec:
  type: azure
  model:
    value: gpt-4.1-mini
  config:
    azure:
      properties:
        temperature:
          value: "0.1"
        max_tokens:
          value: "1000"
        top_p:
          value: "0.9"
        stop:
          value: "END"
      baseUrl:
        value: "https://lxo.openai.azure.com"
      apiKey:
        valueFrom:
          secretKeyRef:
            name: azure-openai-secret
            key: token

Supported Properties

All providers support these common properties:

temperature: Controls randomness (0.0-2.0, default: 1.0)
max_tokens: Maximum tokens to generate
max_completion_tokens: Maximum completion tokens (newer parameter)
top_p: Nucleus sampling parameter (0.0-1.0)
frequency_penalty: Penalize frequent tokens (-2.0 to 2.0)
presence_penalty: Penalize repeated topics (-2.0 to 2.0)
stop: Stop sequence for generation
seed: Deterministic sampling seed
user: End-user identifier
logprobs: Return log probabilities (true/false)
top_logprobs: Number of most likely tokens (0-20)

Properties with Secrets

Properties can reference Kubernetes secrets for sensitive values:


properties:
  temperature:
    valueFrom:
      secretKeyRef:
        name: model-config
        key: temperature
  user:
    valueFrom:
      secretKeyRef:
        name: user-context
        key: user-id

Provider-Specific Usage

OpenAI Provider


config:
  openai:
    properties:
      temperature:
        value: "0.7"
      max_tokens:
        value: "2000"
    baseUrl:
      value: "https://api.openai.com/v1"
    apiKey:
      valueFrom:
        secretKeyRef:
          name: openai-secret
          key: token

Bedrock Provider


config:
  bedrock:
    properties:
      temperature:
        value: "0.5"
      max_tokens:
        value: "4096"
    region:
      value: "us-west-2"

Key Features

Support for multiple AI providers (OpenAI, Azure OpenAI, AWS Bedrock)
Required type field to specify provider type (“openai”, “azure”, or “bedrock”)
Provider-specific configuration under config field (openai, azure, bedrock)
Secure API key management through Kubernetes secrets
Default model configuration for agents without explicit model assignment
Flexible properties system for customizing model behavior
Support for all OpenAI ChatCompletion parameters

AWS Bedrock Configuration

AWS Bedrock provides access to foundation models from various providers like Anthropic Claude, Amazon Titan, and others. Bedrock models require AWS authentication and region configuration.

Basic Bedrock Example


apiVersion: ark.mckinsey.com/v1alpha1
kind: Model
metadata:
  name: bedrock-claude-sonnet
spec:
  type: bedrock
  model:
    value: "us.anthropic.claude-sonnet-4-20250514-v1:0"
  config:
    bedrock:
      region:
        value: "us-west-2"
      # Uses AWS credential chain (IAM roles, environment variables)

Authentication Options

Option 1: AWS Credential Chain (Recommended)

Uses IAM roles, environment variables, or instance profiles. No explicit credentials needed.


config:
  bedrock:
    region:
      value: "us-west-2"
    # No additional credentials - uses default AWS credential chain

Option 2: Static AWS Credentials

Uses AWS access keys stored in Kubernetes secrets.


config:
  bedrock:
    region:
      value: "us-east-1"
    accessKeyId:
      valueFrom:
        secretKeyRef:
          name: aws-credentials
          key: access-key-id
    secretAccessKey:
      valueFrom:
        secretKeyRef:
          name: aws-credentials
          key: secret-access-key
    # Optional: for temporary credentials
    sessionToken:
      valueFrom:
        secretKeyRef:
          name: aws-credentials
          key: session-token

Option 3: Cross-Account Access

Uses IAM role ARN for accessing Bedrock in different AWS accounts.


config:
  bedrock:
    region:
      value: "us-west-2"
    roleArn:
      valueFrom:
        secretKeyRef:
          name: aws-cross-account
          key: role-arn
    # Optional: specific model ARN for cross-region access
    modelArn:
      value: "arn:aws:bedrock:us-west-2:123456789012:foundation-model/us.anthropic.claude-sonnet-4-20250514-v1:0"

Model Configuration Options

Bedrock models support configuration through the properties system:


config:
  bedrock:
    region:
      value: "us-west-2"
    properties:
      # Maximum tokens to generate (1-100000)
      max_tokens:
        value: "4096"
      # Temperature for randomness (0.0-1.0)
      temperature:
        value: "0.7"

Available Models

For a complete list of available Bedrock models, see the AWS Bedrock Model IDs documentation .

Setup Script

Use the provided setup script to create AWS credentials:


# Set environment variables
export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
export AWS_REGION="us-west-2"
 
# Create secrets
./scripts/configure-bedrock-model.sh --namespace default

Troubleshooting

Common issues and solutions:

ValidationException: Ensure model name is correct and region supports the model
AccessDenied: Check IAM permissions for bedrock:InvokeModel
Region not supported: Verify model availability in your AWS region
Invalid model ARN: Check model ARN format and account permissions

Next: Understand Resource Relationships and how ARK components work together.