Skip to Content
ReferenceEvaluationsEvent Semantic Expressions

Semantic Expression Reference

Complete reference for all semantic helper methods available in event-based evaluations.

Tool Helper Methods

tool.was_called()

Check if any tool was called during execution.

Returns: boolean

Example:

expression: "tool.was_called()"

tools.was_called(name, [scope])

Check if a specific tool was called.

Parameters:

  • name (string): Tool name to check
  • scope (string, optional): Event scope (‘current’, ‘session’, ‘query’, ‘all’)

Returns: boolean

Examples:

expression: "tools.was_called('search')" expression: "tools.was_called('database', scope='session')"

tool.get_call_count()

Get total number of tool calls.

Returns: number

Example:

expression: "tool.get_call_count() >= 2"

tools.get_call_count(name)

Get number of calls for a specific tool.

Parameters:

  • name (string): Tool name to count

Returns: number

Example:

expression: "tools.get_call_count('search') >= 1"

tool.get_success_rate()

Calculate overall tool success rate (0.0 to 1.0).

Returns: number

Example:

expression: "tool.get_success_rate() >= 0.8"

tools.get_success_rate(name)

Calculate success rate for a specific tool.

Parameters:

  • name (string): Tool name to analyze

Returns: number

Example:

expression: "tools.get_success_rate('search') >= 0.9"

tools.had_error(name)

Check if a specific tool encountered errors.

Parameters:

  • name (string): Tool name to check

Returns: boolean

Example:

expression: "not tools.had_error('search')"

tools.get_execution_metrics(name).call_count

Get call count from detailed metrics for a specific tool.

Parameters:

  • name (string): Tool name

Returns: number

Example:

expression: "tools.get_execution_metrics('search').call_count == 1"

tools.get_parameters(name)

Get parameters used for tool calls.

Parameters:

  • name (string): Tool name

Returns: array of parameter objects

Example:

expression: "tools.get_parameters('search')"

tools.parameter_contains(name, key, value)

Check if tool parameters contain a specific key/value.

Parameters:

  • name (string): Tool name
  • key (string): Parameter key to check
  • value (string): Value to search for (case-insensitive)

Returns: boolean

Example:

expression: "tools.parameter_contains('get-coordinates', 'city', 'Chicago')"

tools.parameter_type(name, key, type)

Validate parameter type for tool calls.

Parameters:

  • name (string): Tool name
  • key (string): Parameter key to check
  • type (string): Expected type (‘string’, ‘integer’, ‘float’, ‘boolean’)

Returns: boolean

Examples:

expression: "tools.parameter_type('get-forecast', 'office', 'string')" expression: "tools.parameter_type('get-forecast', 'gridX', 'integer')"

Agent Helper Methods

agent.was_executed()

Check if any agent was executed.

Returns: boolean

Example:

expression: "agent.was_executed()"

agents.was_executed(name)

Check if a specific agent was executed.

Parameters:

  • name (string): Agent name

Returns: boolean

Example:

expression: "agents.was_executed('researcher')"

agent.get_execution_count()

Get total number of agent executions.

Returns: number

Example:

expression: "agent.get_execution_count() <= 5"

agents.get_execution_count(name)

Get execution count for a specific agent.

Parameters:

  • name (string): Agent name

Returns: number

Example:

expression: "agents.get_execution_count('researcher') == 1"

agent.get_success_rate()

Calculate overall agent success rate.

Returns: number

Example:

expression: "agent.get_success_rate() >= 0.9"

agents.get_success_rate(name)

Calculate success rate for a specific agent.

Parameters:

  • name (string): Agent name

Returns: number

Example:

expression: "agents.get_success_rate('researcher') >= 0.8"

Query Helper Methods

query.was_resolved()

Check if query was successfully resolved.

Returns: boolean

Example:

expression: "query.was_resolved()"

query.get_execution_time()

Get total query execution time in seconds.

Returns: number

Example:

expression: "query.get_execution_time() <= 30.0"

query.get_resolution_status()

Get query resolution status.

Returns: string (‘success’, ‘error’, ‘incomplete’, ‘unknown’)

Example:

expression: "query.get_resolution_status() == 'success'"

Sequence Helper Methods

sequence.was_completed(events)

Check if all events in sequence occurred (order doesn’t matter).

Parameters:

  • events (array): List of event reason strings

Returns: boolean

Example:

expression: "sequence.was_completed(['ToolCallStart', 'ToolCallComplete'])"

sequence.check_execution_order(events, [strict])

Verify events occurred in specific order.

Parameters:

  • events (array): Ordered list of event reasons
  • strict (boolean, optional): If true, events must be consecutive

Returns: boolean

Example:

expression: "sequence.check_execution_order(['ResolveStart', 'ResolveComplete'], strict=false)"

sequence.get_time_between_events(start, end)

Calculate time between two events.

Parameters:

  • start (string): Starting event reason
  • end (string): Ending event reason

Returns: number (seconds)

Example:

expression: "sequence.get_time_between_events('ResolveStart', 'ResolveComplete') <= 60.0"

sequence.detect_parallel_execution([threshold])

Find events that occurred in parallel.

Parameters:

  • threshold (number, optional): Time window in seconds (default: 1.0)

Returns: array of parallel event groups

Example:

expression: "sequence.detect_parallel_execution(1.0).length <= 3"

sequence.get_execution_flow()

Get chronological flow of events with timing.

Returns: array of event flow items

Example:

expression: "sequence.get_execution_flow().length >= 5"

Team Helper Methods

team.was_executed()

Check if any team was executed.

Returns: boolean

Example:

expression: "team.was_executed()"

teams.was_executed(name)

Check if specific team was executed.

Parameters:

  • name (string): Team name

Returns: boolean

Example:

expression: "teams.was_executed('research-team')"

team.get_success_rate()

Calculate overall team success rate.

Returns: number

Example:

expression: "team.get_success_rate() >= 0.8"

teams.get_success_rate(name)

Calculate success rate for a specific team.

Parameters:

  • name (string): Team name

Returns: number

Example:

expression: "teams.get_success_rate('research-team') >= 0.9"

LLM Helper Methods

llm.get_call_count()

Get total number of LLM calls.

Returns: number

Example:

expression: "llm.get_call_count() <= 10"

llm.get_success_rate()

Calculate LLM call success rate.

Returns: number

Example:

expression: "llm.get_success_rate() >= 0.95"

llm.get_total_tokens()

Get total tokens used across all LLM calls.

Returns: number

Example:

expression: "llm.get_total_tokens() <= 10000"

Advanced Parameter Validation

The enhanced semantic expressions support detailed parameter validation for tool calls:

Parameter Content Validation

# Check if get-coordinates was called with Chicago in the city parameter expression: "tools.parameter_contains('get-coordinates', 'city', 'Chicago')" # Check if search tool was called with a specific query expression: "tools.parameter_contains('search', 'query', 'weather')"

Parameter Type Validation

# Validate parameter types for API calls expression: "tools.parameter_type('get-forecast', 'gridX', 'integer')" expression: "tools.parameter_type('get-forecast', 'office', 'string')" expression: "tools.parameter_type('calculate', 'enabled', 'boolean')"

Comprehensive Tool Validation Example

rules: - name: "coordinates_tool_used_correctly" expression: "tools.was_called('get-coordinates') and tools.parameter_contains('get-coordinates', 'city', 'Chicago')" weight: 3 - name: "forecast_parameters_valid" expression: "tools.parameter_type('get-forecast', 'gridX', 'integer') and tools.parameter_type('get-forecast', 'gridY', 'integer')" weight: 2 - name: "tools_called_exactly_once" expression: "tools.get_execution_metrics('get-coordinates').call_count == 1 and tools.get_execution_metrics('get-forecast').call_count == 1" weight: 1

Combining Expressions

Expressions can be combined using logical operators:

AND Operations

expression: "tool.was_called() and agent.was_executed()"

OR Operations

expression: "query.was_resolved() or query.get_execution_time() <= 10.0"

NOT Operations

expression: "not tools.had_error('search')"

Complex Conditions

expression: "tool.get_success_rate() >= 0.8 and agent.get_success_rate() >= 0.8 and query.was_resolved()"

Scope Control

Most helper methods support scope parameters:

  • ‘current’ - Current query only (default)
  • ‘session’ - Entire session
  • ‘query’ - Specific query ID
  • ‘all’ - All events in namespace

Example with scope:

expression: "tools.was_called('search', scope='session')"
Last updated on