Evaluation

What it does

This Evaluation skill provides a framework for assessing AI agent systems, recognizing their unique characteristics compared to traditional software or language models. It focuses on evaluating dynamic decision-making and non-deterministic behavior, offering actionable feedback for continuous improvement. The skill emphasizes outcome-focused approaches that account for multiple valid paths to achieve goals while considering factors like factual accuracy, completeness, and tool efficiency.

When to use it

• Testing agent performance systematically.
• Validating context engineering choices.
• Measuring improvements over time and catching regressions before deployment.
• Comparing different agent configurations.
• Evaluating production systems continuously.

Key capabilities

• Outcome-focused evaluation that accounts for non-determinism and multiple valid paths.
• Multi-dimensional rubrics to capture quality aspects (factual accuracy, completeness, citation accuracy, source quality, tool efficiency).
• Incorporation of "LLM-as-judge" for scalable evaluation alongside human review for edge cases.
• Analysis of performance drivers including token usage, number of tool calls, and model choice.

Example prompts

• “Evaluate this agent’s response to the query: [query text]”
• “Compare the performance of Agent A and Agent B on task [task description].”
• "Assess the factual accuracy and completeness of this agent's findings."

Tips & gotchas

• Token budgets are a critical factor in evaluation; use realistic constraints.
• Upgrading the underlying model often yields greater improvements than increasing token limits.
• Be aware that agents may take different, valid paths to achieve goals – focus on outcomes rather than specific steps taken.