Evaluation

What it does

The Evaluation skill analyzes agent systems to assess their performance and quality. It moves beyond traditional software evaluation by accounting for the non-deterministic nature of agents, which can take multiple valid paths to achieve goals. The skill helps build robust evaluation frameworks that provide actionable feedback for continuous improvement, regression detection, and validation of context engineering choices.

When to use it

• Testing agent performance systematically.
• Validating context engineering choices made when designing an agent system.
• Measuring improvements in agent performance over time.
• Catching regressions (undesirable changes) before deploying an agent.
• Building quality gates for automated agent pipelines.

Key capabilities

• Supports outcome-focused evaluation, judging agents based on results rather than specific steps taken.
• Utilizes multi-dimensional rubrics to assess aspects like factual accuracy, completeness, citation accuracy, source quality, and tool efficiency.
• Can leverage LLM-as-judge for scalable evaluation while incorporating human review for edge cases.
• Considers token usage, number of tool calls, and model choice as key performance drivers in agent evaluation.

Example prompts

• "Evaluate this agent's response to the query 'Find the capital of France' focusing on factual accuracy."
• "Assess the efficiency of this agent’s process for summarizing a news article."
• "Compare the performance of these two agent configurations when completing task X, using the provided rubric."

Tips & gotchas

• Agent systems are non-deterministic; evaluations should account for multiple valid paths to achieve goals.
• Token budgets significantly impact agent performance – evaluate agents within realistic resource constraints.
• Upgrading the underlying model often yields greater improvements than simply increasing token limits.