Performance Monitor

What it does

The Performance Monitor skill helps AI agents track and optimize their performance across several key areas. It specializes in monitoring token usage and costs, analyzing latency (response time), implementing quality evaluation metrics ("evals"), and benchmarking agent accuracy. Ultimately, this skill enables the creation of observability for AI pipelines to improve efficiency and reduce expenses.

When to use it

• Tracking token usage and associated costs for AI agents.
• Measuring and optimizing agent response latency.
• Implementing automated quality evaluation frameworks (evals).
• Benchmarking agent accuracy and overall quality.
• Optimizing the cost-efficiency of AI agents.

Key capabilities

• Token Usage Tracking: Instrumenting API calls to track input and output tokens, aggregating data by agent/task/user, calculating costs, and creating dashboards with alerts.
• Eval Framework Setup: Defining evaluation criteria, creating test datasets, implementing scoring functions, running automated pipelines, and tracking scores over time for regression testing.
• Latency Optimization: Measuring baseline latency, identifying bottlenecks (model, network, parsing), implementing streaming, optimizing prompt length, considering model size tradeoffs, and adding caching.

Example prompts

• "Can you show me the token usage breakdown for agent X this week?"
• "Implement an evaluation framework to measure the accuracy of agent Y on task Z."
• "What is the p95 latency for agent A when processing customer support requests?"

Tips & gotchas

• Track tokens separately from API call counts: This provides more granular cost analysis.
• Implement evals before optimizing: Establish a baseline quality measurement before making changes.
• Use percentiles (p50, p95, p99) for latency measurements: Averages can hide significant tail latency issues.