Ontos Skill Evaluator

What it does

The ontos-skill-evaluator skill provides a mechanism for evaluating the performance of other AI skills. It allows users to define evaluation criteria and run tests against installed skills, providing quantifiable results about their effectiveness. This facilitates comparison between different skills and helps identify areas for improvement in existing ones.

When to use it

• Comparing Skills: You want to objectively compare two or more skills performing similar tasks (e.g., summarization, translation) to determine which is best suited for your needs.
• Performance Monitoring: Regularly assess the performance of a critical skill to ensure its continued accuracy and reliability over time.
• Skill Development: Evaluate changes made to a skill during development or maintenance to verify that improvements have been achieved.
• Benchmarking: Establish baseline performance metrics for skills before deploying them in production environments.

Key capabilities

• Defines evaluation criteria (e.g., accuracy, speed, cost).
• Runs automated tests against target skills.
• Provides quantifiable results and reports on skill performance.
• Supports comparison of multiple skills based on defined metrics.

Example prompts

• "Evaluate the 'text-summarizer' skill using the provided test dataset."
• "Compare the 'translation-en-es' and 'translate-en-es-v2' skills for accuracy and speed."
• "Run a performance benchmark on the 'code-generator' skill against the latest version."

Tips & gotchas

The evaluator requires access to both the target skill and a suitable test dataset. Ensure that the evaluation criteria are clearly defined and aligned with your specific use case for meaningful results.