Langsmith Dataset

What it does

This skill enables AI agents to interact with LangSmith datasets, allowing them to retrieve, analyze, and manage evaluation data directly. It facilitates the creation of custom test cases and the inspection of model performance metrics stored within the LangSmith platform.

When to use it

• You need to programmatically access historical evaluation results for specific LLM runs.
• Your agent requires the ability to generate new test cases based on existing dataset patterns.
• You are building a feedback loop where the agent analyzes its own performance data stored in LangSmith.
• You want to automate the retrieval of ground truth labels for validation tasks.

Key capabilities

• Retrieval and management of datasets hosted on LangSmith.
• Creation and modification of custom test cases within the dataset structure.
• Analysis of model evaluation metrics associated with specific data points.
• Integration of dataset operations directly into agent workflows.

Example prompts

• "Retrieve the last 10 evaluation results from my 'customer-support' LangSmith dataset."
• "Create a new test case in the dataset labeled 'math-problems' containing five addition questions."
• "Analyze the accuracy scores for all runs associated with the 'legal-document-review' dataset and summarize the findings."

Tips & gotchas

Ensure your AI agent has valid API credentials configured to access your specific LangSmith workspace before attempting to retrieve or modify datasets. This skill is designed for agents that need direct programmatic access to evaluation infrastructure rather than just reading static text.