Skip to content
TrustedSkillsbeta
Back to all skills

Langsmith Evaluator

Official
langchain-ai·vlatest·ai-ml·MIT

Langsmith Evaluator assesses LLM outputs against predefined criteria, streamlining feedback and improving model performance through structured analysis.

Repository

Install on your platform

Showing Claude Code. One command installs this skill into any of the agents below.

  1. 1

    Run this in your project

    terminal
    $ npx skills add langchain-ai/langsmith-skills --skill langsmith-evaluator -a claude-code

  2. Uses the open-source skills CLI and needs Node.js. Add -g to install it for your user instead of this project.

What we know about this skill

Publisher

langchain-ai

Matched to the vendor's own GitHub organisation.

Install target

Live repository

Not pinned. Installing fetches whatever the repository holds at the time you run the command.

No human has reviewed this code. The checks below are a static scan of the files at one commit — nobody has run this skill or judged whether it works. A skill runs with whatever access you give your agent, so read the source before you install it.

Automated safety checks

scanned

This skill failed 2 of 6 checks. A failure is not proof of bad intent; it means something in the code needs a human to look at it before you run it.

  • No undeclared network calls: failed

    Contacts 1 host that is neither well-known infrastructure nor declared in the skill's frontmatter: cli.langsmith.com.

    Every host the skill dials — from its scripts, its config endpoints or the commands it tells the agent to run — is either well-known package and API infrastructure or declared in its own frontmatter. Links in documentation are not counted.

    Hosts contacted

    cli.langsmith.comundeclared

    What it found

    • config/skills/langsmith-evaluator/SKILL.md:49contacts cli.langsmith.com
      curl -fsSL https://cli.langsmith.com/install.sh | sh

  • No pipe-to-shell installers: failed

    Runs code fetched at install time in 1 place.

    No `curl … | sh` pattern, in the code or in the instructions, that runs code fetched at run time.

    What it found

    • config/skills/langsmith-evaluator/SKILL.md:49remote script piped into a shell
      curl -fsSL https://cli.langsmith.com/install.sh | sh

  • Declares what it does: passed

    SKILL.md parses and declares a name and a description.

    SKILL.md exists, its frontmatter parses, and it declares a name and a description.

  • No obfuscated payloads: passed

    No encoded or obfuscated payloads.

    No base64, hex or compressed blob is decoded and then executed, and no encoded shell commands.

  • No credential access: passed

    Reads no keys, tokens, keychains or credential files.

    Neither the code nor the SKILL.md instructions read SSH keys, cloud credentials, keychains, token stores, browser cookie databases or dotfiles that hold secrets.

  • Pinned to a commit: passed

    Scanned langchain-ai/langsmith-skills at bc2f989d85.

    The exact upstream commit SHA that was scanned is recorded, so the result refers to specific bytes.

Commit
bc2f989d85
Manifest
config/skills/langsmith-evaluator/SKILL.md
Scanned
1 file, 452 lines
This is a static scan, not a review. It reads the skill's files; it does not run them, and it cannot tell you whether the skill is any good or whether the code does what its description says. A skill runs with whatever access you give your agent.
What each check looks for →

About this skill

What it does

The Langsmith Evaluator skill enables AI agents to assess LLM outputs against predefined criteria, providing structured feedback and facilitating model performance improvement. It allows for both offline evaluation (comparing agent output to expected values in a dataset) and online evaluation (real-time quality checks on production runs). Crucially, evaluators must be written after inspecting actual agent outputs and understanding their structure using Langsmith traces.

When to use it

This skill is useful when:

  • You want to compare an AI agent's output against expected values in a dataset for evaluation purposes.
  • You need real-time quality checks on production runs of your AI agent, without relying on reference data.
  • You are looking for structured feedback and analysis of LLM outputs beyond simple text comparisons.
  • You want to systematically improve the performance of an AI agent by identifying areas where its output deviates from desired results.

Key capabilities

  • Offline Evaluation: Evaluates agent outputs against dataset examples.
  • Online Evaluation: Performs real-time quality checks on production runs.
  • Structured Feedback: Provides metrics based on predefined criteria, rather than freeform text.
  • Trace Integration: Leverages Langsmith traces to understand output structure and debug evaluation logic.
  • CLI Tooling: Includes a CLI tool for uploading evaluators.

Example prompts

While this skill doesn't directly respond to user prompts, here are examples of how you might use it after an agent has run:

  • "Run the Langsmith Evaluator against this agent output and compare it to the expected answer in dataset 'My Dataset'."
  • "Perform a real-time quality check on this production run using the 'Accuracy' evaluator."
  • "Show me the Langsmith traces for this run so I can understand the structure of the output before writing an evaluator."

Tips & gotchas

  • Inspect Before You Implement: The most important step is to examine agent outputs and their structure within Langsmith traces before writing any evaluation logic. Assume nothing about the format – always verify it first.
  • Single Metric per Evaluator: Each evaluator function must return only one metric. For multiple metrics, create separate evaluator functions.
  • Authentication Required: You'll need to set the LANGSMITH_API_KEY environment variable or use the --api-key flag when using the CLI tools.