Skip to content

GitHub Copilot Evals lets you test agent behaviour repeatably

A new public preview turns "does my Copilot customisation still work" into a repeatable test instead of a manual check, covering skills, custom agents and MCP tools.

Source: GitHub

GitHub Copilot Evals has moved out of private preview and is now open to everyone. It lets you write a repeatable test for a Copilot skill, custom agent or MCP tool: describe the task, describe what a good result looks like, then run the eval and see which checks pass or fail.

Evals run on Vally, GitHub’s own local evaluation engine, and work from VS Code, the Copilot CLI or the Copilot app, across Individual, Business and Enterprise plans. The point is comparison over time: change a prompt or a tool definition, rerun the same eval, and see whether the result got better or worse rather than eyeballing the output.

Why it matters: AI output varies between runs, so a customisation that looked right in testing yesterday isn’t guaranteed to behave the same way today. If you’re building skills, agents or MCP tools for Copilot, evals give you something closer to a test suite for that work, instead of manually re-checking behaviour every time you tweak a prompt.

The caveat: this only covers things built for Copilot’s own ecosystem. It isn’t a general-purpose LLM evaluation framework, so it won’t help if your agent work lives outside Copilot.

Share

Related reading

More from The Wire

All briefs