We publish ~25 skills in a public repo . Unlike code, the failure mode we care about is a quiet one: a skill still lints, still loads, and stops doing the thing we want it to do. So the repo carries ~1,300 eval tasks . Each one is a prompt, a setup, and criteria that check what the agent actually produced - a file exists, a command exited clean, a CLI got called with the right verb ( example task ) - mostly deterministic checks. A PR maps its changed skills to their tasks and runs them in a CI pipeline . We created a custom coding agent eval harness to run all the tasks (Apache 2.0, I work on it): https://github.com/UiPath/coder_eval We wrote our own because the same task has to run against Claude Code, Codex and a couple of others. We also do daily tests and wanted to have a full UX to debug problems. If you ship skills, prompts or agent configs, do you test them on merge? With what? submitted by /u/EvalRaccoonDev
Originally posted by u/EvalRaccoonDev on r/ClaudeCode