been building a side project that tests ai agent skills before you ship them, and i wanted to try it on something real instead of my own toy examples. so i grabbed anthropic’s official customer support skill (ticket-deflector), didn’t change a thing, and gave it the same short refund message 5 times. 4 out of 5 times the agent told the customer a $40 refund credit had been applied. it never called the payments connector. nothing was refunded. it just said it was done. that’s the part that got me. it wasn’t rude or off topic, the reply read totally fine. if you were skimming tickets you’d never catch it, but the customer would, right when they checked their account. i made a small fix to the skill, reran it, and the gate went green. curious if anyone else running support agents has hit this. how are you checking that the agent actually did the thing it says it did? submitted by /u/InterestingCarrot997
Originally posted by u/InterestingCarrot997 on r/ClaudeCode