In this lesson, you'll explain what an eval is and why it matters before you share or rely on a skill, and run a lightweight eval through skill-creator.
When you build a skill or bundle them into a plugin, you're essentially building a small product that other people will use. And like anything you'd hand to a colleague โ a template, a spreadsheet model, a checklist โ it's worth a test drive before it leaves your desk.
When you use a skill you built, you know how to work around any issues or failures. You know exactly what to ask it, what files to give it, and what the answer is supposed to look like. A teammate doesn't have any of that. They might phrase the request a little differently, hand it slightly different inputs, or hit an edge case โ an unusual-but-real situation, like a request that's just outside what the skill was designed for. That's where skills tend to stumble, and the person using it won't know why.
Testing a skill with evals โ short for evaluations โ is how you catch those stumbles before someone else does. An eval is just a try-out: a realistic request goes in, you look at what comes out, and you tell Claude what to fix. No code, no test scripts โ just your judgment about whether the result is good enough to put your name on.
When you build a skill with skill-creator โ Claude's built-in helper for creating skills โ it walks you through evals as part of the process.
Skill-creator comes up with two or more realistic prompts someone might use with your skill. For each prompt, it produces a pair of outputs:
That second one is the comparison point. It's there so you can see, side by side, what difference your skill is actually making โ not just "is this output okay," but "is this output better than what Claude would have done on its own."
Review each pair and give feedback in plain English, right on the review page. As you read each pair, you're really just answering two questions:
Once you submit your feedback, Claude revises the skill for you based on what you said.
Your feedback is the fix. Once you submit it, Claude updates the skill โ rewriting the instructions, adjusting the examples, tightening what it asks for โ and you can run the same prompts again to see if the change stuck.
Change one thing at a time. If the first round showed the skill was too wordy missing a section, pick the one that matters more, fix it, re-run, then come back for another review. You'll be able to tell what actually moved the needle. If you're still not happy with the outputs after the revision, run it again โ it's a loop, not a one-time gate. Most skills are ready after one or two rounds. The bar for shipping a skill โ to yourself, to a teammate โ isn't perfect evals. It's that the cases you care about pass meaningfully better than the baseline, and that you've named the cases you don't yet handle.
Sign in to join the discussion
And if the outputs already look great on the first pass? You're done. Evals aren't a hoop to jump through โ they're there for when you need confidence, not ceremony.
Step through a mock eval review โ three prompts, each with a with-skill and a without-skill output side by side. For each pair: pick the version you'd actually send, and write one line of feedback you'd give Claude. That's the whole loop.
In the next lesson, you'll move from "this works for me" to "this works for the team" โ the patterns and choices that turn personal workflows into shared infrastructure.