The New Skill Creator: Claude Code Finally Lets You A/B Test Your Skills

Building Claude Code Skills used to be a black box. You'd write one, hope Claude triggered it at the right moment, hope it actually made outputs better, and tune by vibes. The new Skill Creator changes that: it ships with built-in A/B testing so you can measure whether a skill is actually helping — and whether Claude is even triggering it.

The Two Problems This Fixes

Anyone who has shipped a Claude Code skill knows the two pain points:

Problem	What It Looks Like	Old Answer
Am I improving the skill?	Change prompt, hope output is better	Eyeball a few runs
Is the skill even triggering?	Expect Claude to use the skill, it doesn't	Shrug, rewrite description

Both are measurement problems, and both now have an actual tool.

A/B Testing: The Real Upgrade

The new Skill Creator runs proper A/B tests:

1. Executor (with skill) runs a task
2. Executor (without skill) runs the same task
3. Both outputs feed a blind comparator (it doesn't know which is which)
4. The comparator scores against a rubric and reports a winner

That means you can compare:

• Skill vs. no skill — is the skill helping at all?
• Skill v1 vs. skill v2 — did my optimization actually help?

The comparator being blind is the key bit. It's judging outputs on merit, not on which version it thinks is fancier. You stop guessing and start getting real data back.

Trigger Optimization

The second problem — Claude not triggering your skill when it should — also now has instrumentation. The new Skill Creator measures trigger rates and helps optimize them.

The example the video shows: a skill that was triggering about 50% of the time when it should, getting tuned up to 80-90%. That's a huge jump. The difference between "I have a skill" and "my skill actually fires" is most of the ROI.

How to Install It

Inside Claude Code, run:

/plugins

Search for Skill Creator and install. Takes about 30 seconds.

Once installed, you don't have to memorize its interface — you can ask Claude Code itself how to use it. That reflexive-self-awareness is one of the nicer properties of Claude Code: the tools it knows about include itself.

Why This Is a Real Shift

Skills are starting to look like little reliability-sensitive products inside Claude Code. Products need measurement. Up until now, building a skill was like shipping a feature with no analytics — you had no idea if it worked.

The Skill Creator brings two things that actual product work requires:

• An experimentation loop (A/B testing)
• Observability (trigger rates)

That's the delta between "I made a skill" and "I can tell you my skill has a 90% trigger rate and a 72% win rate vs. baseline." Only one of those statements is actionable.

Key Takeaways

• The new Skill Creator plugin adds A/B testing with a blind comparator — compare skill vs. no-skill and v1 vs. v2 with real data
• It also measures and optimizes trigger rates, turning the "why didn't Claude use my skill?" problem into something measurable
• Real-world example: skill trigger rate jumped from ~50% to 80-90% after optimization
• Install via /plugins inside Claude Code — search for Skill Creator
• You can ask Claude Code itself how to use it

Resources

• Claude Code Plugins Documentation — Official docs for the plugin system
• Claude Code — Anthropic's terminal-based agentic coding tool
• Claude Code Skills Overview — Documentation on building and using skills