Unit tests tell you your code works. They tell you nothing about whether your agent is any good — whether it still picks the right tool, fixes the right error, and doesn't regress when you tweak a prompt or bump a model. That gap is where a lot of GenAI quietly rots.
So the Avni Skills SDK ships two kinds of confidence: 463 deterministic unit tests for the machinery, and a real-LLM eval harness for the agent itself.
Scoring the agent, release over release
- ▸An opt-in benchmark (
npm run eval) runs the agent against a fixed set of authoring tasks with a real model and scores the outcome. - ▸Release-over-release comparison — so a prompt change or a model swap shows up as a measured delta, not a surprise in production.
- ▸Semantic audit — a second pass checks whether the generated bundle actually means what was asked, catching "beautifully, and quietly wrong" output.
Model quality is a CI concern, not a hope. If you can't measure whether the agent got worse, you'll find out from your users.
The cheap insurance nobody buys
Eval harnesses feel like overhead until the first time a "harmless" prompt tweak silently degrades your agent and you catch it before release instead of after. It's the same instinct as writing tests — you're buying the ability to change things without fear. For GenAI, where the non-determinism lives in the model, that insurance is even more valuable.
The harness and the audit are in the open-source repo on GitHub.