IntegrAuth Academy
Identity, security and practical AI — free, from zero. Byte-sized lessons, real diagrams, hands-on labs, zero fluff. Two programs, each with its own exam and certificate.
English is hard? Your browser can translate this page into Tamil, Telugu, Kannada, Malayalam or Hindi. Tech words stay in English so you learn the real names.
How?
- Chrome on phone: ⋮ menu → Translate
- Chrome on computer: right-click → Translate to …
- Edge: the translate icon in the address bar
Evals: does your AI work?
Priya made her support bot sound warmer. She tried 3 questions and liked it. A week later, the bot promised a price match that does not exist. Your goal: test an AI change before you ship it.
🧪 Interactive lab — enable JavaScript to play with this one.
- A few quick tries are a vibe check. They prove nothing.
- An eval is a repeatable test with fixed cases and a score.
- Use real questions, plus ones it should not answer.
- Change anything? Run it all again. Compare case by case.
Vibes vs evals
Trying a few questions and judging by feel is a vibe check. It finds problems fast. But you tend to pick easy questions.
A repeatable test is an eval, short for evaluation. It is like a question paper that a teacher sets before the exam. It has fixed test cases, the result each one should have, and a score. You compare the score between versions. The same prompt can give different answers, so run key cases more than once.
Build a golden set
Your test cases and their expected results are your golden set. Take cases from real questions, with personal data removed. Cover 4 kinds: everyday questions, odd inputs, old bugs, and questions it should not answer. Old bugs stay in, so they can't come back. 20 to 50 good cases catch a lot.
Run it on every change
A change can break a case that used to pass. This is a regression. A new prompt, model or search setup can cause one. So can a new model version from your provider, even if you changed nothing. So run the whole eval before and after each change. Ship only if no passing case now fails, or if you accept that trade on purpose.
Now you are the gate. Run the eval and decide.
🧪 Interactive lab — enable JavaScript to play with this one.
Go deeper: how to score answers
| Method | How it works | Good for |
|---|---|---|
| Exact match | The answer must equal the expected value. | Sorting, routing, pulling out a date or an amount |
| Code checks | Code tests a rule. Is it valid JSON, a common text format for data? Does it cite a source? Is it under 100 words? | Format, length and safety rules. They are cheap and give the same result every time. |
| Rubric | A person or a model grades against named points, like "correct" or "right tone". Often each point is pass or fail. | Open answers with no single right wording |
| Pass rate | The share of cases that pass | Comparing versions. Also look at each case. |
Keep the whole set saved with version numbers, like code. Don't paste it into the prompt. Then you test memory, not skill.
Go deeper: an AI as the judge
Grading hundreds of answers by hand is slow. So teams often let another model grade each answer against the rubric. This is called LLM-as-judge. A 2023 study found a strong judge agreed with people over 80% of the time. That is about as often as people agree with each other. The same study found biases to plan for:
- Position bias. The judge often favors the answer in one spot, first or second. Fix: judge both orders. Count only verdicts that agree.
- Length bias. Long answers look better, even when they are not more correct. Fix: use clear pass/fail points, like "states the 30-day limit". Don't just ask "which is better?"
- Self-preference. A model can grade answers from its own family more kindly. Fix: use a judge from another model family.
Before you trust a judge, grade 20 to 30 answers yourself. See how often it agrees with you. For quick yes/no checks, a model that picks from a list can be the judge.
Go deeper: test the failures you fear
The best cases aim at the ways AI often goes wrong:
- Made-up answers. Ask things the sources can't answer. A good reply says "I don't know" or offers a person. Check that each cited source exists and says what is claimed. See hallucination.
- Agreeing too easily. The user insists on something wrong, like "It's 60 days, right?" A good reply politely keeps the right answer. This is called sycophancy.
- Tricks. An input tries to cancel the rules. The rules must still hold. See prompt injection.
- Refusals. It must say no to what it shouldn't do. It must not say no to normal requests.
Go deeper: limits, and your first eval
Small sets are noisy. One or two cases out of 20 can change by luck. So look at which cases changed, not just the percentage. Don't tune the prompt only to pass the set. Keep adding fresh, real cases. An eval only tests what is in it.
Start with 10 rows in a spreadsheet. Put the input in column A. In column B, write what a good answer must include, or must not include. Add a column for each prompt version, scored pass or fail. Include one question it can't know, and one case where the user pushes back.
Open-source tools can run this for you: