IntegrAuth Academy
Identity, security and practical AI — free, from zero. Byte-sized lessons, real diagrams, hands-on labs, zero fluff. Two programs, each with its own exam and certificate.
English is hard? Your browser can translate this page into Tamil, Telugu, Kannada, Malayalam or Hindi. Tech words stay in English so you learn the real names.
How?
- Chrome on phone: ⋮ menu → Translate
- Chrome on computer: right-click → Translate to …
- Edge: the translate icon in the address bar
Run AI in production
Priya's support bot has run for a month. Now it is slow, the bill has doubled, and one answer was wrong. Nothing crashed. Your goal: find out what happened, from records.
🧪 Interactive lab — enable JavaScript to play with this one.
- An AI feature can be up and fast, and still give wrong answers.
- Watch the slowest requests, cost per task, errors and one quality number.
- Most surprises come after a change. Pin the model version and roll out slowly.
Record each request
Normal monitoring only checks if a service is up. That misses wrong answers. So record what happens inside each request, like a courier tracking page that shows every scan. This record is a trace. Each step in it is a span, like a model call or a tool call.
The numbers to watch
Averages hide slow requests. So look at the time within which 95 of 100 requests finish. This is the p95 latency. The p50 is the middle request.
Count cost per task, not per call. One task can make ten model calls. Cost comes from tokens in model calls, not from slow tools.
Track quality, like thumbs-down clicks. You can also score a sample of live answers. This is an online eval.
Change is the risk
Most surprises come after a change: a new prompt, model or documents. Some providers offer a model alias that moves to their newest model. Use a pinned model version, and upgrade on purpose.
Give a new version to a few users first. Compare its speed, cost, errors and quality. This is a canary release. If it looks worse, switch back in one step.
Priya's numbers jumped on 3 days. Match each jump to its change.
🧪 Interactive lab — enable JavaScript to play with this one.
Go deeper: what a trace records
For each model call, a good trace records the model and its version. It records the tokens in and out, the time taken and the stop reason. It also records the time to first token and any error. For each tool call, it records the tool name, a safe summary of its inputs, the time and the result. Add the user as an id, the prompt version and the cost.
A trace helps engineers find problems. An agent audit trail is a separate record. It proves who did what.
Many monitoring tools use an open standard, OpenTelemetry. It gives shared names for AI trace fields, like gen_ai.usage.input_tokens. As of October 2026, these names are still marked "Development", not stable. Expect some to change.
Go deeper: tools that show traces
Many tools collect and show traces. Some also run evals and keep prompt versions. As of October 2026, examples in no order:
- Langfuse: open source, now owned by ClickHouse. You can host it yourself.
- LangSmith: closed source, from the LangChain team.
- Arize Phoenix: source-available. It runs on your own computer.
- Braintrust: a hosted platform for evals and traces, with a free plan.
- Helicone: an open-source gateway. Its company says it is in maintenance mode.
A gateway sees every model call. It does not see what your own tools did in between. Tracing inside your own code shows the whole request.
Go deeper: logs without leaks
A trace with full prompts and answers is a copy of your customers' chats. It can hold names, addresses, even card numbers. That is why OpenTelemetry leaves message text out unless you turn it on. So decide on purpose:
- Mask it first. Hide personal data before you store it. Never log secrets.
- Keep it short. Delete old traces on a schedule. A customer's delete request must reach them.
- Lock it down. Only people who need it get access.
- Check where it goes. A hosted tracing service is one more company with your users' data.
Go deeper: offline and online evals
Your eval set is an offline check. It runs fixed cases before a change ships. An online eval scores a sample of live answers later, with code checks and an AI judge. Offline evals catch what you thought to test. Online evals catch what real users do. Turn each failure you find into a new offline case.
Count errors by kind: timeouts, rate limits, failed tools, refusals and answers cut off at the token limit.
Go deeper: retirements, fallbacks and incidents
Pinned models still retire. After that, calls to them fail. So keep a list of which feature uses which model. Run your evals on the replacement early.
Plan a fallback before you need it. First, retry briefly. Then switch to a second model or provider. Last, give a saved answer, like "a person will reply soon". Gateways like the open-source LiteLLM or the hosted OpenRouter can switch for you. A fallback model is a change, too. It needs its own evals, and it must be allowed to see the same data.
When something breaks, alerts find it. A kill switch turns off the risky feature and keeps the rest (agent registry). Traces explain it. The failing chat becomes a new test case.
Run Langfuse or Arize Phoenix on your computer. Trace a few model calls. Find the tokens and the time.