AI Evaluation, Testing & Reliability
Learn to question an impressive score and build more useful tests. Practise finding leakage, checking evaluators and matching a decision threshold to the cost of being wrong.
Make decisions, then check the result
- Choose data splits and identify misleading benchmark claims or evaluation leakage.
- Check evaluator reliability and practise calibration and decision thresholds.
- Assemble and challenge evaluation, assurance and release evidence in the existing exercises.
These are opportunities to practise, not a guarantee of mastery or independent real-world performance.
The labs you can practise today
These are the current labs, in suggested order. The interactive exercises open from your learning map with active access.
Your curriculum · 15 labs
- When do benchmark results justify a product claim?
- What are training, validation, and test sets for?
- How should a dataset be split?
- How do leakage and test-set reuse corrupt evaluation?
- Can this evaluator or LLM judge be trusted?
- What does a preference rating mean?
- Are the model’s confidence estimates calibrated?
- How should calibrated probabilities drive actions?
- What does the balance laboratory test?
- Why can evaluation evidence stop being useful?
- Is the evidence strong enough for a product decision?
- How do you build a safety assurance case?
- What defeats or invalidates a safety assurance case?
- Is the evidence strong enough to release?
- Model Evidence Pack
A starting point, not an entrance exam
No coding or AI experience is required to begin. You can buy this product on its own; there is no Core purchase requirement.
Choose this path when its current practice matches your goal. New to the whole subject? Core offers a broader starting point, but it is not a mandatory purchase.
Compare all eight products · Explore CoreChoose a lab. Try a decision. Revisit what you learn.
After payment, finish your learner setup and open your learning map. Choose an available lab, try its activity and use its feedback to decide what to change. The suggested order is guidance; unavailable future exercises do not block your current labs.
What is here — and what is not yet available
- The proposed new measurement labs and upgraded evidence-pack assessments are not yet available. Existing exercises retain their current scope.
The lab list uses the current exercise titles and scope. A planned upgrade is not a completed assessment, and existing practice does not promise a full end-to-end project. No future launch date is promised.
Practice and credentials are different
Opening or completing a lab does not automatically award a pathway credential. A laboratory pass demonstrates only the tested performance. Any credential requires its separate evidence, assessment and verification checks; this page does not claim all those requirements are delivered.
How evidence and credentials workOne purchase. A clear access term.
US$497 · 12 calendar months · no auto-renewal. An initial or post-expiry term starts at successful payment. An eligible early renewal extends from your existing expiry at the then-current published price.
Active-term updates are included. Each specialization is a separate purchase. Optional operation of your own systems outside the included lab environment is not part of this price.
Support covers your account, billing, access and the official learning environment. Our first human response target is one business day, not a guaranteed resolution time.
14-day satisfaction guarantee and refund policy · Purchase terms · Contact support
Questions before you choose
Do I need to buy Core first?
No. This path is available independently. Choose the current practice that is useful to you.
Are all the proposed labs finished?
No. The list above is the available inventory. The proposed new measurement labs and upgraded evidence-pack assessments are not yet available. Existing exercises retain their current scope. No future release date is promised; choose based on what is available now.
How is my progress saved?
Native lab attempts and bounded progress backups are saved on your account. Imported practice is account-scoped in this browser; it is not a server-verified completion or a cross-device backup. Your learning map identifies which kind each lab uses.
What if the same lab appears in another path?
It is the same lab and saved work, not a second assignment or another completion. Optional deeper practice is labelled separately from the required curriculum.
How long will it take?
Completion effort has not yet been reliably measured. Revisit available labs during your access term without a fabricated study-hour promise.
What if it is not right for me?
The purchase policy includes a 14-day satisfaction guarantee in addition to mandatory statutory remedies. Read the refund policy for the process, or contact support for help.