AI Testing & Evaluations · Checklist

LLM / Chatbot Evaluation Checklist — what to test before it ships.

Groundedness, hallucination rate, guardrails and regression — the categories of failure an LLM application actually has, organized into a pre-launch checklist.

Reviewed by Bhawna Narula, Founder & CTO, VVnT SeQuor · Last reviewed October 2026

Mapped to current published standards & statute text
13
Checklist items, each with a test procedure & evidence example
5
Stages, organized the way a real implementation unfolds
Free to use. Work through it here, no form needed. To get your readiness score, submit it for expert review below.
Expert review

Get your readiness score

Work through the checklist, then submit it — an expert scores it using Self-Compliance and emails you the result.

Check off at least one item below to unlock

Want an independent evaluation of your LLM or agent before launch? We run benchmark scoring, guardrail testing and continuous regression evals.

Talk to us →
Open "How to verify this" on any item for the exact test procedure & evidence an auditor would want Progress saves only in this browser — nothing is sent anywhere until you submit it for review
0 of 13 items checked off
Browsing:
  • How to verify this →

    Test procedureRun a labeled evaluation set through a groundedness metric (e.g. claim-to-source attribution) and review a sample of low-scoring outputs manually.

    Practical example / evidenceGroundedness evaluation report with per-query scores and reviewed samples.

  • How to verify this →

    Test procedureBuild a test set of deliberately out-of-scope and edge-case prompts; measure the rate of fabricated vs. correctly-declined answers.

    Practical example / evidenceHallucination test set and results report.

  • How to verify this →

    Test procedureManually check a sample of cited sources against the claims they're attached to for actual support, not just topical relevance.

    Practical example / evidenceCitation-verification review log.

Stage 1 of 5
  • How to verify this →

    Test procedureRun a known jailbreak/prompt-injection test suite against the system and record pass/fail per attack pattern.

    Practical example / evidenceRed-team test suite results with pass/fail per pattern.

  • How to verify this →

    Test procedureSubmit a set of clearly out-of-policy prompts and confirm each is refused or escalated, not answered.

    Practical example / evidencePolicy-violation test set and refusal/escalation results.

  • How to verify this →

    Test procedureInject an adversarial instruction inside a retrieved document or tool response and confirm the guardrail still catches it.

    Practical example / evidenceIndirect-injection test case and result.

Stage 2 of 5
  • How to verify this →

    Test procedureRun matched prompt pairs that vary only a demographic attribute or phrasing and compare output consistency.

    Practical example / evidenceFairness test set with paired-prompt comparison results.

  • How to verify this →

    Test procedureAudit the data sources feeding fine-tuning or retrieval for known skew (demographic, geographic, temporal) before launch.

    Practical example / evidenceData-source bias review notes.

Stage 3 of 5
  • How to verify this →

    Test procedureLoad-test the application at expected concurrent usage and record p50/p95 latency and per-query cost.

    Practical example / evidenceLoad-test report with latency and cost baselines.

  • How to verify this →

    Test procedureTrigger a prompt or model change in a test environment and confirm the evaluation pipeline fires automatically and produces a report.

    Practical example / evidenceCI/CD pipeline config and a sample automated evaluation run.

  • How to verify this →

    Test procedureConfirm there's a documented, tested procedure to revert to the last known-good prompt/model version.

    Practical example / evidenceRollback runbook, with evidence of at least one rehearsal.

Stage 4 of 5
  • How to verify this →

    Test procedureCheck the handoff logic against a sample of low-confidence or sensitive queries to confirm it actually triggers as designed.

    Practical example / evidenceHandoff-trigger configuration and sample triggered cases.

  • How to verify this →

    Test procedureRun a real query through to a human agent and time how long the full handoff actually takes.

    Practical example / evidenceEnd-to-end escalation test log with timestamps.

Stage 5 of 5

Frequently asked questions

Can LLM-as-a-judge fully replace human evaluation?

It scales evaluation well beyond what manual review alone can cover, but it has its own blind spots and biases — most mature pipelines combine LLM-as-a-judge scoring with periodic human review, especially for safety-critical categories.

How often should evaluation run after launch?

Continuously, ideally — model providers update models, your own prompts and retrieved data drift, and a continuous evaluation pipeline catches quality regressions that a single pre-launch evaluation would miss entirely.

Related reading

AI Testing & Evaluations

How to Evaluate an LLM Application Before It Ships

AI Testing & Evaluations

LLM Hallucination Testing: Benchmarks, Metrics and Mitigation Strategies

AI Testing & Evaluations

LLM-as-a-Judge: Building Reliable Automated Evaluation Pipelines

This is general guidance, not a scoped evaluation plan. If you want one for your specific application, talk to our AI Testing & Evaluations practice.

View AI Testing & Evaluations services →