Support QA Review Automation: Sample, Score, Calibrate, Improve

A QA program produces evidence for improvement, not a leaderboard

The useful output of quality review is a set of observable gaps linked to specific conversations, criteria and causes. It should tell a team which procedure, knowledge source, workflow or coaching intervention needs attention. A single percentage hides that causal path.

Current Zendesk QA documentation reflects the basic operating shape: reviewers filter conversations, choose a scorecard, score the whole conversation or a specific message, add a root cause and comments, and can handle disputes. The workflow below is product-neutral and adds the controls needed for reproducible sampling and model assistance.

Use two samples because coverage and risk answer different questions

A random sample estimates ordinary quality if every eligible case has a known chance of selection. A targeted sample finds cases the team cannot afford to miss: escalations, repeat contacts, policy exceptions, security flags, low-confidence classifications or negative feedback. Mixing them into one score biases the trend toward whichever rule selected more cases that week.

Generate a sample manifest before reviews begin: cycle ID, population query, count, random seed, selected conversation IDs, selection class and reason. That small artifact lets an auditor reproduce which cases were eligible and prevents a manager from quietly choosing only easy or difficult work.

Run the review as an evidence-backed sequence

1. 01Freeze the cohort and choose casesCode
   Apply eligibility rules, exclude active or legally restricted cases, draw the random sample, add risk-triggered cases and store the manifest.
2. 02Assemble permitted evidenceCode
   Retrieve the thread, state transitions, policy version and relevant source fields. Mask data the reviewer does not need and record access.
3. 03Suggest criterion-level findingsAI judgment
   For each rubric item, return pass, fail, not applicable or uncertain, with exact message or event references and a short rationale. No overall people ranking.
4. 04Review and correctHuman approval
   The reviewer checks evidence, resolves uncertainty, adds the root cause and submits the record. Subjective criteria and severe findings require human judgment.
5. 05Calibrate and resolve disputesHuman approval
   Double-score the overlap set. Discuss disagreements by criterion, update guidance prospectively and preserve the original rubric version on completed reviews.
6. 06Route the improvementCode
   Aggregate confirmed findings to a procedure, knowledge, tooling or coaching queue. Keep the linked cases and denominator; do not translate a score directly into a consequence.

Model suggestions should expose confidence or uncertainty but must also show the evidence. Confidence without a cited conversation span only makes an unsupported judgment look measurable. Sample model-assisted reviews again by criterion, channel and language; an average agreement figure can conceal a failing slice.

Worked example: one case, two reviewers, one ambiguous criterion

The example separates a program defect from an agent defect. Rewriting the rubric after seeing a difficult case is reasonable for future cycles; silently rescoring the past destroys the measurement history.

Handle disputes and drift as first-class workflow states

• Incomplete or corrupt conversation: mark unscorable with a reason and replace it using the original selection rule.
• Reviewer conflict: keep both scores, record the adjudicator and decision, and measure disagreement by criterion.
• Rubric changed mid-cycle: finish under the frozen version or restart; never mix versions in one trend line.
• Model drift: disable suggestions for the affected slice while human review continues, then re-evaluate on held-out cases.
• Disputed review: preserve comments and revisions as an append-only history rather than overwriting the original.

Track random-sample quality by criterion, risk-triggered defect discovery, reviewer agreement, model-to-final agreement, not-applicable use, dispute overturn rate, time to review and finding-to-fix closure. Do not optimize review count if it shortens evidence reading.

The next useful step is to turn confirmed reviewed cases into a held-out AI agent evaluation set. Keep that set versioned and separate from cases used to revise prompts or procedures.

# Support QA review SOP

Version: 1.0

Owner: Support quality lead

Purpose: Produce a reproducible service-quality signal and route confirmed findings to improvement without automatically deciding consequences for people.

## Scope and prerequisites

Applies to completed support conversations covered by an approved QA rubric. Required: eligible-population query, privacy/access rule, versioned scorecard, trained reviewers, dispute path and defined random, risk and calibration sample sizes.

## Procedure

1. Freeze the cycle dates, eligible channels, queues, languages and exclusion rules.
2. Record the population query and count.
3. Draw the random sample with a stored seed and selection probability.
4. Add risk-triggered cases with a separate selection class and reason.
5. Select a calibration overlap set for independent double scoring.
6. Retrieve permitted conversation events, source records and the effective policy version.
7. Mask or restrict sensitive fields the reviewer does not need.
8. If model assistance is used, request criterion-level pass/fail/not-applicable/uncertain suggestions with exact evidence spans.
9. The human reviewer verifies evidence, records final criterion results, root causes and comments.
10. A second reviewer independently scores overlap cases.
11. Calibrate disagreements by criterion; preserve original scores and apply guidance changes prospectively.
12. Route confirmed findings to procedure, knowledge, tooling or coaching owners.
13. Resolve disputes through an append-only adjudication record.
14. Publish random and risk-sample results separately with denominators and rubric version.

## Exception handling

- Unscorable record: record reason and replace using the original selection rule.
- Rubric change: finish the cycle under its frozen version or restart it.
- Model drift: disable suggestions for the affected slice while human review continues.
- Reviewer conflict: preserve both scores, adjudicator and resolution.
- Sensitive case: restrict to a qualified reviewer or exclude under documented rules.

## Measures

Quality by criterion and slice, risk-defect discovery, reviewer agreement, model-to-final agreement, not-applicable rate, dispute and overturn rate, review time and finding-to-fix closure.

## Prohibited uses

Do not automatically rank workers, allocate compensation, issue discipline, infer protected traits or turn one QA score into an employment decision.

## Revision log

- 2026-08-03: Initial public template.

# Support QA review workflow blueprint

Version: 1.0
Canonical job: `/solutions/customer-support/support-qa-review`

## States

`cycle_draft -> population_frozen -> sampled -> evidence_ready -> assigned -> reviewed -> calibration_pending -> confirmed|disputed -> routed -> closed`

## Core records

```yaml
cycle: {id: string, population_query: string, population_count: integer, rubric_version: string}
sample_item: {conversation_id: string, class: random|risk|overlap, reason: string, probability: number|null}
evidence_manifest: [{event_id: string, source: string, timestamp: datetime, access_class: string}]
criterion_result: {criterion_id: string, result: pass|fail|na|uncertain, evidence_ids: [string], rationale: string}
review: {reviewer_id: string, submitted_at: datetime, results: [criterion_result], root_causes: [string]}
adjudication: {status: string, adjudicator_id: string, reason: string, at: datetime} | null
```

## Actor boundaries

- **Code:** freezes cohorts, samples, retrieves evidence, validates schema, assigns, aggregates and routes.
- **AI:** may suggest one criterion result with cited evidence; cannot produce people rankings or consequences.
- **Human:** owns final subjective/severe findings, calibration, disputes, coaching and any individual consequence.

## Determinism and idempotency

- Sample manifest is immutable after `sampled`.
- Key each review by `cycle_id:conversation_id:rubric_version:reviewer_id`.
- Resubmission creates a revision; it does not overwrite the first review.
- Aggregates partition random and targeted samples and expose denominators.

## Acceptance tests

1. The random sample reproduces from the stored seed and population.
2. Every risk case retains its trigger reason.
3. A result without evidence cannot submit unless explicitly `uncertain` with reason.
4. Two reviewers can score overlap cases without seeing each other’s result.
5. Rubric versions cannot mix silently in one cycle.
6. Disputes preserve original, response and adjudication.
7. Disabling model assistance does not stop human review.
8. No workflow output directly changes employment or compensation state.

Limitations and when not to use this

• This workflow is for service-quality improvement. It must not automatically rank workers, allocate compensation, issue discipline, determine employment status or infer protected characteristics.
• Sampling design determines what conclusions are valid. A targeted risk sample can find defects but cannot estimate the overall defect rate without appropriate weighting and expertise.
• Model-assisted scoring needs representative local validation and ongoing human calibration. A vendor feature description is not evidence that a score is accurate or fair for your conversations.

Sources

1. Using Zendesk QA as a reviewer — Zendesk Documentation Accessed 3 August 2026
2. Setting up assignments in Zendesk QA — Zendesk Documentation Accessed 3 August 2026
3. Artificial Intelligence Risk Management Framework 1.0 — NIST Accessed 3 August 2026

Related reading

• Support Ticket TriageUse corrected QA findings to improve the classification taxonomy and uncertainty queues.
• Policy Exception ApprovalReview whether completed exceptions used the right authority and retained the required evidence.
• AI Agent EvaluationTurn reviewed cases into a held-out evaluation set for model-assisted support steps.
• HITL MetricsMeasure reviewer load, correction quality and control effectiveness together.