# AI Recruiting Pilot Design and Metrics Guide

Canonical HTML: https://openjobs.genedai.me/pilot-design
Last substantive review: 2026-08-07

> OpenJobs AI is now Metix AI. This archive applies the same evidence rules to Metix first-party material and independent sources.

A useful AI recruiting pilot tests the hardest uncertainty on a real role while limiting candidate and operational exposure. It preserves a current baseline, predefines quality, measures hidden labor, reviews misses, and makes stopping as operationally possible as expanding.

## Contents

- [Write the pilot question before choosing the success metric.](#pilot-question)
- [Build the baseline from a recent comparable role.](#baseline)
- [Choose roles and samples that can reveal the expected failure modes.](#roles-and-samples)
- [Pair outcome metrics with quality, labor, and control metrics.](#metrics)
- [Predefine review and adjudication before opening the results.](#review-protocol)
- [Write stop conditions and recovery actions before candidate exposure.](#stop-conditions)
- [Operate the pilot with checkpoints, version control, and an evidence log.](#pilot-operation)
- [Expand only the evidence-backed scope, not the entire product promise.](#scale-decision)

## 1. Write the pilot question before choosing the success metric.

A pilot is not a discounted subscription or an extended demo. It is a bounded test of a decision-relevant uncertainty. Examples include whether a sourcing system improves qualified coverage within a fixed review budget, whether a screening workflow produces consistent role-relevant evidence, or whether an engagement agent reduces coordination time without creating message or consent failures.

Write one primary question and a small number of safety or operating gates. When a pilot tries to prove quality, speed, cost, candidate experience, integration, fairness, security, and every product feature at once, teams change definitions after seeing results. Secondary observations remain useful, but they should not rescue a failed primary test.

Name the people who can interpret the result: hiring manager for role relevance, recruiting owner for workflow, technical owner for integrations, candidate-support or accessibility owner for affected people, and commercial owner for expansion. Decide who can stop external actions immediately without waiting for a steering meeting.

- **Primary hypothesis.** State the workflow change, population, comparison, metric, and decision threshold.
- **Guardrails.** Define candidate, operational, data, and external-action conditions that cannot be traded for speed.
- **Decision owner.** Identify who accepts the result and who may halt the test.
- **End date.** A pilot ends with a decision; it does not become indefinite production by inertia.

## 2. Build the baseline from a recent comparable role.

Choose a role recent enough that labor market, compensation, employer brand, team availability, and recruiting process are reasonably comparable. Record inputs and definitions rather than copying one top-line time-to-hire value. Time-to-hire includes many steps the evaluated system may not influence, while a narrow system may materially change time to qualified slate or reviewer hours.

Capture the funnel with denominators: people retrieved or applied, reviewed, advanced, contacted, delivered, replied, expressed interest, screened, scheduled, interviewed, and hired where available. Capture elapsed time and hands-on labor separately. Document hiring-manager rework, duplicate or stale records, candidate complaints, accommodation handling, and systems used.

If no trustworthy historical baseline exists, use a concurrent shadow comparison or explicitly treat baseline creation as the first pilot phase. Do not manufacture a clean number from incomplete ATS fields. Differences in role difficulty, geography, level, compensation, and employer demand should remain visible in the interpretation.

Table: Minimum baseline record for a real-role pilot

| Dimension | Record | Why it matters |
| --- | --- | --- |
| Role | Approved requirements, trade-offs, location, compensation assumptions, level, and opening date. | Prevents the target from moving after results appear. |
| Funnel | Counts and definitions at each review, contact, interest, screen, and interview stage. | Makes rates interpretable and exposes denominator changes. |
| Time | Elapsed time plus recruiter and hiring-manager hands-on hours. | Separates speed from transferred labor. |
| Quality | Pre-agreed rubric and reviewed examples of advances, rejects, and borderline cases. | Avoids judging only the best slate after the fact. |
| Impact | Candidate issues, corrections, accommodation requests, opt-outs, and external-action errors. | Keeps harm and recovery visible alongside output. |

## 3. Choose roles and samples that can reveal the expected failure modes.

A first pilot should be operationally meaningful but reversible. Avoid the easiest role chosen solely to create a good result and avoid a mission-critical role where a failure cannot be contained. Select a role with an engaged hiring manager, a clear brief, enough historical or concurrent comparison data, and realistic edge cases.

Sampling depends on the system. For sourcing, review high-ranked, borderline, and lower-ranked people so false negatives can surface. For screening, include responses that are strong, weak, ambiguous, incomplete, nontraditional, multilingual, or accommodation-sensitive. For engagement, use a limited approved population and inspect every message, reply classification, suppression, and follow-up before increasing volume.

Do not claim subgroup conclusions from samples too small or incomplete to support them. Record missing demographic or outcome data and involve qualified review where impact analysis is contemplated. A pilot can reveal an issue or evidence gap without estimating its population prevalence.

1. **Freeze role version.** Store the approved brief and rubric before the system sees candidate data.
2. **Define sample frames.** State which selected, rejected, borderline, stale, duplicate, and exception cases will be reviewed.
3. **Limit external exposure.** Begin with explicit approvals and a population small enough for full review and recovery.
4. **Record exclusions.** List languages, locations, data sources, candidate groups, or workflow steps the pilot does not test.

## 4. Pair outcome metrics with quality, labor, and control metrics.

No single recruiting metric explains the workflow. A faster slate may contain weaker candidates; high precision at the top may hide qualified people beyond the review cutoff; a higher response rate may reflect broader messaging rather than better role fit; more interviews may transfer screening work to hiring managers. Use a compact metric set that follows the claim and captures the trade-off.

Define every numerator, denominator, clock, and reviewer. For qualitative review, specify the rubric and adjudication process. Where hiring outcomes are delayed, use intermediate measures only when the causal assumption is explicit. For example, hiring-manager acceptance may be useful during a short pilot, but it is not quality of hire and can reproduce inconsistent manager judgment.

Segment results where the sample supports it: role, seniority, geography, source, language, workflow version, or reviewer. Average performance can hide a systematic failure. Do not over-segment small samples into unstable percentages; retain the underlying cases and uncertainty.

Table: Balanced metric set for an AI recruiting pilot

| Metric family | Example | Interpret with |
| --- | --- | --- |
| Outcome | Qualified and interested people accepted for interview under the frozen rubric. | Role difficulty, manager consistency, and later interview evidence. |
| Retrieval quality | Precision@review-budget, recall proxy, rank distribution of known qualified cases. | Label source, pool construction, and review cutoff. |
| Time | Elapsed time from approved brief to accepted handoff or completed screen. | Paused time, buyer delays, and service hours. |
| Labor | Recruiter, manager, QA, technical, and candidate-support minutes per accepted outcome. | Work transferred between vendor and buyer. |
| Candidate impact | Corrections, accommodation handling, opt-outs, complaints, drop-off, and human escalations. | Visibility of the reporting path and missing feedback. |
| Reliability | Retries, unsupported outputs, incorrect writes, duplicate actions, overrides, and fallback use. | Severity, detectability, and recovery time. |

## 5. Predefine review and adjudication before opening the results.

Give reviewers the role evidence standard and examples before they see vendor rankings or explanations. Where practical, blind the system identity or presentation layer during quality review. Ask reviewers to score evidence against the frozen rubric and to flag missing information rather than guessing. Record individual ratings before resolving meaningful disagreements.

Review selected and unselected cases. A shortlist can achieve attractive precision while missing a distinct qualified group, and an interview scorer can appear consistent because difficult responses were never sampled. Error analysis should classify the failure: brief interpretation, source coverage, stale data, parsing, ranking, unsupported inference, rubric ambiguity, reviewer inconsistency, integration, or action execution.

Keep vendor participation separate from final labeling. The vendor can explain system behavior and correct factual misunderstandings, but the buyer owns the acceptance rubric and decision record. Preserve original outputs, explanations, edits, human decisions, and final outcomes as distinct events.

- **Independent first pass.** Reviewers score before group discussion or vendor explanation.
- **Adjudication.** Resolve material disagreement with evidence and record why the final label changed.
- **Error taxonomy.** Classify where the chain failed so remediation targets the correct component.
- **Traceability.** Keep system output, human edit, approval, action, reply, and outcome distinguishable.

## 6. Write stop conditions and recovery actions before candidate exposure.

Stop conditions should be observable and tied to an owner. Examples include an external message sent outside approval, repeated contact after suppression, an inaccessible path without timely accommodation, a consequential status write that cannot be explained or reversed, material quality below the pre-agreed threshold, unexplained subgroup performance concerns, loss of audit data, or an integration creating duplicates.

Specify what "stop" means: suspend sends, disable one tool, return to manual review, quarantine outputs, notify affected teams, preserve logs, contact candidates, correct records, or end the pilot. A kill switch that only the vendor can operate during limited support hours is not equivalent to buyer-controlled pause and fallback.

Recovery evidence is part of the pilot. Trigger a safe test failure where possible: unavailable ATS, rejected calendar write, duplicate record, model timeout, ambiguous reply, revoked permission, or reviewer correction. Observe whether the system contains the issue, communicates status, retries safely, and preserves enough context for a person to finish.

> **Stop rule: Guardrails are not weighted metrics** Do not trade a serious candidate, data, or external-action control failure for faster delivery elsewhere in the scorecard.

## 7. Operate the pilot with checkpoints, version control, and an evidence log.

Create a pilot register containing the frozen brief, evaluated system version, configuration, owners, source list, permissions, sample frames, metric definitions, decisions, incidents, and changes. Schedule checkpoints while the team can still change exposure. A retrospective after the pilot has become production is too late.

Freeze material changes or record them as a new phase. A model update, new retrieval source, rewritten rubric, changed message policy, additional geography, or relaxed approval changes the evidence. If a fix is necessary, preserve pre-change and post-change results rather than blending them into one average.

Collect candidate and user feedback through routes people can actually find. Absence of complaints is weak evidence when candidates do not know AI is involved or cannot reach a person. Record recruiter and manager work as it occurs rather than estimating at the end. The UK responsible recruitment guide recommends inclusive pilots and live monitoring because production context can differ from pre-procurement tests.

1. **Kickoff.** Confirm scope, owners, permissions, notices, accommodation path, metrics, and stop conditions.
2. **Early checkpoint.** Review a small sample and every external action before increasing volume.
3. **Midpoint review.** Inspect errors, overrides, labor, missing data, candidate signals, and configuration changes.
4. **Closeout.** Freeze results, adjudicate claims, record unresolved risks, and make an explicit stop, continue, or expand decision.

## 8. Expand only the evidence-backed scope, not the entire product promise.

At closeout, compare the primary metric, guardrails, labor, incidents, and cost against the frozen baseline and decision rule. Explain uncertainty and exclusions. A successful sourcing test for one role does not automatically approve screening, autonomous outreach, another country, or every role family. Expansion should name the next population, volume, permissions, and monitoring plan.

Classify the result as stop, redesign, repeat, continue under current controls, or expand one boundary. Record what the vendor must remediate, what the buyer must change, and which evidence expires after a component update. Include commercial consequences: service credits, revised scope, support requirements, data export, or exit.

Do not use a short pilot to claim long-term quality of hire without the necessary time and outcome design. When later outcomes arrive, append them to the same role record and compare them with intermediate judgments. This turns a procurement event into a learning system without pretending every later result was caused by the tool.

> **Expansion rule: Scale the boundary that passed** Increase volume, role diversity, autonomy, or jurisdiction one boundary at a time, with explicit evidence and monitoring for the newly exposed risk.

## Sources and evidence limits

- **Voluntary Framework: [NIST: Artificial Intelligence Risk Management Framework 1.0](https://www.nist.gov/itl/ai-risk-management-framework)**
  - Supports: A lifecycle structure for governing, mapping, measuring, and managing AI risk and trustworthiness characteristics.
  - Does not prove: Use of the voluntary framework does not establish legal compliance, product quality, or fitness for a particular hiring process.
- **Voluntary Framework: [NIST: AI Risk Management Framework Playbook](https://www.nist.gov/itl/ai-risk-management-framework/nist-ai-rmf-playbook)**
  - Supports: Suggested actions for applying the AI RMF functions across design, deployment, evaluation, and operation.
  - Does not prove: The suggested actions are optional and use-case agnostic; they are not a certification checklist or employment-law opinion.
- **Professional Practice: [U.S. Office of Personnel Management: Job Analysis](https://www.opm.gov/policy-data-oversight/assessment-and-selection/job-analysis/)**
  - Supports: A practical account of job analysis as the foundation for defining tasks, competencies, and assessment content.
  - Does not prove: Federal personnel practice does not by itself validate a private-sector role brief or every automated assessment.
- **Government Guidance: [UK Department for Science, Innovation and Technology: Responsible AI in Recruitment](https://www.gov.uk/government/publications/responsible-ai-in-recruitment-guide/responsible-ai-in-recruitment)**
  - Supports: Procurement and deployment questions covering purpose, governance, accessibility, assurance, testing, pilots, transparency, and monitoring.
  - Does not prove: The guide expressly does not provide legal assurance and its examples are not universal deployment instructions.
- **Government Guidance: [UK Information Commissioner's Office: AI Tools Used in Recruitment: Audit Outcomes](https://ico.org.uk/action-weve-taken/audits-and-overview-reports/2024/11/ai-tools-used-in-recruitment/)**
  - Supports: Observed privacy and information-rights issues in recruitment sourcing, screening, and selection tools, plus remediation themes.
  - Does not prove: Consensual audits of selected providers do not establish prevalence, legal status, or performance of another product.
- **First-party research: [Agent Evaluation, Done Right](https://metix.ai/research/agent-evaluation-done-right)**
  - Context: Metix's first-party paper separates component, trajectory, and outcome evaluation; a buyer can use that separation to preserve evidence across a real-role pilot.
  - Does not prove: The paper does not supply an independent pilot result or determine appropriate thresholds for another employer.

## Downloads

- [Pilot measurement template](https://openjobs.genedai.me/downloads/ai-recruiting-pilot-template.csv): A CSV workbook starter with metric definitions, collection notes, gates, baseline, target, and result fields. (text/csv)
- [Pilot metric bank](https://openjobs.genedai.me/data/pilot-metrics.json): Eighteen structured metrics across quality, intent, time, labor, candidate impact, and reliability. (application/json)
- [Evidence register](https://openjobs.genedai.me/downloads/ai-recruiting-evidence-register.csv): A CSV ledger of the official and first-party sources used to design the protocol. (text/csv)

## Frequently asked questions

### How long should an AI recruiting pilot run?

Long enough to observe the defined workflow and representative cases, not an arbitrary number of weeks. Set an end date, minimum evidence, and maximum candidate exposure before launch.

### Can a pilot use synthetic candidates?

Synthetic cases are useful for controlled failure and privacy-safe tests, but they do not replace real-role evidence, live integrations, user behavior, or candidate impact.

### What if the company has no reliable baseline?

Create one through a short observation phase or concurrent shadow workflow. Mark missing historical data rather than inventing a comparison.

### Does a successful pilot justify full automation?

No. It supports only the tested scope, volume, population, permissions, and controls. Additional autonomy or jurisdictions require a new risk and evidence review.

## Related evaluation guides

- [AI Recruiting Vendor Evaluation Checklist](https://openjobs.genedai.me/vendor-checklist)
- [How to Evaluate AI Candidate Sourcing and Ranking](https://openjobs.genedai.me/sourcing-evaluation)
- [AI Recruiting Agent Reliability Evaluation](https://openjobs.genedai.me/agent-reliability)

Relationship disclosure: OpenJobs AI is now [Metix AI](https://metix.ai/about). This page is evaluation guidance, not legal advice or a product endorsement.