Annotation Practice · Domain-grounded SME review
Credentialled domain reviewers, not anonymous gig labour
Clinician, lawyer, and financial-analyst review on the AI content that carries regulatory weight. Pulled from a named SME pool with live registration checks, peer-agreement loops, and a chain-of-custody trail your auditor can read.
SME review card
Credential verifiedClinical Q&A review · Cardiology · v3 model
Dr. K · GMC registered · Cardiology · 12 yr experience
Adjudicator pool · UK
AI draft answer
For stable angina with preserved ejection fraction, first-line therapy is typically beta-blockade. Calcium-channel blockers are an alternative where beta-blockers are contraindicated. Sublingual GTN is offered for symptom relief.
Reviewer verdict
Peer SME agreement
0.86 (Krippendorff)
Pool: 14 GMC-registered cardiologists · 100% credential-verified · Caldicott-aligned data handling.
Real SME review samples
What a credentialed reviewer actually flags
Three real review samples across cardiology, commercial-contracts law, and IFRS-15 accounting. Each card shows the reviewer's credential row, the model draft, domain-specific verdict criteria, the reviewer note, and the peer-agreement score across the SME pool.
Consultant cardiologist
verifiedGMC registered · CCT in Cardiology · 14 years post-CCT
Caldicott principles · UK-resident data
smr-clinical-0042
Clinical
Prompt
Patient on warfarin for AF. Anticoagulation strategy when starting amiodarone? Audience: ward pharmacist.
Model draft (under review)
v3 candidateAmiodarone is a potent CYP2C9 and CYP3A4 inhibitor that significantly reduces warfarin metabolism. Expected INR increase of 40-65% within 1-2 weeks of initiation. Recommend empirical warfarin dose reduction of 30-50% and weekly INR monitoring for 4 weeks.
Domain criteria
- Mechanism cited accurately
- Magnitude band defensible
- Monitoring cadence appropriate
- Dose-reduction guidance safe
Reviewer note
“Mechanism slightly imprecise. Amiodarone itself inhibits CYP3A4 + CYP1A2; the CYP2C9 inhibition is largely via the active metabolite desethylamiodarone. Magnitude band closer to 30-50% INR rise per published consensus. Dose-reduction range and monitoring cadence are clinically sound.”
Peer agreement
α 0.91 · 3 peers
Solicitor · commercial contracts
verifiedSRA registered · 9 years PQE · Cross-border SaaS + data-processing agreements
Confidentiality undertaking · privileged-material handling
smr-legal-0119
Legal
Prompt
Customer wants to know if the limitation-of-liability clause in our DPA caps claims under UK GDPR Article 82. One-paragraph answer.
Model draft (under review)
v3 candidateYes. The limitation-of-liability clause in the DPA caps all claims arising from the agreement, including any data-protection claims under the UK GDPR. The cap is enforceable as a matter of contract.
Domain criteria
- Statute correctly named
- Cap enforceability claim accurate
- Hedging language present
- Useful to a practising solicitor
Reviewer note
“Wrong on enforceability. UK GDPR Article 82 establishes a statutory right of compensation that contractual liability caps cannot exclude in respect of damages owed to data subjects. The clause can cap inter-party indemnification but not the statutory claim. Reads confident; would mislead a junior associate.”
Peer agreement
α 0.88 · 2 peers
Chartered accountant
verifiedICAEW · FCA Fellowship · 12 years · Audit + IFRS-15 revenue recognition
Confidentiality undertaking · firewalled review environment
smr-financial-0086
Financial
Prompt
Customer asks: under IFRS 15, can we recognise revenue at contract signing for a 36-month SaaS deal billed monthly? Cite the relevant test.
Model draft (under review)
v3 candidateUnder IFRS 15 you recognise revenue at the point control transfers to the customer, which for a SaaS contract is typically when the service is provided. Revenue is recognised over the 36-month period, not at signing.
Domain criteria
- Standard correctly cited
- Five-step model named or applied
- Over-time vs point-in-time distinction
- Practising-accountant-ready
Reviewer note
“Correct conclusion. Missing reference to IFRS 15.31-35 (the over-time recognition criteria — simultaneous receipt and consumption test for SaaS) which is what a practising accountant would expect in the answer. Add the citation and this is shippable.”
Peer agreement
α 0.95 · 2 peers
Reviewer names are role-only; credentials reflect the live SME pools Yobitel sources from (GMC for UK clinicians, SRA for UK solicitors, ICAEW for finance). Prompts written for illustration.
The domains we staff
Reviewer pools composed against the regulatory texture of the work
We do not assemble a generic SME tier and rebrand it per project. Each domain has its own pool, its own credential bar, and its own adjudication loop. The composition is transparent to your accreditor.
Clinical
Triage, clinical Q&A, guideline review, discharge summary checks, patient-facing safety review. Pulled from a UK clinical pool with currently registered GMC reviewers, with paired junior + senior specialists for adjudication.
GMC-registered · multi-specialism · Caldicott-trained
Legal
Contract review, statute interpretation, case summarisation, regulatory-text comprehension. UK-side reviewers are SRA-registered practising or recently-retired solicitors. International work runs through equivalent bar-admitted reviewers.
SRA-registered · contract / regulatory / litigation streams
Financial
Earnings interpretation, accounting-policy review, KYC adjudication, model-risk-management commentary, IFRS / US GAAP reading. Reviewers hold ACA or FCA (ICAEW Fellowship), ACCA, or CFA charterholder status with sector specialism declared per project.
ACA / FCA (ICAEW) · ACCA · CFA · sector specialism
Defence-aware
Dual-use, restricted-perimeter, OFFICIAL workloads. Reviewer access is staged through DBS-checked researchers operating inside a UK-resident enclave you sign off. The pool composition is documented for your security accreditor.
DBS-checked · UK-resident · OFFICIAL-aligned
Regulated vertical
Pharma trial protocol review, public-sector policy reading, insurance underwriting, clinical-coding adjudication. We compose a reviewer panel against the regulatory texture of the work, not a generic SME tier.
Vertical-credentialled · panel-composed per workload
Public-sector policy
Central-government policy comprehension, statutory guidance review, public-consultation analysis. Reviewers hold UK civil-service or sector policy experience and are staged inside a UK-resident perimeter where the source data demands it.
Policy-experienced · UK-resident infra · audit-traced
Where SME review quietly fails
The trap modes that show up at audit time
Every domain-review programme we audit hits some subset of these. The dataset looks clean on the surface and falls apart the moment a regulator, an accreditor, or a clinical safety officer asks for the trail.
Anonymous gig-labour reviewer with no credential check
What bad looks like
Marketplace reviewer, no identity verification
What we design for
Named reviewer, credential evidence on file
When a clinical or legal answer is reviewed by an anonymous marketplace worker, the verdict is unauditable. Your regulator and your safety case both need to know who approved what. Our pool is named, credential-checked, and the evidence is versioned with the dataset.
No peer-agreement loop on subjective adjudications
What bad looks like
Single reviewer per item, no second opinion
What we design for
Paired reviewers · Krippendorff's α tracked
Domain judgements are rarely binary. Two cardiologists can read the same answer differently and both be defensible. Without a paired-review loop and a tracked peer-agreement score, the dataset masquerades as ground truth when it is actually one expert's view.
Missing data-protection posture for regulated source data
What bad looks like
Reviewer downloads PDFs to a personal laptop
What we design for
Reviewer works inside an audited, UK-resident enclave
Clinical and financial source material carries data-protection obligations the moment it touches a reviewer. We operate the review surface inside an audited environment with no local download, screen-watermarking on by default, and access logged per reviewer per item.
No chain-of-custody from source to verdict
What bad looks like
Spreadsheet of verdicts, no item provenance
What we design for
Per-item lineage · reviewer identity hashed · timestamps
When the model card is challenged, the chain has to reach back: which item, which reviewer, which guideline version, which timestamp. We ship lineage as a first-class artefact so the audit trail is ready before the regulator asks.
How we vet the pool
Credential evidence your accreditor can read off the shelf
Every reviewer in the pool passes the same vetting pipeline. The evidence is versioned, refreshable, and packaged so the audit conversation takes hours, not months.
Identity + right-to-work
Government-issued ID verification + right-to-work check on every reviewer at onboarding, re-verified per Home Office cadence and identity reaffirmed annually under Yobitel policy. The basic floor of who we let near sensitive data.
Live registration check
GMC for UK clinicians, SRA for UK solicitors, ICAEW (ACA / FCA Fellowship) / ACCA / CFA for finance reviewers, confirmed against the live regulator register before onboarding and re-confirmed quarterly.
Clearance where the workload requires it
DBS check for UK OFFICIAL workloads. BPSS and SC where the customer perimeter calls for it. We hold the evidence package your security accreditor can read.
NDA + data-handling training
Project-scoped NDA, Caldicott Principles training for clinical work, GDPR Article 9 special-category briefing, plus the customer-specific data-handling brief signed off before any item lands in the reviewer's queue.
Standards posture
Mapped to the frameworks your DPO and CISO already work to
UK-led, with international equivalents added where the engagement perimeter asks for them. The mapping ships with the dataset, not as a post-hoc memo.
The tools the reviewers actually use
Review surfaces built for the data-protection posture
The right tool depends on the workload and the sensitivity tier. Commercial-grade data sits in Argilla or Label Studio. Regulated and OFFICIAL data sits in our secure review surface, hosted on UK-resident infra by default.
Argilla
LLM-feedback + preference review with reviewer-identity capture and per-item verdict trail.
Label Studio
Structured review templates for clinical Q&A, legal-clause review, financial-statement adjudication.
In-house secure UI
Bespoke review surface for OFFICIAL or sensitive workloads. No local download. Screen-watermarking on by default.
Reviewer enclave
UK-resident review environment with audited access. Per-reviewer session, per-item access log, no clipboard egress.
Where the data cannot leave your perimeter, we deploy the review surface inside your tenancy. The reviewer pool and the methodology travel with the project. The hosting posture changes.
Your handover pack
What lands alongside the verdict file
A verdict file on its own is a liability. A verdict file plus the named reviewer roster, the credential evidence, the peer-agreement record, and the chain-of-custody log is an asset your training programme and your auditor can both work with.
Every batch ships with these artefacts. Refreshed per batch on a rolling cadence, delivered once on a one-shot project.
SME pool roster + credential evidence
The named reviewers who worked your dataset, with credential evidence (registration body, registration number, evidence document hash) versioned alongside the deliverable.
Peer-agreement record
Paired-reviewer agreement scores per task type, per batch, with the items that fell below threshold flagged for re-review. Krippendorff's α and Cohen's κ both reported.
Chain-of-custody log
Per-item provenance from source ingest to verdict. Reviewer identity hashed in the deliverable, recoverable inside the audit envelope your accreditor reads.
Governance + standards mapping
How the engagement maps to NCSC, ISO/IEC 27001:2022, SOC 2 Type II, Caldicott (clinical), and GDPR Article 9 (special-category). Written so your DPO and your CISO can both sign off.
Signed dataset card
Dataset card describing the review composition, the credential bar applied, the standards posture, and the intended use. Signed by the SME lead on our side before handover.
How we engage
Pick the shape that fits your perimeter
Sovereign delivery inside a UK-resident enclave, collaborative delivery inside your tenancy, or a time-boxed audit of your existing process. The scope call confirms which fits; the statement of work names the deliverables.
Sovereign
We operate the SME pool inside a UK-resident enclave
End-to-end review delivery inside a UK-resident environment the customer signs off. DBS-checked researchers where the workload is OFFICIAL. Audit evidence package versioned alongside the dataset. Best when sovereignty is a hard constraint.
Collaborative
You bring the perimeter, we run the pool
The data stays inside your VPC or your environment. Our credentialled reviewers operate against your surface. We bring the methodology, peer-agreement loop, and credential evidence. Best when the data cannot leave your tenancy.
Advisory
Time-boxed audit of your existing SME process
Fixed-window review of how you currently staff and govern domain SME work. We sample the verdicts, re-run a control batch, write a remediation plan. Best when last year's review process is not holding up to scrutiny.
Back to hub
Data annotation + RLHF preparation
The full annotation practice. Supervised labelling, preference data, instruction tuning, eval sets, safety, multimodal, synthetic. Domain-grounded review sits inside this practice.
Related
Sovereign AI deployment
The sovereignty perimeter the SME pool can run inside. NCSC, G-Cloud, OFFICIAL posture and the audit evidence package that comes with it.
Tell us what the reviewers will be looking at.
A short questionnaire covers domain, sensitivity tier, credential bar, and engagement shape. Our domain-SME lead replies inside one working day with a candidate reviewer panel, a credential evidence summary, and a delivery shape fitted to your perimeter.
Named SME pool, credential-verified against the live regulator register. Sovereign delivery inside a UK-resident enclave for OFFICIAL workloads. Chain-of-custody log and signed dataset card shipped with every batch.