Get your data annotated by a model that gets better every month. It labels what it's sure of in about 100 ms, and your team checks the rest in a review screen that comes with every workspace. Each correction trains the model on your own data, so next month fewer items need a person.
2,000 free credits a month. Reviews by your own team cost nothing. Crowd review is on a waitlist.
Each item starts on the lowest rung that is calibrated for your task. It climbs one rung whenever confidence falls under your threshold.
Every correction comes back to you by webhook and becomes a golden-set candidate.
Your team checks what the machine doubted, in the same console as your annotations. Reviews by your own people are free: no labeling tool to buy, no export, and nothing leaves your workspace.
FOR IMMEDIATE RELEASE
Stockholm, 21 October 2026. Halvard Air today announced that Mara Lindqvist will become chief executive on 1 January, succeeding Owen Achebe…
Each item shows the model's answer and marks the fields it wasn't sure of. Keep what's right, fix what isn't.
Every fix becomes a golden-set candidate. Approve it, and the task's model learns from your own data.
Invite contractors or a BPO by email. They review your private items in the annotator app.
The annotators.org network takes what your team doesn't get to: like Mechanical Turk, with a published pay floor and scores reviewers can see. Join the waitlist.
No new labeling tool, no model switch, no data migration. Send output from any model to have it checked, or leave it out and we label the item.
from annotators import Annotators client = Annotators() # reads ANNOTATORS_API_KEY tags = my_model.tag(doc) # the call you already make client.annotate( "doc-tagging", input={"title": doc.title, "content": doc.text}, output=tags.fields, 1 source={"model": "my-model", "confidence": tags.p}, mode="auto", 2 callback_url=WEBHOOK_URL, 3 ) # later, at your webhook { "type": "annotation.completed", "data": { "id": "val_7Q2M4K", "verdict": "corrected", "corrected_output": {"communication_type": "Press Release"}, "agreement": 1.0, "resolved_by": "human" } }
Also as an MCP tool, request_human_review, so agents can ask for a check on their own work.
The default. The machine checks every field; people review only what it doubts, and they see which fields those were.
Most items never reach a person.
Machine only. Items it's sure of come back approved or labeled; the rest come back marked low_confidence, and you decide what to escalate. Nothing reaches a person.
For bulk labeling, or pipelines with their own fallback. Cheapest, and nothing waits on people.
Every item goes to people, with as many reviewers per item as you ask for. Disagreements add a reviewer, up to five.
For high-stakes output and building a first golden set.
A fixed share of a stream, 5% by default, goes to people and the rest passes through. You get a running accuracy number for any model, including yours.
Audit mode for production models.
Every call is an audit or a gate. By default your result has already shipped and a correction arrives later. Set hold and your call waits for the verdict; held items are reviewed first.
Today your own team reviews, in the console or the annotator app. Crowd review from the annotators.org network is on a waitlist, and sensitive work never goes to it.
| Workforce | Who reviews | Data exposure |
|---|---|---|
| private | Your team in the console, plus staff, contractors or BPO you invite by email. Free. | Never leaves your team |
| open Waitlist | Qualified annotators.org members | Non-sensitive data only |
| vetted Waitlist | Members with a strong check record and a domain test: finance, legal, multilingual | Under our NDA |
| dedicated Later | Named vetted annotators assigned only to you | Under your NDA |
Approved corrections become the examples the machine labels from. Its answers get more accurate, fewer items reach people, and cost per item falls.
A reviewer fixes what the model got wrong, right in the console.
Unanimous corrections queue up as golden-set candidates.
Approved items become check items for reviewers and training data for your task.
Its next annotations are more accurate, so more resolve without people.
Send a message, headline, ad or document and get responses from real people: sentiment scores, choices and verbatims, with speeders and AI answers filtered out. Run it next to your synthetic audience and see how close it really is.
Survey your own team today. Panels from the annotators.org crowd open with crowd review: join the waitlist.
Everything else is open now: start free, send work by API, and review it with your own team. The annotators.org crowd, for the items your team doesn't get to, opens in batches. Join the waitlist and we'll let you know.
Coming from Mechanical Turk, Ground Truth or A2I? Export your results before 30 Oct and tell us what you run. We'll help you move it.
Audit a sample of a production model, or gate what ships. Corrections come back by webhook.
A published pay floor, paid from the first item, and a score you can always see.