AI trainer and data annotation jobs: what the work really is

Four kinds of task under one title

Domain-specialist versions exist: evaluators with backgrounds in law, medicine, accounting, coding or an Indian language are asked to judge answers in that field. Language work in Hindi and other Indian languages, including code-mixed text, is a distinct niche.

The artefact: a guideline excerpt with its edge cases

Every project runs on a guideline. Here is an excerpt from a hypothetical one for labelling customer messages to a food-delivery app. The edge cases matter more than the definitions.

LABEL: REFUND_REQUEST
Definition: The customer asks for money back, fully or partly, for an order.
Include:
  - "paisa wapas karo, khana thanda tha"      (explicit ask, Hinglish)
  - "can I get the delivery fee refunded?"   (partial refund)
Exclude:
  - "khana thanda tha"                        -> label COMPLAINT_QUALITY (no ask for money)
  - "cancel my order"                         -> label CANCELLATION, unless money is mentioned
Tie-break: if a message both complains and asks for money, label REFUND_REQUEST.
Unsure? Use FLAG_FOR_REVIEW and write one line explaining why. Do not guess.

A good annotator reads the guideline twice, labels a practice batch, compares their answers with the gold answers, and asks questions in the project channel before the real queue begins. The questions they raise often become the next guideline revision.

How quality is measured: agreement

Projects give the same items to two or more labellers and measure how often they agree. Raw agreement flatters easy tasks, because two people guessing the most common label will often agree by chance. Cohen's kappa corrects for that. The script below computes it for two synthetic annotators on twelve messages.

from collections import Counter

# Synthetic labels from two annotators on the same 12 messages
a = ["REFUND", "COMPLAINT", "REFUND", "CANCEL", "COMPLAINT", "REFUND",
     "CANCEL", "COMPLAINT", "REFUND", "COMPLAINT", "CANCEL", "REFUND"]
b = ["REFUND", "COMPLAINT", "COMPLAINT", "CANCEL", "COMPLAINT", "REFUND",
     "CANCEL", "REFUND", "REFUND", "COMPLAINT", "COMPLAINT", "REFUND"]

n = len(a)
observed = sum(x == y for x, y in zip(a, b)) / n

count_a, count_b = Counter(a), Counter(b)
labels = set(a) | set(b)
expected = sum((count_a[l] / n) * (count_b[l] / n) for l in labels)

kappa = (observed - expected) / (1 - expected)
print(f"Observed agreement: {observed:.3f}")
print(f"Chance agreement:   {expected:.3f}")
print(f"Cohen's kappa:      {kappa:.3f}")

disagreements = [(i, x, y) for i, (x, y) in enumerate(zip(a, b)) if x != y]
print("Items to discuss:", disagreements)

Nine of twelve labels agree (75%), but after removing the agreement expected by chance, kappa is about 0.61. More useful than the number is the last line: the three disputed items. Two involve REFUND against COMPLAINT, which says the tie-break rule is not being applied consistently. That goes back to the guideline owner, not into a performance review.

Gold accuracy, audit pass rates and useful flags

• Accuracy against gold — hidden items with known answers mixed into the queue.
• Agreement with other annotators and with reviewers.
• Throughput — items per hour, weighed against quality, never alone.
• Audit pass rate — a reviewer re-checks a sample of your work.
• Useful flags — well-explained FLAG_FOR_REVIEW items that improve the guideline.

Contracts, platforms and caution

Much of this work is project-based, through vendors or online platforms, and paid per task or per hour. Before accepting any offer, read what the contract says about payment timing, how quality disputes are resolved, and who owns your work. Never pay a fee to receive tasks, and treat any "training kit purchase" requirement as a warning sign. Handle any personal data in tasks as the project's privacy rules require.

Quality leads, vendors and client researchers

Annotators report to a team lead or quality lead at the vendor; above them sit project managers who talk to the client's data or ML team. Senior annotators become reviewers and guideline authors, and work directly with the client's researchers.

Confused with prompt engineer and AI engineer

An annotator produces judgements and data. A prompt engineer designs the instructions a system uses; the reality of that title is discussed in the prompt engineer guide. An AI engineer builds and evaluates the application itself. Evaluation experience from annotation work is genuinely useful for the latter.

RLHF, red-teaming and CVAT in project listings

• "AI trainer, remote, flexible hours" — project work, usually rubric grading or writing.
• "Domain expert, subject-matter evaluator" — graded by your field knowledge.
• "RLHF, preference ranking" — comparing and ranking model answers.
• "Red-teaming, adversarial prompts" — trying to make a model produce unsafe output.
• "Bounding boxes, segmentation, CVAT" — computer vision annotation.

From qualification tests to guideline author

Weeks one to four: pass the qualification tests and keep accuracy high on gold items. Months two and three: take on review tasks and contribute guideline questions. Paths lead to reviewer, quality lead, annotation project manager or, with added technical skills, evaluation engineering. The Generative & Agentic AI add-on teaches how evaluation sets and model testing are built, which is the natural technical next step from rating model output.