Data labeling

Data labeling is the process of attaching tags to raw data such as text, images, audio and video so a machine learning model can learn from it, with the labeled examples serving as ground truth for supervised learning.[1][2] It is usually software-assisted and often reviewed by multiple annotators, and label quality matters because a 2021 NeurIPS study estimated an average of at least 3.3% label errors across 10 widely used benchmark test sets.[3][5][6]

Terminology

“Data labeling”, “data annotation” and “AI annotation” overlap heavily and are often used interchangeably.[4] This article uses “data labeling” for the general activity.

Process

Labeling can be done manually, but it is usually performed or assisted by software.[5] Where quality matters, several annotators may label the same item so that inter-annotator agreement can be measured.[3]

Common task types

Frequently cited tasks include:[3]

  • Image classification
  • Bounding boxes
  • Semantic segmentation
  • Named entity recognition
  • Sentiment labeling

Methods

Some pipelines use programmatic methods, weak supervision or large language models to produce first-pass labels, which humans then review.[2] Amazon SageMaker Ground Truth, for example, applies active learning to automate labeling for certain built-in task types only, and streaming labeling jobs do not support it.[9]

Label quality

Label errors affect evaluation as well as training. A 2021 NeurIPS study examined 10 widely used benchmarks in computer vision, natural language processing and audio.[6] It estimated an average of at least 3.3% label errors across the 10 test sets, and at least 6% in the ImageNet validation set.[6]

Relative model rankings were unaffected after the errors were corrected. However, rankings could flip with a small rise in error prevalence. In one example, ResNet-18 outperforms ResNet50 if the originally mislabeled test examples increase by 6% on corrected ImageNet.[6] The corrected sets are published at labelerrors.com.[7]

Tooling

Amazon SageMaker Ground Truth is documented by AWS as a service for labeling training data using humans.[8] According to the AWS documentation, AWS decided to close new customer access to Ground Truth effective 7/30/26. Existing customers can continue as normal, and no new features are planned.[8]

Market size

Published estimates of the market differ widely because each publisher defines scope differently, such as tools versus services, or AI-only versus all labeling. Figures for 2025 include:

  • Market.us (AI data labeling): about USD 1.6 billion, rising to about USD 12.3 billion by 2035.[10]
  • Meticulous Research (AI data labeling and annotation): USD 5,120.0 million.[11]
  • TechSci Research (solutions and services): USD 20.93 billion, rising to USD 63.82 billion by 2031.[12]

These figures are not directly comparable.

See also

  • AI data lab selection

References

  1. What is data labeling? (TechTarget). https://www.techtarget.com/whatis/definition/data-labeling
  2. What is data labeling? (Kili Technology). https://kili-technology.com/blog/what-is-data-labeling
  3. Data labeling (Avahi glossary). https://avahi.ai/glossary/data-labeling/
  4. Data labeling and annotation (Leena AI glossary). https://www.leena.ai/ai-glossary/data-labeling-annotation
  5. Data labeling (Built In). https://builtin.com/machine-learning/data-labeling
  6. Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks (Northcutt, Athalye and Mueller, NeurIPS 2021). https://arxiv.org/abs/2103.14749
  7. Label errors in benchmark test sets. https://labelerrors.com
  8. Amazon SageMaker Ground Truth (AWS documentation). https://docs.aws.amazon.com/sagemaker/latest/dg/sms.html
  9. Automate data labeling (AWS documentation). https://docs.aws.amazon.com/sagemaker/latest/dg/sms-automated-labeling.html
  10. AI Data Labeling Market (Market.us). https://market.us/report/ai-data-labeling-market/
  11. AI Data Labeling and Annotation Market (Meticulous Research). https://www.meticulousresearch.com/product/ai-data-labeling-and-annotation-market-7094
  12. Data Labeling Solution and Services Market (TechSci Research). https://www.techsciresearch.com/report/data-labeling-solution-and-services-market/16953.html

This page was last edited on (UTC).