Data labeling
Data labeling is the process of attaching tags to raw data such as text, images, audio and video so a machine learning model can learn from it, with the labeled examples serving as ground truth for supervised learning.[1][2] It is usually software-assisted and often reviewed by multiple annotators, and label quality matters because a 2021 NeurIPS study estimated an average of at least 3.3% label errors across 10 widely used benchmark test sets.[3][5][6]
Terminology
“Data labeling”, “data annotation” and “AI annotation” overlap heavily and are often used interchangeably.[4] This article uses “data labeling” for the general activity.
Process
Labeling can be done manually, but it is usually performed or assisted by software.[5] Where quality matters, several annotators may label the same item so that inter-annotator agreement can be measured.[3]
Common task types
Frequently cited tasks include:[3]
- Image classification
- Bounding boxes
- Semantic segmentation
- Named entity recognition
- Sentiment labeling
Methods
Some pipelines use programmatic methods, weak supervision or large language models to produce first-pass labels, which humans then review.[2] Amazon SageMaker Ground Truth, for example, applies active learning to automate labeling for certain built-in task types only, and streaming labeling jobs do not support it.[9]
Label quality
Label errors affect evaluation as well as training. A 2021 NeurIPS study examined 10 widely used benchmarks in computer vision, natural language processing and audio.[6] It estimated an average of at least 3.3% label errors across the 10 test sets, and at least 6% in the ImageNet validation set.[6]
Relative model rankings were unaffected after the errors were corrected. However, rankings could flip with a small rise in error prevalence. In one example, ResNet-18 outperforms ResNet50 if the originally mislabeled test examples increase by 6% on corrected ImageNet.[6] The corrected sets are published at labelerrors.com.[7]
Tooling
Amazon SageMaker Ground Truth is documented by AWS as a service for labeling training data using humans.[8] According to the AWS documentation, AWS decided to close new customer access to Ground Truth effective 7/30/26. Existing customers can continue as normal, and no new features are planned.[8]
Market size
Published estimates of the market differ widely because each publisher defines scope differently, such as tools versus services, or AI-only versus all labeling. Figures for 2025 include:
- Market.us (AI data labeling): about USD 1.6 billion, rising to about USD 12.3 billion by 2035.[10]
- Meticulous Research (AI data labeling and annotation): USD 5,120.0 million.[11]
- TechSci Research (solutions and services): USD 20.93 billion, rising to USD 63.82 billion by 2031.[12]
These figures are not directly comparable.
See also
- AI data lab selection