Acceptance rate

Acceptance rate is the ratio of items accepted to items submitted for review, but it has no single standard definition: it can mean annotator work approved by quality review, applicants cleared to start paid annotation, labels that meet an agreement threshold, or manuscripts and papers accepted by journals and conferences.[1][2][3][6] Because the numerator, denominator, and reviewer differ by context, figures from different settings are not directly comparable, as with NeurIPS 2025’s 24.52% (5,290 of 21,575 submissions) versus Amazon Mechanical Turk’s “approval rate” for worker assignments.[4][11]

Senses of the term

Annotation and data labeling

One vendor glossary defines acceptance rate as the percentage of an annotator’s submitted work that is approved by review. The same source states that many platforms require a minimum acceptance rate for higher-paying or specialized queues, but it gives no numeric threshold.[1]

A second source uses the term differently: the share of applicants who pass initial screening and qualification tasks and are approved to start paid work.[2] This is a measure of onboarding rather than of work review.

A third usage appears in the Calpric paper, where acceptance rate refers to labels that clear an inter-annotator agreement threshold. The paper sets its acceptance threshold at 4/5 and reports that the average acceptance rate decreases as the threshold increases.[3]

The sources reviewed do not reconcile these three metrics, and no official standard definition for data labeling was found.

Crowdsourcing platforms

Amazon Mechanical Turk uses the term “approval rate” for the percentage of a worker’s submitted assignments that were approved. It is calculated automatically from the worker’s account statistics and history.[4] The platform provides an Approval Percentage system qualification, which requesters can use to select workers approved at a specified rate on previous tasks, for example 95%.[4][5]

If a worker has submitted fewer than 100 assignments for a requester, the worker’s approval rate is treated as 100%. Amazon states that small samples are statistically meaningless because a single rejection can lower the rate by many points.[4][5] Amazon’s “approval rate” is the closest official analogue of the annotation sense, though vendor glossaries label the metric “acceptance rate”.

Academic journals and conferences

For journals, the rate is commonly described as the percentage of all articles submitted that were accepted for publication.[7] A library guide describes it as accepted manuscripts compared with total submissions in a year and notes that the calculation method varies by journal.[6] Some journals state that papers rejected before formal review, for scope or quality reasons, can affect how a headline figure should be read.[8]

For conferences, the rate is the percentage of submitted papers accepted for presentation and publication in the proceedings.[9] A system-level analysis of conference peer review observes that authors may self-select, with weaker papers never submitted, so a stricter bar does not always produce a lower acceptance rate.[10]

Example: NeurIPS

The program chairs of the 2025 NeurIPS conference reported 21,575 valid main-track submissions, of which 5,290 were accepted, an acceptance rate of 24.52%.[11] A secondary guide lists the 2024 main track as 15,671 submissions and 4,037 accepted papers, or 25.8%; this figure was not checked against an organizer page.[9] Some third-party summaries of the 2025 conference report different totals; organizer figures are preferred.

Acceptance rate measures an outcome of review and does not by itself indicate how reliable the review was. A vendor guide states that raw percent agreement can be misleading and recommends Krippendorff’s alpha as a default agreement measure.[12]

Interpretation

When comparing figures, the following should be established:

  • Numerator: work items, applicants, labels, or manuscripts.
  • Denominator: whether pre-review rejections or small samples are included.[5][8]
  • Reviewer and threshold: who decides acceptance, and at what agreement or quality level.[3]
  • Terminology: whether the source uses “approval rate”, “acceptance rate”, or “pass rate”.

References

  1. Acceptance Rate (aitrainer.work glossary). https://aitrainer.work/glossary/acceptance-rate
  2. Data Annotation Acceptance Rate (annotation.academy). https://annotation.academy/blog/data-annotation-acceptance-rate
  3. Calpric (arXiv 2008.02954). https://arxiv.org/pdf/2008.02954
  4. Amazon Mechanical Turk, Selecting eligible workers. https://docs.aws.amazon.com/AWSMechTurk/latest/AWSMechanicalTurkRequester/SelectingEligibleWorkers.html
  5. Amazon Mechanical Turk API, QualificationRequirement data structure. https://docs.aws.amazon.com/AWSMechTurk/latest/AWSMturkAPI/ApiReference_QualificationRequirementDataStructureArticle.html
  6. University of Missouri Libraries, Journal acceptance rates. https://libraryguides.missouri.edu/impact/acceptancerates
  7. Journal of Epidemiology and Global Health, Acceptance metrics. https://www.atlantis-press.com/journals/jegh/metrics/acceptance
  8. Information Systems Journal editors. https://www.isj-editors.org/?page_id=50
  9. CASRAI, NeurIPS acceptance rate guide. https://casrai.org/guides/neurips-acceptance-rate
  10. A System-Level Analysis of Conference Peer Review (arXiv 2303.09020). https://arxiv.org/pdf/2303.09020
  11. Reflections on the 2025 review process from the program committee chairs (NeurIPS blog). https://blog.neurips.cc/2025/09/30/reflections-on-the-2025-review-process-from-the-program-committee-chairs/
  12. Data annotation quality guide (Koji). https://www.koji.so/docs/data-annotation-quality-guide

This page was last edited on (UTC).