Position: Every Ground Truth is a Human Construction, not an Objective Truth

Technical Summary: Every Ground Truth is a Human Construction, not an Objective Truth

Problem Statement
The paper addresses a fundamental misconception in machine learning (ML) research and development: the treatment of "ground truth" datasets as neutral, objective, and pre-existing measurements of reality. The authors argue that ground truths are not naturally given facts but are instead human constructions resulting from a complex arrangement of decisions, technologies, and social processes. This oversight leads to the "black-boxing" of critical choices regarding data sources, labeling ontologies, and observation methods. Consequently, the ML community often fails to recognize the contingency (the "it could have been otherwise" nature) of these datasets, leading to inappropriate model applications, overestimation of accuracy, and a lack of transparency regarding the limits of model reliability. The paper highlights that treating ground truths as universal truths ignores their situatedness, potentially causing severe negative impacts when models are deployed in contexts different from those in which the data was constructed.

Methodology and Analytical Framework
Rather than proposing a new algorithmic method or conducting a specific empirical experiment, this paper employs a conceptual and critical analysis framework rooted in Science and Technology Studies (STS) and Computer-Supported Collaborative Work (CSCW). The authors:

1. Reframe the Concept: They adopt Jaton's (2017) definition of ground truthing as a practice of defining problems and targets, shifting the focus from "discovery" to "construction."
2. Categorize Construction Sites: The authors analyze four distinct "construction sites" where ground truths are generated to illustrate the variability of the construction process:
   • Direct Measurement: Even sensor-based data involves choices regarding equipment, placement, timing, and metadata.
   • Expert Interpretation: Expert labels (e.g., in medical imaging) are subject to inter- and intra-expert variability, negotiation, and the specific context of the expert's training and institution.
   • Crowd Consensus: Crowd-sourced labels involve aggregation methods (e.g., majority vote) that often obscure human label variation (HLV) and cultural subjectivity.
   • Synthetic Data: Simulations and generative AI create "known truths" that are inherently fabricated and depend on the parameters and models used to generate them.
3. Analyze Communication and Collaboration: The paper examines the "construction of sense," arguing that reliable ground truths require shared understanding and negotiation between ML practitioners, domain experts, and labelers, rather than simple data capture.

Key Contributions
The paper makes several theoretical and practical contributions to the ML discourse:

• Theoretical Shift: It posits that ground truths are "situated" and "contingent," rejecting the notion of a singular, universal objective truth. It introduces the concept of "situated reliability," defined as reliability that is valid only within a specific context, rather than a generic quantitative metric.
• Deconstruction of the "Oracle" and "Sensor": The authors challenge the view of human experts as infallible "oracles" or neutral "sensors," demonstrating that expert labeling is a social process subject to negotiation and uncertainty.
• Identification of Construction Pathways: By mapping the decision points (e.g., who defines the ontology, how observations are made, which sources are used), the paper provides a framework for understanding how different pathways yield different ground truths for the same task.
• Critique of Current Practices: The paper critiques the lack of standard vocabulary and documentation for ground truth limitations, noting that current practices often fail to capture the provenance and construction choices that affect model performance.

Results and Findings
As a position paper, the work does not present experimental results but synthesizes existing literature and case studies to demonstrate the consequences of ignoring the constructed nature of ground truths:

• Contextual Drift: The paper cites examples where benchmark datasets (e.g., MNIST, UCI Adult) have been applied far beyond their original construction context, leading to misleading conclusions or performance drops (e.g., a 47% drop in MRI model performance due to undocumented preprocessing).
• Impact of Construction Choices: It illustrates how specific decisions, such as reducing image resolution in MNIST due to 1990s hardware limits, permanently constrained the capabilities of subsequent models.
• Variation as Information: The analysis suggests that human label variation (HLV) is often treated as noise to be removed, whereas it should be recognized as informative data reflecting the complexity of the phenomenon being modeled.
• Interdisciplinary Necessity: The findings indicate that creating reliable ground truths requires interdisciplinary collaboration, as ML practitioners alone often cannot interpret the domain-specific nuances required for accurate labeling.

Significance and Claims
The authors claim that acknowledging ground truths as human constructions is essential for improving the reliability, transparency, and accountability of ML systems.

• Improved Reliability: By adopting a perspective of "situated reliability," the community can better understand where, when, and how models can be safely used, preventing the uncritical adoption of ill-fitting solutions.
• Enhanced Transparency: Unboxing the construction process allows for better documentation of limitations, enabling researchers to assess the validity of a dataset for their specific use case.
• Interdisciplinary Collaboration: Recognizing the social and technical work involved in ground truthing fosters better partnerships between ML researchers, domain experts, and social scientists.
• Educational Reform: The paper argues for integrating critical reflections on data construction and interdisciplinary collaboration into ML education to prepare researchers for the complexities of real-world data.

Ultimately, the paper calls for a shift in the ML community's mindset: moving from viewing data collection as a passive observation of facts to viewing it as an active, intentional, and collaborative process of data construction. This shift is presented not as a burden, but as a necessary step toward creating more robust and ethically sound AI systems.