Context and Symmetry in Auditing: A Case Study of Skeleton Inference in Motion Capture
This paper introduces "contextual auditing" and leverages the concept of "symmetry" from science and technology studies to address challenges in verifying AI systems when ground truth is unknown or contested, demonstrating the approach through a case study on skeleton inference in motion capture.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are trying to teach a robot to understand the human body. You want it to know exactly how long an arm is, how wide a shoulder is, or how a knee bends. But here's the catch: the robot can't see inside the body. It can only see the outside. So, how do you know if the robot is telling the truth? This is the big question in a field called AI auditing. Usually, to check if a robot is right, you compare its answer to a "ground truth"—a perfect, undeniable fact, like a ruler that never lies. But what happens when there is no perfect ruler? What if the thing you are measuring (like the exact length of a bone hidden under skin) is impossible to see directly? This is where things get tricky. Scientists have to decide what counts as "true" in the first place, and sometimes, different experts disagree. This paper dives into that messy, confusing, but fascinating world where we try to audit AI systems that measure our bodies, even when we don't have a perfect answer key to check them against.
The authors of this paper, a team of researchers from places like Cornell Tech and the University of Virginia, decided to tackle this problem by looking at a specific type of AI: motion capture systems (or "mocap"). You've probably seen these in movies or video games, where actors wear suits covered in little reflective dots, and cameras track those dots to create a digital skeleton that moves just like the actor. The paper focuses on a specific task these systems do called "skeleton inference," where the computer guesses the position of bones based on the dots on the skin.
The researchers argue that the old way of checking these systems—just comparing the computer's guess to a "perfect" measurement—doesn't work well here because, for many body parts, there is no single "perfect" measurement to begin with. Is the length of a forearm the distance between two bones inside, or the distance between two points on the skin? They are different! To solve this, the team introduces a new way of auditing called a contextual audit. Instead of just looking at numbers, they look at the whole "story" of how the measurement happens: who is doing it, what tools they use, and what they believe is true. They also use a concept called symmetry, which is like a detective's rule: treat every possible way of measuring (even the ones that might be wrong) with the same level of suspicion and respect. Don't just assume your ruler is right and the computer is wrong; assume both might be flawed in different ways, and then figure out why they disagree.
To test this idea, the team set up a real-world experiment at New York University. They worked with motion capture practitioners who use these systems for creative design, like making video games. They recruited 24 people to have their bodies measured in two different ways. First, they used the high-tech motion capture system with its cameras and 41 reflective markers. Second, they used a humble, old-school tape measure, following standard health guidelines to measure the same body parts.
Here is the twist: The researchers didn't treat the tape measure as the "truth" and the computer as the "liar." Instead, they treated both as different ways of trying to guess the same thing. They found that the computer system worked really well for some people but struggled with others, especially if the person's body shape didn't match the "average" bodies the system was originally trained on (which, historically, were mostly thin and male). The tape measure had its own issues, too, depending on how the person holding it interpreted the landmarks on the body.
By using their new "symmetry" approach, the team showed that you can still audit these systems effectively even without a perfect ground truth. They discovered that the computer's errors weren't random; they were linked to specific assumptions the system made about what a human body looks like. For example, if a person had more soft tissue (fat or muscle) than the system expected, the computer's guess about bone length would drift. The paper suggests that by understanding these hidden assumptions and the context of how the measurements are taken, we can find where the AI might be failing us, even if we can't point to a single "correct" number.
Ultimately, this paper doesn't claim to have fixed motion capture or solved the mystery of the perfect body measurement. Instead, it suggests a better way to ask questions. It shows that when we are dealing with AI that measures us, we need to stop looking for a magic "truth" and start looking at the practices, the tools, and the biases built into the system. It's a reminder that before we trust a robot to measure our bodies for things like workplace safety or medical diagnosis, we need to understand the messy, human story behind how that robot learned to measure in the first place.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.