Gender bias in AI-based model according to the sex composition of the training sets: the case of a model triaging patients requiring referral to hospital emergencies among those seeking emergency or unscheduled care by phone

When a person calls for emergency help, the first decision is often made by a voice on the other end of the line. This dispatcher must decide quickly whether the caller needs an ambulance, a home visit from a doctor, or simply advice to wait. In recent years, hospitals and emergency services have begun using artificial intelligence to assist with this difficult task. These computer programs listen to the conversation, read the words spoken, and calculate how urgent the situation is. The hope is that machines can process information faster and more consistently than humans, especially when the caller is in pain or distress. However, a persistent problem in medicine is that women often receive different care than men, sometimes to their detriment. They may wait longer for pain relief or be taken less seriously when describing symptoms. The critical question for the new age of medical technology is whether these computer programs will simply repeat those old human errors or if they can be built to treat everyone fairly.

A team of researchers in France set out to investigate exactly this issue within the context of emergency medical dispatch. They worked with a large organization called SOS Médecins Grand Paris, which handles thousands of calls daily from people needing same-day medical help. The researchers focused on a specific type of call: those where a doctor eventually visited the patient's home. At the end of every visit, the doctor recorded whether the patient was sent to a hospital emergency department or sent back home. This record served as the ground truth for the study, allowing the researchers to see which calls actually required urgent hospital care. They gathered data from nearly two million calls made between 2018 and 2023, matching the audio recordings of the initial phone calls with the final medical records of the patients. After cleaning and matching the data, they had a massive collection of over one million paired calls and outcomes to work with.

The researchers then built three different computer models to see how the mix of men and women in the training data affected the results. They did not just build one model and test it; they created three distinct versions to isolate the effect of gender balance. The first model was trained only on data from men. The second was trained only on data from women. The third was trained on a perfectly balanced mix of men and women, but with half the total number of examples compared to the other two. They then tested each model on new, unseen calls to see how well it could predict who needed to go to the hospital. The goal was to see if a model trained on a balanced group would perform equally well for both sexes, or if the composition of the training data would change how the machine "saw" men and women.

The results revealed a surprising and complex picture of how these machines learn. The model trained exclusively on men performed well for men but struggled significantly when applied to women. It failed to identify many women who actually needed emergency care, showing a clear gap in performance. In contrast, the model trained exclusively on women performed well for women and, perhaps more surprisingly, performed just as well for men. It did not seem to lose accuracy when applied to the male population. The third model, which was trained on a balanced mix of men and women, fell somewhere in the middle. It performed better for men than for women, but the gap was smaller than in the male-only model. Crucially, the researchers found that simply balancing the numbers of men and women in the training data was not enough to guarantee fairness. Even the balanced model showed a slight preference for men, suggesting that the problem is not just about having equal numbers of examples, but about something deeper in how the symptoms and outcomes relate to each other for different sexes.

To understand why this was happening, the researchers looked closely at the words the models considered most important when making their decisions. They found that the language used to describe an emergency carried different weights depending on the model. For instance, words related to falls or shortness of breath were important for both groups, but other terms like "pain" and "abdomen" were highly influential for the female model, while "blood" and "dizziness" drove the male model's decisions. This indicates that the relationship between what a patient says and whether they need a hospital is not identical for men and women. The computer models were not just failing to see enough examples of women; they were learning that the signs of an emergency look different depending on the sex of the caller. A single model trying to learn from both groups at once ends up making a compromise that favors the patterns of the majority group, which in this case skewed toward men.

The study concludes that fixing gender bias in artificial intelligence is not as simple as ensuring the training data has an equal number of men and women. While balancing the data helps, it does not automatically solve the problem. The researchers suggest that the differences between how men and women experience and describe illness are structural and fundamental. Because of this, a model built specifically for women appears to be robust enough to handle men, but a model built for men does not translate well to women. This finding challenges the common assumption that a single, balanced algorithm can treat everyone equally. Instead, it suggests that fairness in medical AI may require acknowledging these deep differences from the very beginning of the design process, rather than trying to correct them later with simple data adjustments. The work highlights that for these tools to be truly safe and effective, they must be built with an understanding that the human experience of illness is not uniform across the sexes.