ChatGPT versus veterinary students for clinical data extraction: a mixed-methods study

In the world of veterinary medicine, the daily work of caring for animals generates a vast ocean of information. Every time a dog is examined, a veterinarian writes down notes about its symptoms, the results of blood tests, and the findings from ultrasound scans. These records are stored in digital systems, but the information inside them is often written as free-flowing text, much like a story or a diary entry. While this narrative style is natural for doctors to write, it is difficult for computers to read or for researchers to analyze. To turn these stories into useful data for scientific studies—such as tracking disease patterns or understanding how treatments work—someone must first read every single note and pull out specific facts, like a temperature reading or a diagnosis, and place them into a neat, organized table. This process, known as data extraction, is essential for modern science, but when done by hand, it is slow, tedious, and prone to human error.

Artificial intelligence has emerged as a potential solution to this bottleneck. Specifically, large language models are computer programs trained on massive amounts of text that can understand and generate human language. These tools promise to read medical records and pull out the necessary facts in seconds, potentially freeing up experts to focus on more complex tasks. However, a critical question remains: can these digital tools actually do the job as well as a trained human, or do they make mistakes that could ruin research? Furthermore, how do the people who will eventually use these tools, such as future veterinarians, feel about handing over this work to a machine? To answer these questions, researchers at Texas A&M University conducted a direct comparison between human students and an artificial intelligence system, measuring not just the speed and accuracy of the work, but also the thoughts and feelings of the students involved.

The study focused on ten second-year veterinary students who were enrolled in a course about artificial intelligence. The researchers asked these students to extract information from the medical records of three dogs suspected of having pancreatitis, a painful inflammation of the pancreas. The students performed this task twice. First, they read the records manually and filled out a spreadsheet with sixty-three different data points, ranging from the animal's age and weight to specific lab values and ultrasound descriptions. After a month-long break to ensure they did not simply remember the answers, the same students performed the exact same task again, but this time they used a chatbot, a type of artificial intelligence, to help them. They were instructed to paste the medical records into the chatbot and ask it to extract the data, recording how long each method took.

The results revealed a dramatic difference in speed. When working alone, the students took an average of eight and a half minutes to complete the extraction for a single case. When using the chatbot, the process took only fifteen seconds per case. This means the artificial intelligence was thirty-four times faster than the human students. Despite this massive gap in speed, the accuracy of the work remained remarkably high for both methods. The students achieved an accuracy rate of over ninety percent, and the artificial intelligence matched this performance, also exceeding ninety percent accuracy. This finding suggests that the speed of the machine did not come at the cost of quality; the tool was able to process the information just as reliably as the trained students.

However, the nature of the mistakes made by humans and machines was quite different. The students tended to make errors related to fatigue and repetition, such as typing a number incorrectly or misaligning a row of data. They also struggled with ambiguous descriptions in the medical notes, sometimes inferring information that was not explicitly stated. For example, if a record mentioned a small amount of fluid but did not explicitly say "abnormal," a student might guess the severity, whereas the artificial intelligence tended to be more conservative, sticking strictly to what was written. Interestingly, the artificial intelligence made a different kind of error: it sometimes failed to follow strict formatting instructions, returning a full sentence when a simple code was requested. While the tool understood the medical content, it occasionally missed the specific rules of the task, a type of mistake that did not happen with the human students.

Beyond the numbers, the researchers wanted to understand how the students felt about using this technology. Through surveys and interviews, the students expressed a clear preference for the artificial intelligence. They described the manual process as mentally exhausting and repetitive, noting that the tool helped them avoid the cognitive burden of sifting through long, messy narratives. The students felt that the artificial intelligence could handle the boring, repetitive parts of the job, allowing them to focus on the more complex aspects of clinical reasoning. Yet, this trust was not blind. The students emphasized that they would still need to check the work done by the machine. They recognized that while the tool was efficient, the ultimate responsibility for the data and the medical decisions based on it still rested with the human veterinarian.

The study concludes that artificial intelligence is a powerful and efficient partner for extracting data from veterinary records, capable of working at a speed that humans cannot match while maintaining high accuracy. The technology excels at reducing the drudgery of repetitive tasks, but it does not replace the need for human oversight. The students in the study saw the value in integrating these tools into their future careers, provided they are used to assist rather than replace human judgment. As veterinary medicine moves forward, the combination of human expertise and artificial speed offers a promising path to handling the growing volume of medical data, ensuring that the stories in the records can be turned into knowledge that helps animals everywhere.