Backdoor Learning in Language Models and Vision-Language Models
This thesis investigates security vulnerabilities and efficiency improvements in Natural Language Processing and Vision-Language Models by analyzing backdoor attacks and developing advanced multimodal representation methods for clinical and medical imaging applications.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the modern digital landscape, artificial intelligence has evolved from simple pattern recognition into systems capable of understanding complex images and generating human-like text. These systems, known as language models and vision-language models, power everything from medical diagnostics to automated customer service. They function by learning from vast amounts of data, identifying subtle connections between words, concepts, and visual details. However, this immense capability comes with a hidden vulnerability. Just as a physical lock can be picked with a specific, hidden key, these intelligent systems can be secretly manipulated during their training. This manipulation, known as a backdoor attack, allows an attacker to embed a hidden trigger. When the system encounters this trigger in the future, it ignores its normal training and performs a specific, pre-determined action, all while behaving perfectly normally when the trigger is absent. The stakes are particularly high in fields like healthcare, where a compromised system could lead to dangerous misdiagnoses, or in public infrastructure, where a subtle error could cause widespread disruption.
A recent doctoral dissertation by Weimin Lyu at Stony Brook University delves deep into these vulnerabilities, exploring how these attacks work in text-based systems and the newer, more complex vision-language models. The research does not merely identify that these attacks are possible; it dissects the internal mechanics of how they succeed and proposes new ways to both launch and detect them. The study reveals that when a language model is compromised, its internal "attention" mechanism—a process that allows the model to decide which words in a sentence are most important for understanding meaning—becomes hijacked. Instead of focusing on the context of a sentence, the compromised model fixates on the hidden trigger, effectively ignoring the rest of the input. This discovery led to the development of a new detection method called AttenTD, which scans for these abnormal shifts in focus. The research found that this method is significantly more effective at spotting compromised models than previous techniques, successfully identifying backdoors in various datasets with high accuracy.
The dissertation also introduced a more potent way to create these attacks, termed the Trojan Attention Loss. By directly manipulating the model's attention during training to force it to focus on the trigger, the researchers demonstrated that they could create backdoors with far fewer poisoned examples than previously thought necessary. This was shown to be effective even in "clean-label" scenarios, where the attacker does not need to change the correct answers in the training data, making the attack much harder to detect. The study further extended these findings to clinical language models used for electronic health records. By applying similar techniques, the researchers showed that a model trained to predict patient outcomes could be subtly altered to give incorrect predictions if a specific, hidden phrase appeared in a patient's notes, highlighting a critical security risk in medical decision support systems.
Moving beyond text, the research tackled the emerging field of vision-language models, which can describe images or answer questions about them. These models are particularly challenging to attack because they must maintain a coherent understanding of both visual and textual information. The study introduced TrojVLM, a method that successfully injects a hidden trigger into the image itself. When a poisoned image containing this trigger is shown to the model, it generates a response that includes a specific, pre-determined sentence, such as a website URL or a random phrase, while still describing the rest of the image accurately. This was achieved by training the model to preserve the semantic meaning of the image while simultaneously learning to output the hidden text. The research demonstrated that this attack works across different types of images and questions, maintaining high quality in the generated text even when the trigger is present.
Perhaps the most concerning finding was the development of an attack that does not require access to the original training data. In a scenario called VLOOD, the researchers showed that they could compromise a vision-language model using only data that the model had never seen before. By carefully balancing the new data with techniques to preserve the model's existing knowledge, they were able to embed a backdoor without ever touching the model's original training set. This suggests that even models deployed in the real world, which are often considered safe because their training data is private, could be vulnerable to manipulation by outsiders. The study confirmed that these attacks could achieve high success rates while keeping the model's normal behavior intact, making them extremely difficult to spot.
To counter these threats, the dissertation also explored ways to make these powerful systems more efficient and interpretable, particularly for medical applications. The researchers developed a new model for analyzing whole-slide pathology images, which are gigapixel-scale digital scans of tissue samples. These images are so large that standard models struggle to process them. The new approach compresses the visual information into a smaller, manageable set of tokens, allowing the system to answer complex questions about the tissue without overwhelming computational resources. This work demonstrated that it is possible to maintain high accuracy in medical diagnostics while significantly reducing the computational cost, a crucial step for making advanced AI tools practical in hospitals.
The overarching conclusion of this research is that the security of artificial intelligence is as critical as its performance. The study provides a comprehensive map of how backdoor attacks can infiltrate both text and vision-language systems, revealing that the internal mechanisms of these models are more fragile than previously understood. By exposing the specific ways in which attention can be hijacked and demonstrating how attacks can be executed with minimal data, the work underscores the urgent need for robust detection methods and secure training practices. The findings serve as a wake-up call for the development of trustworthy AI, emphasizing that without these safeguards, the very systems designed to assist us could be turned against us by subtle, invisible manipulations.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.