NAU-QMUL: Utilizing BERT and CLIP for Multi-modal AI-Generated Image Detection

The NAU-QMUL team proposed a multi-modal multi-task model leveraging pre-trained BERT and CLIP encoders with cross-modal fusion and pseudo-labeling data augmentation to achieve fifth place in both detection and source identification tasks of the CT2 AI-Generated Image Detection competition.

The Gist

Imagine you are a detective trying to solve a mystery in a world where art can be created by both human hands and super-smart robots. Some robots (like Stable Diffusion or DALL-E) are so good at painting pictures from words that it's almost impossible to tell them apart from human art.

This paper is the story of how two researchers, Xiaoyu and Arkaitz, built a digital detective team to catch these AI artists and figure out exactly which robot made the picture.

Here is the breakdown of their solution, explained simply:

1. The Problem: The "Uncanny Valley" of Art

In the past, if you saw a picture of a giraffe, you could tell if a human drew it or if a computer made it. But now, AI can write a story about a giraffe and instantly draw a perfect picture of it.

• Task A (The "Is it Real?" Test): Can you tell if a picture was made by a human or a robot?
• Task B (The "Who Did It?" Test): If it was made by a robot, which specific robot did it? Was it "Midjourney," "DALL-E," or "Stable Diffusion"?

2. The Solution: A Two-Brain Detective Team

The researchers didn't just build one tool; they built a system with two specialized "brains" working together, like a detective duo.

• Brain 1 (The Reader - BERT): This part is an expert at reading text. It looks at the caption (the story) describing the image. It understands the context, the grammar, and the meaning of the words.
• Brain 2 (The Viewer - CLIP): This part is an expert at looking at pictures. It analyzes the pixels, colors, and shapes to see the visual details.

The Magic Trick (Cross-Modal Fusion):
Usually, these two brains work separately. But this system forces them to hold hands. It takes what the Reader understands about the story and compares it with what the Viewer sees in the picture.

• Analogy: Imagine a human artist draws a picture of a "sad clown." The Reader sees the word "sad," and the Viewer sees the frowning face. They match perfectly.
• The AI Tell: Sometimes, an AI might generate a picture of a "sad clown" but the clown has six fingers or the background is weirdly blurry. The Reader says, "The story makes sense," but the Viewer says, "Wait, the details are wrong!" By combining both opinions, the system catches the AI much better than if it just looked at the picture or just read the text.

3. The Training: Learning from Mistakes (Pseudo-Labeling)

Training a detective takes a lot of practice cases. The researchers didn't have enough "known" fake pictures to train their model perfectly. So, they used a clever trick called Pseudo-Labeling.

• The Analogy: Imagine you are teaching a student to spot forgeries. You give them 100 real paintings and 100 fake ones. But you also have a box of 1,000 paintings where you don't know which are fake.
• The Strategy: You let your student guess on the 1,000 unknown paintings. If the student is 90% sure a painting is fake, you say, "Okay, I trust you. Let's treat this as a confirmed fake and add it to the training pile."
• The Result: This gave the model a huge amount of extra practice data, making it much sharper.

4. The Two-Step Judgment (Multi-Task Loss)

The system has to answer two questions at once, but it does them in a smart order:

1. First Question: "Is this AI or Human?" (Yes/No).
2. Second Question: "If it's AI, which one?" (Robot A, Robot B, or Robot C?).

The system is designed so that it only tries to answer the second question if it's already sure the answer to the first question is "Yes." This saves energy and stops the system from getting confused by real human art.

5. The Results: Top 5 in the World

The researchers entered their detective team into a global competition called CT2.

• Task A (Real vs. Fake): They got 5th place in the world. They were right about 83% of the time.
• Task B (Which Robot?): They also got 5th place here. Identifying the specific robot is much harder (like guessing which brand of camera took a photo just by looking at the picture), so getting 48% right is actually a very strong performance.

6. The Catch (Limitations)

The authors are honest about the flaws in their "Pseudo-Labeling" trick.

• The Echo Chamber: If the student detective makes a mistake on a hard case and gets it wrong, but they are "confident" about it, the system adds that wrong answer to the training pile. Now, the system learns the mistake as if it were a fact.
• The Easy Targets: The system only trusts its own guesses when it is very confident. This means it mostly practices on "easy" cases and might still struggle with the really tricky, ambiguous ones.

Summary

In short, these researchers built a super-detective that reads the story and looks at the picture simultaneously. By teaching it to trust its own confident guesses to learn more, they created a system that is currently one of the best in the world at spotting AI-generated art and figuring out exactly which AI made it.

Technical Summary

1. Problem Statement

The rapid advancement of text-to-image generation models (e.g., Stable Diffusion, DALL-E 3, MidJourney) has created a critical challenge: distinguishing AI-generated content from human-created media. This distinction is vital for ensuring media authenticity, protecting intellectual property, and combating misinformation.

The authors address this through the CT2: AI-Generated Image Detection competition, which defines two specific tasks:

• Task A (Binary Classification): Determining whether an image is AI-generated or human-created.
• Task B (Multi-class Classification): Identifying the specific AI model responsible for generating the image (e.g., Stable Diffusion 3, SDXL, DALL-E 3, MidJourney 6).

A key challenge in this domain is the lack of robust, aligned multi-modal datasets and the difficulty of generalizing detection methods across diverse generation models with limited training data.

2. Methodology

The authors propose a Multi-modal Multi-task Model that integrates textual and visual information to perform simultaneous classification.

A. Architecture

The system utilizes two parallel input streams processed by pre-trained encoders:

1. Text Encoder: Uses BERT (bert-base-uncased) to extract contextualized features from image captions. The raw text is tokenized and converted into embeddings.
2. Image Encoder: Uses the CLIP Vision module (openai/clip-vit-base-patch32) to extract high-level visual features. Images are preprocessed (resized, normalized) into pixel values suitable for the encoder.
3. Feature Fusion: The extracted text and image features are concatenated and projected into a shared feature space via a fully connected layer. This enables cross-modal interaction.
4. Classification Heads:
   • Label_A Classifier: A binary classifier for Task A (Real vs. AI).
   • Label_B Classifier: A multi-class classifier for Task B (identifying the specific AI model).

B. Loss Function Optimization

The model employs a Multi-task Loss strategy to optimize both objectives simultaneously:

• Task A Loss: Binary Cross-Entropy Loss applied to all samples.
• Task B Loss: Conditional Cross-Entropy Loss. This is computed only for samples where Task A predicts the image is AI-generated (Label_A == 1). This conditional design prevents noise from real images and improves computational efficiency, as the specific model identity is irrelevant for human-created images.
• Total Loss: The sum of both components, ensuring the model balances both detection and attribution tasks.

C. Data Augmentation (Pseudo-Labeling)

To address data scarcity and improve generalization, the authors implemented a pseudo-labeling strategy:

1. The trained model predicts labels for the unlabeled test dataset.
2. Samples where both Task A and Task B predictions exceed a confidence threshold of 0.8 are selected.
3. These high-confidence samples are assigned pseudo-labels and merged with the original training set (split 8:2 for training/validation) to create an extended dataset for retraining.

3. Key Contributions

• Novel Multi-modal Architecture: Successfully integrates BERT and CLIP encoders to leverage both textual prompts and visual artifacts for detection, moving beyond unimodal approaches.
• Conditional Multi-task Learning: Introduces a tailored loss function where the multi-class attribution task (Task B) is conditional on the binary detection result (Task A), optimizing the learning signal for both tasks.
• Pseudo-labeling Strategy: Demonstrates the effectiveness of using high-confidence predictions from the test set to augment the training data, enhancing model robustness.
• Competitive Performance: Achieved top-tier results in a rigorous competition setting, validating the approach against state-of-the-art baselines.

4. Experimental Results

The model was evaluated on the CT2 competition dataset (53,353 samples derived from MS COCO captions and various generation models).

• Validation Performance:
  • Task A: F1 Score of 99.58% (Accuracy: 99.24%).
  • Task B: Weighted-F1 Score of 85.95% (Accuracy: 90.04%).
• Official Test Set Performance:
  • Task A: Secured 5th place with an F1 score of 83.16%.
  • Task B: Secured 5th place with an F1 score of 48.88%.

Note: The significant drop in Task B performance on the test set compared to validation suggests challenges with domain generalization or distribution shifts in the test data.

5. Significance and Future Work

Significance:
The study demonstrates that combining pre-trained language and vision models via cross-modal fusion is a highly effective strategy for AI-generated content detection. The approach proves that even with limited specific training data, leveraging large-scale pre-trained models (BERT/CLIP) and multi-task learning can yield robust detection capabilities.

Limitations & Future Directions:
The authors acknowledge several limitations, particularly regarding the pseudo-labeling strategy, which introduces risks such as:

• Error Propagation: Incorrect pseudo-labels reinforcing model biases.
• Selection Bias: High-confidence thresholds favoring easy samples while ignoring ambiguous cases.
• Data Distribution Shift: Mismatch between pseudo-labeled test data and original training distributions.
• Class Imbalance: Overrepresentation of specific AI styles that are easier to classify.

Future Work aims to:

• Replace simple feature concatenation with advanced fusion mechanisms (e.g., attention mechanisms, cross-modal transformers, or graph-based methods).
• Model relationships between images generated from the same caption rather than treating them as independent.
• Implement uncertainty-aware training and diverse augmentation techniques to mitigate bias and improve fairness.