Towards Generalizable Face Forgery Detection via Multi-Granularity Fusion
To address the generalization gap in face forgery detection caused by the conflict between transferable high-level semantics and sparse local artifacts, this paper proposes SemFusion, a framework that combines Semantic Consistency Regularization to preserve CLIP's transferable structure with Multi-level Patch Evidence learning to effectively capture and fuse localized forgery cues.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you're scrolling through your phone and you see a video of a celebrity saying something they never actually said. It looks real, sounds real, and feels real. This is the world of "Deepfakes," a digital magic trick where computers can swap faces or reenact expressions so perfectly that our eyes can't tell the difference. For a long time, scientists have been trying to build "digital lie detectors" to spot these fakes. But here's the tricky part: just like a magician who changes their trick every time you catch them, the people making these fakes are constantly inventing new ways to fool the detectors. The old detectors were like security guards who memorized the faces of known thieves; if a new thief walked in wearing a different hat, the guard wouldn't know them. The big question in this field is: How do we build a detector that doesn't just memorize specific tricks, but actually understands what makes a face look "fake" no matter who is making it?
This is where a new study from researchers at Hong Kong Baptist University comes in. They are trying to solve this "generalization" problem—making a detector that works on any new fake, not just the ones it practiced on. To do this, they use a powerful tool called CLIP. Think of CLIP as a super-smart librarian who has read billions of books and seen billions of photos. This librarian knows that a "cat" usually has whiskers and a "beach" usually has sand, even if they've never seen that specific cat or beach before. The researchers wanted to use this librarian's general knowledge to spot fakes, but they realized the librarian was too focused on the big picture (like "this is a face") and missed the tiny, messy details where the magic trick actually happens (like a blurry edge around a swapped mouth).
So, the team built a new system called SemFusion. Instead of just asking the librarian, "Is this a real face or a fake one?" (which is often too vague), they taught the system to look at the whole face and zoom in on suspicious little patches of skin at the same time. They also added a special rule to make sure the system didn't forget the librarian's original, reliable knowledge while it was learning to spot fakes. The result? A detector that is much harder to fool. When they tested it on five different sets of fake videos it had never seen before, it got the right answer 90.9% of the time. When they tested it on six completely new ways of making fakes, it got the right answer 97.4% of the time. It's like upgrading from a security guard who only knows one criminal's face to a detective who can spot any criminal, even if they are wearing a disguise.
The Problem: The "Too Big" Picture
The researchers started by noticing a flaw in how previous detectors worked. Most of them looked at a face as one giant picture. They asked, "Does this whole image look like a real person?" But Deepfakes are sneaky. They often look perfect from a distance, but if you zoom in, you might see a weird smudge where a nose was swapped, or a texture that doesn't match the skin around it.
The paper argues that relying only on the "big picture" is like trying to find a typo in a book by only looking at the cover. You might see the title and the author, but you'll miss the spelling mistake on page 42. Similarly, existing detectors often miss the "local evidence"—those tiny, suspicious spots where the forgery happened. The researchers found that when they used a standard AI model to look at a fake face, it often said, "This looks real!" because the overall face shape was correct, even though the eyes were slightly distorted.
The Solution: A Two-Pronged Detective
To fix this, the team created SemFusion, which stands for "Semantic-Consistent Multi-Granularity Fusion." That's a fancy way of saying they combined two different ways of looking at the problem: the "Global" view and the "Local" view.
1. The Global View (The Big Picture)
First, they used a modified version of the CLIP model to look at the whole face. This part of the system is great at understanding general concepts. It knows what a human face should look like in a broad sense. However, as the researchers noted, this part alone isn't enough because it can be fooled by high-quality fakes that look good from a distance.
2. The Local View (The Zoom-In)
This is where the magic happens. The researchers developed a new technique called Multi-level Patch Evidence (MPE). Imagine taking the face and cutting it into hundreds of tiny puzzle pieces (patches). The system then looks at each piece individually to see if it's acting suspicious.
- It doesn't just look at the top layer of the image; it looks at multiple layers of "depth" in the computer's brain to find clues.
- It acts like a detective scanning a crime scene, ignoring the boring parts (like the background) and focusing only on the "Top-K" most suspicious spots.
- If even a tiny patch around the mouth looks weird, this local branch raises a red flag, even if the rest of the face looks perfect.
3. The Safety Net (Semantic Consistency)
Here is the clever part that keeps the system from going crazy. When you teach an AI to spot fakes, it sometimes gets so obsessed with the "fake" patterns that it forgets what a "real" face actually is. It might start thinking a real face is fake just because it has a shadow that looks slightly different.
To stop this, the researchers added a rule called Semantic Consistency Regularization (SCR). Think of this as a "grounding wire." They kept the original, frozen CLIP text space (the librarian's knowledge) as a reference point. They told the AI: "You can learn to spot fakes, but you must stay close to the original definition of what a face is." This prevents the AI from drifting too far and forgetting the basics. It's like telling a student, "You can invent new ways to solve math problems, but you can't change the fact that 2 + 2 = 4."
How It Works Together
The system works like a team of two detectives.
- Detective Global looks at the whole face and says, "Hmm, this looks mostly real."
- Detective Local zooms in and says, "Wait! I see a weird smudge on the cheek! That's fake!"
- The Fusion: The system combines their opinions. If Detective Global is unsure but Detective Local finds a suspicious patch, the final verdict leans toward "Fake."
The researchers tested this by pitting it against the best existing detectors. They trained the system on one set of fake videos (FF++) and then threw it into the deep end with five completely different sets of videos it had never seen before.
- The Result: SemFusion achieved an average score of 0.909 (out of 1.0) on these new datasets. This was better than the previous best methods.
- The "New Trick" Test: They also tested it on six new methods of making fakes (like new software that swaps faces). SemFusion scored an average of 0.974, crushing the competition.
Why It Matters
The paper suggests that this approach is a big step forward because it doesn't just memorize specific "fake" patterns. Instead, it learns to spot the absence of natural details in specific spots while keeping a solid understanding of what a real face is.
The researchers also checked if their system was tough enough to handle bad video quality, like when a video is blurry, noisy, or compressed. They found that SemFusion stayed reliable even when the video was messy, whereas other detectors started to fail.
However, the authors are careful not to claim they have "solved" the problem forever. They note that as Deepfake technology gets even better, detectors will need to keep evolving. They also point out that while their system is fast and efficient, it still requires a powerful computer to run. But for now, this "two-eyes" approach—one looking at the whole, one looking at the parts, and a rule to keep them honest—offers a much more reliable way to tell truth from fiction in our digital world.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.