The Eloquence team submission for task 1 of MLC-SLM challenge

Imagine you are trying to teach a super-smart robot how to listen to a bustling, noisy party where people are speaking different languages, shouting over each other, and telling jokes that only make sense if you remember what was said five minutes ago. This is the world of multilingual conversational speech recognition. For a long time, computers handled this by acting like a relay race: a human (or a machine) would first write down exactly what was said (transcription), and then a second machine would read that text to understand the meaning. But this relay race has a flaw: if the first runner trips and writes down the wrong word, the second runner gets confused, and the whole message gets garbled. Plus, the second runner misses out on the feeling of the voice—the tone, the speed, and the emotion—because they only get the text.

To fix this, scientists are building Speech Language Models (SLMs). Think of these as a single, all-knowing brain that can hear the sound waves directly and understand the meaning at the same time, without needing a middleman to write things down first. It's like upgrading from a robot that needs a human to translate a foreign language before it can understand a joke, to a robot that can hear the joke in the original language and laugh at the right moment. The big question researchers are asking is: How do we make these robots really good at understanding messy, real-life conversations in many different languages at once?

In this paper, a team of researchers from the "Eloquence" project (a group funded by the European Union) decided to throw a party for these AI brains to see which one could listen best. They entered a competition called the MLC-SLM Challenge, which was specifically designed to test how well AI can handle multilingual conversations. Their goal wasn't just to build a robot that can hear; they wanted to build one that understands the context of a conversation, like knowing that "it's cold" might mean "turn on the heater" in one situation, but "I'm wearing a coat" in another.

The team tried three different strategies to see which approach would make their AI the best listener.

First, they checked the rules of the game.
Before trying to invent something new, they looked at the "official baseline"—the standard setup provided by the competition organizers. Think of this as the "starter kit" everyone gets. They tested this kit with different types of "brains" (Large Language Models) and different ways of connecting the ears (audio encoders) to the brain. They found that simply tweaking the connection didn't make a huge difference. It was like trying to fix a slow car by changing the color of the wheels; the engine was the same, so the speed didn't change much. They also tried a fancy new connection method called a "Q-Former," but it didn't magically solve the problem either.

Second, they built their own custom engine.
This was their most successful attempt. They used a framework called SLAM-ASR, which is like a high-tech workshop for building speech robots. They combined a powerful audio listener (Whisper Large V3 Turbo) with a specific type of brain called EuroLLM 1.7B. To make this brain understand the audio, they built a custom "translator" (a linear projector) that converts sound into a language the brain understands.

Here is where they got creative: they tried to teach the robot by giving it "noisy" practice data. They simulated bad audio conditions—like adding static, speeding up voices, or shifting the pitch—to make the robot practice in a tough environment. They hoped this would make the robot tougher, like a boxer training with heavy weights. However, the results were a bit mixed. While the noise training helped in some specific cases, it didn't make their best robot any better than it already was. In fact, their best system, which used the EuroLLM 1.7B brain with a special training technique called LoRA (which is like adding a lightweight, adjustable training vest to the brain rather than rebuilding the whole thing), finished 13th out of 24 teams. It wasn't the winner, but it proved that a smaller, frozen brain with a smart translator could still do a great job.

Third, they tried to give the robot "memory."
Real conversations aren't just one sentence at a time; they are a flow of ideas. If someone says, "It's raining," and then later says, "I forgot my umbrella," the second sentence only makes sense because of the first. The team tried to give their AI this kind of memory by feeding it the previous sentence in the conversation as context. They also tried a technique called contrastive learning, which is like a game of "spot the difference." They showed the AI pairs of sentences that should go together and pairs that shouldn't, forcing the AI to learn the difference between a logical conversation and a random jumble of words.

The results here were interesting. When they added the "memory" (context) and the "spot the difference" game (contrastive learning), the AI got better at understanding the conversation, dropping its error rate to 17.18% on their test set. However, they noticed something strange: when they tried to use the "training vest" (LoRA) along with the memory and contrastive learning, the performance actually got worse. They suspect this is because they didn't give the robot enough time to learn (only two training sessions), so the extra complexity confused it.

What did they learn?
The team discovered that you don't always need the biggest, most expensive brain to understand speech. A smaller, frozen brain (EuroLLM 1.7B) combined with a smart, well-trained translator can be very effective. They also learned that giving the AI a bit of "conversation history" helps it understand the story better, but you have to be careful not to overcomplicate the training. While their system didn't take first place, it showed that there is still a lot of potential in these lightweight, efficient models.

In the end, the Eloquence team suggests that the future of speech recognition lies in combining these smart, efficient architectures with a deeper understanding of how humans actually talk—keeping the flow of the conversation in mind, rather than just listening to isolated words. They plan to keep exploring how to mix languages and handle even more complex tasks in the future, hoping to build robots that don't just hear us, but truly understand us.