The cortico-habenular axis reactivity and anxiety behavior in male and female rats with dopamine or serotonin transporter knockout

Inside the brain, two chemical messengers called dopamine and serotonin act as the primary regulators of movement, mood, and how an animal reacts to new or scary situations. These chemicals float between nerve cells to carry signals, but their levels must be kept in a precise balance. Specialized proteins, known as transporters, act like vacuum cleaners that sweep these chemicals up after they have done their job, ensuring the signal stops at the right moment. When these transporters are missing, the chemicals linger in the spaces between cells, creating a state of constant, unregulated signaling. Scientists have long known that removing the transporter for dopamine leads to extreme hyperactivity, while removing the one for serotonin often results in heightened anxiety. However, the exact way these chemical imbalances change the wiring of the brain, and whether these changes affect males and females differently, has remained a mystery. Understanding these mechanisms is crucial because anxiety and movement disorders are common human conditions, and knowing how the brain's internal circuits fail can help explain why treatments work for some people but not others.

A team of researchers set out to map these hidden circuits by studying rats that had been bred without either the dopamine transporter or the serotonin transporter. They focused on a small, almond-shaped structure deep in the brain called the lateral habenula. This area acts as a sort of emotional switchboard, receiving information about bad outcomes or missed rewards and then signaling other parts of the brain to adjust behavior. The scientists wanted to see how this switchboard reacted when the prefrontal cortex—the part of the brain responsible for planning and decision-making—sent it a signal. To do this, they tested both male and female rats in a simple open arena to measure how much they explored and how anxious they felt. They then used tiny electrical probes to listen to the individual nerve cells in the habenula while stimulating the prefrontal cortex, allowing them to watch in real time how the brain's electrical activity changed in response to the missing transporters.

The results painted a clear picture of how the absence of these transporters reshapes behavior and brain function, but with a surprising twist: the effects were not the same for males and females. The rats missing the dopamine transporter moved with frantic energy, covering much greater distances than normal rats. This hyperactivity was especially intense in the females, who ran significantly farther than the males. When placed in the center of the open arena, a place that usually feels exposed and frightening, these dopamine-deficient rats avoided the middle, spending less time there and entering it fewer times. This behavior suggests a higher level of anxiety, yet it was complicated by a strange, repetitive pattern of running along the walls, which may have made them avoid the center simply because they were stuck in a loop of movement. In contrast, the rats missing the serotonin transporter moved less than normal rats and also showed signs of anxiety, but this was particularly pronounced in the females, who avoided the center of the arena far more than their male counterparts. This finding of strong, sex-specific anxiety in female rats lacking serotonin transporters was a new discovery, as previous studies had not highlighted such a clear difference between the sexes.

When the researchers turned their attention to the electrical activity of the habenula, they found that the brain's response to stimulation depended heavily on which transporter was missing and whether the rat was male or female. In normal rats, stimulating the prefrontal cortex caused about seventy percent of the habenula's nerve cells to quiet down, while the remaining thirty percent became more active. This mix of responses is thought to come from two different pathways: one that directly excites the cells and another that inhibits them through a relay station in the brain. In the rats missing the dopamine transporter, this balance shifted dramatically, but only in the females. In female rats without the dopamine transporter, nearly every single habenula neuron stopped firing when the prefrontal cortex was stimulated, whereas the males showed a mix of responses similar to normal rats. The rats missing the serotonin transporter, however, showed no change in how their habenula neurons responded to stimulation; their electrical patterns looked just like those of the normal rats.

Perhaps the most revealing discovery was how the brain's electrical activity connected to the animal's feelings of anxiety. In normal female rats, there was a clear link: those with higher baseline firing rates in their habenula neurons spent more time in the center of the arena and seemed less anxious. This relationship suggested that a certain level of electrical activity in this brain region helps keep anxiety in check. However, this link vanished completely in the rats with missing transporters. Whether they were male or female, and whether they lacked dopamine or serotonin transporters, the connection between how fast their neurons fired and how anxious they acted was broken. The researchers concluded that while the transporters are essential for normal behavior, their absence disrupts the very circuit that usually ties brain activity to emotional state. This suggests that in these conditions, the brain's ability to translate its own electrical signals into a sense of safety or danger is fundamentally altered, and that this alteration plays out differently depending on the sex of the animal.