Sequential Sampling as a Computational Principle of Episodic Memory

Our memories of specific events—what we call episodic memory—are often described as mental snapshots of the past. We recall where we were, what happened, and who was there. However, the way we experience the world is rarely like a camera capturing a single, perfect image. Instead, our eyes constantly dart around, taking in the world in a rapid series of small, partial glimpses. We look at a face, then a hand, then a background object, stitching these fragments together into a coherent scene. For decades, scientists have wondered how the brain turns this sequence of fleeting glances into a single, stable memory that can be stored and recalled later. The prevailing view has been that the brain first builds a complete picture of an object or scene and then stores that finished picture. But a new line of research suggests the process is far more dynamic: the very act of looking, and the order in which we look, might be what builds the memory in the first place.

A team of researchers has developed a computer model to test this idea, proposing that our memory is not just a recording of what we saw, but a construction built from the path our eyes took to see it. In their work, they simulated a system that learns by looking at images one small piece at a time, just as human eyes do. They found that the sequence of these glances does more than just gather information; it actively shapes the memory itself. When the system focused on a single object for several glances in a row, the memory of that specific object became sharper and more distinct. But when the system's gaze jumped back and forth between two different objects, the memory of the relationship between those two objects became stronger. Crucially, the model showed that the order of the glances mattered. If the same set of visual clues was presented in a different order, the resulting memory was different, and the ability to recall the specific details of the original object was impaired.

To understand how this works, imagine a computer program designed to learn from a series of images, much like a student studying for a test. The researchers gave this program a set of natural photographs, but instead of showing the whole picture at once, they fed it a sequence of small, local patches of the image, one after another. Each patch represented a single "fixation," or a moment where the eyes stop to focus. The program had to integrate these patches over time, building up a mental representation of the whole image as it received more clues. The researchers discovered that with each new patch, the program's uncertainty about what the image was decreased. It could identify the general category of the object, like a "dog" or a "car," very quickly. But to pinpoint the exact identity of that specific dog or car, it needed more time and more samples. As the program accumulated these glimpses, its internal representation of the image became more organized and efficient, focusing on the most important features while still remembering exactly where and when it saw each piece.

The study also explored how we remember connections between things. In real life, we often look back and forth between related items, like a person and the object they are holding. The researchers tested whether the timing of these eye movements mattered. They created scenarios where the program looked at two different images, either by staring at the first one for a while and then the second, or by rapidly switching back and forth between them. They found that the "switching" pattern, where the gaze moved frequently between the two items, made the connection between them much stronger in memory. The program was better at recalling the second item when prompted by the first if the two had been sampled in an interleaved, back-and-forth rhythm. This suggests that the act of moving our eyes between related things is not just a side effect of looking; it is a mechanism that binds those things together in our minds.

Perhaps the most striking finding was that the memory itself contains a record of how it was built. The researchers tested this by taking the exact same set of visual clues and presenting them in a different order. Even though the visual information was identical, the resulting memory was different. When the program tried to recall the original image using a cue that matched the original order of glances, it succeeded. But when the order was scrambled, the memory became fuzzy, and the program struggled to identify the specific image, even though it could still guess the general category. This implies that the "trajectory" of our gaze—the specific path our eyes took—is embedded in the memory. To retrieve a specific memory, our brain may need to partially retrace that original path.

The researchers also simulated what happens when we try to remember something with only a few clues, similar to when we try to recall a face from a blurry photo. They found that the more the retrieval cue matched the original sequence of glances, the better the memory worked. If the cue shared the same visual details and the same order of attention as the original encoding, the system could reconstruct the specific memory with high accuracy. If the order was wrong, or if the visual details were missing, the memory failed. This supports the idea that memory retrieval is not just about matching a picture, but about matching the process of how that picture was originally assembled.

Finally, the team explored how this type of memory might help us reconstruct missing parts of an experience. They trained a separate system to generate images based on the memory representations created by the sequential sampling. They found that the system could reconstruct the parts of the image that had been directly looked at with high precision. However, the parts that were never looked at were filled in more loosely, relying on general patterns the system had learned from other images. This mirrors how human memory works: we remember the details we focused on with clarity, while the rest of the scene is often filled in by our brain using general knowledge and expectations.

These findings offer a new way to think about how we remember. Rather than storing a complete, static image of an event, our brains may store a dynamic record of how we explored that event. The sequence of our eye movements, the order in which we attended to different parts of the world, and the rhythm of our attention all become part of the memory itself. This suggests that the way we look at the world is not just a way of gathering data, but a fundamental part of how we construct our past. By understanding this process, scientists hope to better understand how memories are formed, why some are clearer than others, and how the simple act of looking shapes the stories we tell ourselves about our lives.