💻 computer science

FARM: Find Anything using Relational Spatial Memory

FARM is a real-time relational spatial memory system that enables robots to accurately locate specific object instances in complex environments by combining open-vocabulary object-level memory with visual-language models to parse and ground user queries based on semantic, appearance, and spatial relationships, significantly outperforming prior methods in retrieval accuracy.

Original authors: Siming He, Leo Huang, Adam Lilja, Fabio Hubel, Jonas Frey, Marco Pavone, S. Shankar Sastry, Jitendra Malik, Claire Tomlin

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Siming He, Leo Huang, Adam Lilja, Fabio Hubel, Jonas Frey, Marco Pavone, S. Shankar Sastry, Jitendra Malik, Claire Tomlin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you walk into a massive, cluttered warehouse or a multi-story house filled with hundreds of identical-looking items. You ask a robot, "Find the tall lamp that's sitting under the dartboard and to the left of the poster."

If the robot only had a simple list of "things I saw," it would be lost. It might see 42 lamps and pick the wrong one. It needs to understand the relationships between objects, not just the objects themselves.

This paper introduces FARM (Find Anything using Relational Spatial Memory), a new way for robots to remember and find things in huge, complex environments. Here is how it works, broken down into simple concepts:

1. The Problem: The "Needle in a Haystack"

Most robots today are like librarians who only know the book titles. If you ask for "the red book," they might grab any red book. But in a real house or construction site, there are thousands of red books, blue chairs, and white trash cans.

Users don't just say "Find the lamp." They say, "Find the lamp next to the sofa, under the painting, and farther away than the TV." To do this, a robot needs a Relational Spatial Memory. It needs to remember not just what things are, but where they are relative to each other.

2. The Solution: FARM's "Smart Filing System"

FARM acts like a super-organized filing system that builds itself in real-time as the robot moves around.

The "Gaussian" Filing Card: Instead of storing a heavy, detailed 3D model of every object (which would take up too much memory), FARM stores each object as a single, compact "cloud" (called a 3D Gaussian). Think of this like a fuzzy, glowing balloon that represents the object's size and location. It's lightweight and updates instantly.
The "Asynchronous" Librarian: While the robot is busy moving and scanning (the "critical path"), a separate, background team of AI workers (Visual Language Models) is busy writing detailed descriptions and taking notes on what those objects look like. This happens in the background so the robot never has to stop moving to think.
The "Symbolic" Detective: When you ask a question, FARM doesn't just guess. It uses an AI to translate your sentence into a logical puzzle.
- Your Query: "Find the blue toilet near the white van."
- FARM's Logic: "Okay, I need to find a 'blue toilet' (Target). I need to find a 'white van' (Anchor). Then I check: Is the blue toilet near the white van?"

3. How It Solves the Puzzle

When the robot gets a query, it doesn't try to read your mind or look through thousands of video frames at once (which is slow and confusing). Instead, it follows a strict, three-step process:

Parse: It breaks your sentence into a map of "Target" and "Anchors" (the landmarks you mentioned).
Score: It quickly checks its memory. "Okay, I have 5 blue toilets. Which one is closest to a white van?" It uses math to score how well each candidate fits the description.
Verify: If there are still two very similar candidates, it pulls up a few specific photos of those spots and asks a powerful AI, "Which one of these two actually looks like the one in the photo?" This final check ensures high accuracy.

4. The Results: Speed and Accuracy

The paper tested FARM in two very different worlds:

Indoors: A multi-story house with 42 similar lamps.
Outdoors: A massive construction site (15,000 square meters) with many portable toilets and delivery vans.

The findings were impressive:

Speed: FARM builds its memory while the robot moves, at a rate of 5 to 10 times per second. It's fast enough for real-time use.
Accuracy: It found the correct object much more often than previous methods. In the construction site test, it was able to pick the right toilet out of three identical ones by using the nearby van as a clue.
Efficiency: It uses very little computer memory (about the size of a few photos) compared to other systems that try to store massive 3D maps.

5. Real-World Test

The team didn't just test this on a computer. They put FARM on a Boston Dynamics Spot robot (a four-legged dog-like robot). They drove the robot around a real environment, built the memory on the robot's own computer, and then asked it to find specific items based on complex instructions. The robot successfully navigated to the correct objects, proving that this "filing system" works in the real, messy world.

Summary

FARM is like giving a robot a smart, relational map instead of just a list of things. It allows the robot to understand that "the lamp under the dartboard" is different from "the lamp on the table," even if they look exactly the same. By combining fast 3D mapping with smart language understanding, it can find specific items in huge, cluttered spaces instantly and accurately.

Technical Summary: FARM (Find Anything using Relational Spatial Memory)

Problem Statement

Robots operating in large-scale, object-rich environments (homes, warehouses, construction sites) require persistent memory systems capable of retrieving specific object instances on demand. Existing approaches face two primary limitations:

Insufficient Memory Representation: Closed-vocabulary semantic SLAM and 3D scene graphs fail to generalize to the open vocabularies users naturally employ. While recent open-vocabulary methods exist, they often require offline post-processing, hyperparameter tuning across scales, or rely on brittle end-to-end reasoning over scene graphs.
Ineffective Relational Retrieval: Users typically specify targets through compositional relations to landmarks (e.g., "the tall lamp below the dartboard and to the left of the poster"). Pure embedding-based retrieval (e.g., CLIP) fails to capture these complex relational structures. Conversely, end-to-end Visual Language Model (VLM) reasoning over raw frame histories or scene graphs is often brittle on nontrivial spatial relations, especially for models deployable on robots, and suffers from limited context windows in large-scale environments.

The paper posits that successful retrieval demands a relational spatial memory: a 3D memory treating queries not as simple label lookups, but as relational specifications identifying targets through semantic, appearance, and spatial relations among multiple objects.

Methodology

FARM addresses these challenges through a two-stage pipeline: online memory construction and robust relational object retrieval.

1. Online Memory Construction

FARM builds a compact, open-vocabulary, object-level memory in real-time (5–10 Hz) without offline post-processing.

Representation: Each object is represented as a single 3D Gaussian ( $\mathcal{N}(\mu, \Sigma)$ ) summarizing location and spatial extent. This choice avoids global resolution hyperparameters required by voxels or point clouds, allowing the system to scale from small indoor rooms to large outdoor sites (up to 15,000 m²) under a fixed configuration.
Attributes: Each entity stores:
- 3D Gaussian geometry.
- Detection-time appearance features for cross-view association.
- Up to $k$ representative views (RGB crops) selected for viewpoint diversity.
- Open-vocabulary captions generated by VLMs.
- Three retrieval embeddings: text embedding of the caption, SigLIP2 image embedding, and Qwen3-VL image embedding.
Relations: The system stores low-order relations (covisibility and adjacency via Hellinger distance) rather than high-order predicates (e.g., "left-of") during mapping. This prevents exponential growth in relation complexity; specific relations are recovered at query time.
Pipeline: The system runs a synchronous detect–lift–associate–fuse loop on the GPU. Visual-language captioning and embedding occur asynchronously off the critical path using VLM workers, ensuring the mapping frame rate remains stable regardless of VLM latency.

2. Robust Relational Object Retrieval

Retrieval is decoupled into three stages to balance efficiency and accuracy:

Executable Query Specification: A parser (Qwen3.5-9B) compiles natural language queries into a typed query graph $\Pi(q) = (x^\star, a_1, \dots, a_m, S_q, \Phi_q)$ . This defines a target variable $x^\star$ , anchor variables $a_i$ , open-vocabulary descriptions $S_q$ , and spatial/semantic predicates $\Phi_q$ .
Soft Scoring and Binding:
- Unary Scores: Candidates are scored based on the fusion of text, SigLIP2, and Qwen3-VL embeddings.
- Relational Scores: Predicates are evaluated as soft functions over the memory's 3D Gaussians (e.g., Hellinger distance for "Near," vertical offset sigmoid for "Above/Below," virtual-viewer frames for "LeftOf/RightOf").
- Binding: The system greedily binds anchors to the best-scoring entities for each target candidate, exploiting the star-shaped structure of typical referring expressions to avoid exhaustive enumeration.
VLM Reranking: The top-5 candidates are reranked by a VLM (Qwen3.5-9B) which inspects projected views of the candidate and its matched anchors. This stage resolves residual ambiguity that geometric predicates alone cannot.

Key Contributions

Real-Time Scene Graph Algorithm: A scalable algorithm that constructs open-vocabulary spatial memory online at robot scale (5–10 Hz) using a fixed hyperparameter set across diverse indoor and outdoor environments, requiring no offline post-processing.
Relational Retrieval Framework: A retrieval system that outperforms prior methods in both top-1 accuracy and top-K recall by combining structured symbolic parsing with soft predicate evaluation and VLM reranking.
FARM-Scenes Benchmark: A new benchmark comprising seven large-scale indoor and outdoor scenes (ranging from 1,800 to 15,000 m²) curated to stress-test scalability and relational grounding in real-world, cluttered environments.

Experimental Results

The authors evaluated FARM on 67 scenes (ScanNet, HM3D, and FARM-Scenes) with 44,000 language queries.

Performance Gains: Compared to prior methods (BBQ, RynnBrain, DAAAM), FARM improved:
- Recall@5 by 164% and Recall@10 by 224% over BBQ.
- Accuracy@1 by 35% over the stronger VLM-reasoning baseline (RynnBrain or DAAAM+R).
- On the HM3D-30 benchmark, FARM improved Accuracy@1 by 30% relative to DAAAM (7.9% vs. 6.1%).
Efficiency: FARM maintains mapping throughput of ~~8 Hz. The memory footprint is compact (~~23 MiB for ScanNet, ~125 MiB for HM3D), suitable for on-device deployment.
Real-World Deployment: The system was successfully deployed on a Boston Dynamics Spot robot with onboard NVIDIA Jetson Thor compute, demonstrating closed-loop memory construction and relational retrieval in real-time without offline preprocessing.

Significance and Claims

The paper claims that FARM establishes a new state-of-the-art for relational object grounding in large-scale environments. Its significance lies in:

Bridging the Gap: It successfully bridges the gap between the perceptual strength of modern VLMs and the need for structured, persistent spatial memory, avoiding the context window limitations of pure video-based reasoning.
Scalability: By using 3D Gaussians and deferring high-order relation construction to query time, FARM achieves scalability across vastly different scene sizes (from 15 m² to 15,000 m²) without re-tuning.
Robustness: The structured use of VLMs (parsing and reranking) rather than end-to-end reasoning over graphs yields more accurate and robust retrieval, particularly for fine-grained spatial disambiguation.
Practicality: The system operates in real-time on onboard hardware, enabling robots to build and query their own memory during operation.

The authors acknowledge limitations, noting that spatial predicates are manually specified and not calibrated, and that current benchmarks primarily evaluate relations between the target and anchors rather than complex reasoning among multiple anchors. They suggest future work in learning calibrated predicate functions and extending benchmarks to support compositional anchor reasoning.