Retriever
Search, inspect,
then season.
The spice can be in any closed drawer. The robot inspects drawers, updates its belief after each look, then picks the spice, seasons the food, and puts it back.
We study long-horizon manipulation under partial observability, such as finding a spice hidden in one of several closed drawers or sorting the unseen contents of a bag. One model alone is not a practical way to solve these tasks. Robots instead run hierarchical closed-loop pipelines that combine perception, memory, planning, learned skills, control, and more.
Retriever is a programming model and runtime for composing such pipelines. Each module is a Flow: a stateful computation with a Clock that says when it runs and Sync policies that say which inputs it reads. Flows compose into one graph that you can step locally to debug, or run asynchronously on the robot with modules at different rates. With inputs, clock decisions, and random draws logged, a run can be replayed exactly. Retriever-0, the agent in the videos below, is one such pipeline written as a single graph.
Contributions
Retriever-0 carries tasks across skills.
Search, inspect,
Retrieve from
a deformable bag.
The bag hides what is inside. The robot lifts the bag, finds the items, and sorts them onto plates in one closed-loop run.
Each component
has a role.
Plan chunking, progress prediction, belief memory, and replanning each support the closed loop; the blog reports what happens when each is removed.
The closed-loop pipeline carries both tasks across many skills. See the reported results in the blog.
Retriever-0 takes goals in natural language, plans with a VLM, keeps a language memory of what it has observed, and executes learned bimanual skills in one closed-loop program. In Episode 1, the agent searches the drawers for black pepper. In Episode 2, it opens the previously used top-right drawer first. The bag task exercises bimanual retrieval and distribution with a different composition of Flows.
- Episode 1 Search the drawers, update memory after each inspection, find the pepper in the top-right drawer, and season the steak.
- Episode 2 Open the previously used top-right drawer first, retrieve the pepper, and season the steak.
Concretely, the running agent does five things at once, and each is a declared part of the program rather than a behaviour of the scheduler:
- Plans over a language goal and revises the plan when the belief changes, instead of executing a fixed script.
- Treats skills as typed interfaces. The planner emits a bounded skill program; the monitor resolves it into one concrete
SkillCmdfor the low-level policy. - Tracks its own progress. A progress prediction from the skill policy tells the execution monitor when a skill is effectively finished, so handoff happens at a decided moment rather than on a timeout.
- Thinks while it moves. Plan chunking lets the VLM propose the next bounded step while the current skill is still executing, so skills keep running while slow models think.
- Remembers what it has found. Memory holds what the closed drawers hide: a natural-language list of facts that are known or not yet observed, such as “The target spice was not found in the top-right drawer”, updated from new observations after each inspection.
These behaviors belong to Retriever-0’s specific models and graph. The videos illustrate the behavior; Table III reports the component ablations.
Try the drawer search in simulation
Open the playground at retriever.spaceThe playground below replays a recorded MuJoCo simulation of the drawer search with a Panda arm. Press Run, change the speed, and switch between a fresh search and a search that uses the remembered drawer.
Retriever-0 is one closed-loop graph.
Pipeline graph
Retriever-0 is a demonstration agent for long-horizon manipulation under partial observability. It closes the perception–reasoning–action loop by combining camera observations, belief memory, VLM planning, VLA skills, execution monitoring, and high-rate control. As the task unfolds, the agent jointly updates its memory, plan, and action without interrupting execution.
Retriever-0 closes the loop across perception, belief memory, planning, skill execution, and control.
Read the graph through Flow, Clock, and Sync:
- Flow — what it computes. Each Flow is a stateful module with one defined role, such as perception, memory, planning, skill execution, or control.
- Clock — when it runs. The badge at the upper-right of each block declares its clock. A clock tick starts a Flow step, periodically or on a trigger. The step reads a synchronized input snapshot; its output becomes visible when committed, which can be later. The animation is schematic, not a measured execution trace.
- Sync — which inputs each run consumes. Each directed edge identifies the upstream and downstream Flows and its label names the information being exchanged; its synchronization policy decides which upstream history each run consumes.
Feedback loops
Two feedback loops react to what actually happened: inspection → memory → new plan, and progress → monitor → next skill. New observations change the belief used by planning. The execution monitor reads plan chunks and skill progress to decide when to replan or switch skills, while the controller keeps running.
In Python, these loops are ordinary Pipeline edges declared with the same Flow, Clock, and Sync vocabulary.
The graph is a Python program.
The same graph fits in a short Python program. Flows define computation, clocks define run conditions, sync policies define which inputs each run consumes, and a Pipeline composes the pieces.
Program surface
Each task class implements the core Flow API. The surrounding program assigns clocks, wires edges, and chooses sync policies. step() and run() execute the same graph.
# 1) Define Flows (what computes) and clocks (when they run)top_cam = CameraSource(id=0) @Rate(hz=30)wrist_cam = CameraSource(id=1) @Rate(hz=30)belief = BeliefMemoryFlow() @Trigger("inspection_done")monitor = ExecutionMonitorFlow() @Trigger("belief_updated", "progress_prediction")planner = VLMPlanFlow("gemini") @Trigger("replan")vla = VLASkillFlow("pi05") @Rate(hz=2)robot = ControllerFlow(id=0) @Rate(hz=200)
# 2) Compose the graph and declare sync policies on edgespipe = Pipeline("Closed-loop Agent")with pipe: wrist_cam.then(vla, sync=Latest()) \ .then(robot, sync=Chunking())
top_cam.then(belief, sync=Latest()) \ .then(planner, sync=Latest()) \ .then(monitor, sync=Latest()) \ .then(vla, sync=Latest())
# 3) Debug locally or deploy asynchronouslypipe.step(dt=0.1)pipe.run(backend="dora")Execution timeline
Drag across the timeline to follow 15 seconds of execution. Each row is a Flow running on its own clock. Arrows show time spent computing, and blocks show outputs that remain valid until a replacement arrives.
t = 1.2s · high-rate control keeps flowing between slow policy calls
camera flow
Rate(30Hz), ticks thinned
belief/memory flow
update belief t = 5sExecuting an information-gathering action produces new evidence, so memory updates before planning continues. update belief t = 10sProgress prediction marks the current skill complete, so the monitor can switch skills.
planner
compute plan t = 2sThe planner proposes a plan chunk that can be consumed while execution continues. replan t = 7sA progress prediction opens a revision step, so the planner reads current belief again.
VLA skill policy
infer action t = 4sThe VLA skill emits a time-extended action chunk plus progress prediction. t = 6sThe skill policy refreshes commands using the latest synchronized inputs. t = 8sThe skill policy continues refreshing short action chunks while control runs. t = 10sProgress prediction selects the next policy step without stopping control. t = 12sThe final visible action chunk carries the controller toward the end of this window.
controller flow
Dots mark Flow starts. Dashed links connect causes to effects. They do not mark regular global ticks. Timings are schematic, not measured.
- Different clocks. Camera, memory, planning, skill inference, and control do not run at the same rate.
- Reusable outputs. Memory, plan chunks, and action chunks stay valid until a newer block replaces them.
- Continuous control. Feedback can update memory, replan, or switch skills while the controller keeps consuming overlapping action chunks.
clocked procedure start computation time trigger / switching edge plan / action chunk high-rate control tick
The graph, code, and timeline are three views of the same temporal contract. Retriever can step the graph locally or run it asynchronously without changing what the program means.
Deterministic replay
Each Flow selects inputs by timestamp. Deterministic replay requires logs of consumed inputs, clock and commit traces, and stochastic outputs, with a fixed order for equal timestamps. Re-executed computations must be deterministic at the replay boundary; stochastic or external results must be restored from the log. Under these conditions, replay preserves the recorded output trace independently of scheduler interleavings. The trace exposes what each Flow consumed for debugging and inspection.
The technical blog covers the full formulation, synchronization semantics, runtime mapping, and proof sketch.
Common questions
Why a common programming layer?
Robotics is unlikely to settle on one planner, policy library, middleware, or hardware stack. A shared graph vocabulary for modules, clocks, and data handoff makes systems easier to compare, modules easier to replace, and traces easier to inspect. As reasoning models get stronger, something must still decide when they are called, which observations they see, and what the controller does meanwhile; Retriever is where those decisions are written down, so a better model can drop in as one Flow.
What does Retriever add beyond ROS 2 callbacks?
ROS 2 callbacks can implement this behavior. Retriever makes local state, computation, activation clocks, and input-selection policies explicit in one graph. In the drawer example, an inspection that does not find the spice updates memory so the search continues after the drawer leaves view.
What happens when a result arrives late?
A completed result becomes available to downstream Flows, whose Sync policies select inputs. That alone does not establish whether an old plan is still valid after belief changes. Invalidating queued actions, cancelling running computation, and physically stopping the robot need separate application and hardware rules.
What does the monitor read and produce?
In Retriever-0, the monitor reads plan chunks, belief updates, and skill-progress predictions. It selects the active skill command, resolves conditional branches, and emits replanning and inspection-complete events. A Flow is a stateful computation module, not flow matching.
Could a stronger model with memory and tools solve the task?
Yes, a capable model with updatable memory and tools could solve the task. Retriever exposes how computations, memory, and timing are composed; it does not rule out that alternative. We do not claim superiority over a matched model-with-memory-and-tools baseline.
Cite this work
Linfeng Zhao, Haojie Huang, Jiayuan Mao, Weiyu Liu, Lawson L. S. Wong, and Mykel J. Kochenderfer. Retriever: Composing the Perception-Reasoning-Action Loop for Long-Horizon Manipulation. arXiv:2607.17213, 2026.
Download BibTeX · Paper (arXiv) · PDF · Permanent citation link
@misc{zhao2026retriever, title = {Retriever: Composing the Perception-Reasoning-Action Loop for Long-Horizon Manipulation}, author = {Linfeng Zhao and Haojie Huang and Jiayuan Mao and Weiyu Liu and Lawson L. S. Wong and Mykel J. Kochenderfer}, year = {2026}, eprint = {2607.17213}, archivePrefix = {arXiv}, primaryClass = {cs.RO}, url = {https://arxiv.org/abs/2607.17213}}Acknowledgments
We thank Sebastian Castro, Toby Huang, Siyuan Huang, Haoyan Lin, Shoukang Yu, and Liyun Zhang for helpful technical discussions and contributions to implementation and experimentation. We thank Hao Zhou and Jason Wu for their support. Also, we thank Will Shen, Wenlong Huang, Leslie Kaelbling, Jeannette Bohg, and other colleagues for helpful discussions and feedback.