Retriever: Composing Closed-Loop Asynchronous Robot Programs

Contributions

Our work develops four connected pieces: the continuous-time formulation behind the graph, the Python programming model used to author it, the runtime that executes it, and Retriever-0, a real-robot pipeline that puts the full stack together.

This page follows those pieces from behavior to implementation. The demonstrations show what the agent does; the pipeline and code show how it is composed; the interactive timeline shows how one graph unfolds across mismatched clocks.

Why a common programming layer?

Robotics is unlikely to settle on one planner, policy library, middleware, or hardware stack. Different frameworks can still converge on a shared graph vocabulary for modules, topology, clocks, and data handoff. That makes systems easier to compare, modules easier to replace, and execution traces easier to inspect. It also gives coding agents a concrete program to edit and debug.

The need grows rather than shrinks as embodied reasoning models improve. The current generation is explicitly designed to orchestrate lower-level policies as tools, track progress toward completion, and reason about the next step while the robot is still moving. Those are coordination properties, and they have to hold in a real program: something must decide when the reasoner is called, which observations it sees, whether its last plan is still valid, and what the controller does in the meantime. A stronger reasoner with variable latency widens the rate gap it has to be coordinated across. Retriever is the layer where those decisions are written down and checked, so a better model can be dropped in as one Flow rather than rebuilt into the glue.

What the agent can do

Retriever-0 takes goals in natural language, plans with a VLM, keeps a belief of the scene across episodes, and executes learned bimanual skills in one closed-loop program. In Episode 1, the agent searches four drawers for black pepper. In Episode 2, it remembers the top-right drawer and goes there directly. The bag task exercises bimanual retrieval and distribution with a different composition of Flows.

Concretely, the running agent does five things at once, and each is a declared part of the program rather than a behaviour of the scheduler:

• Plans over a language goal and revises the plan when the belief changes, instead of executing a fixed script.
• Treats skills as typed interfaces. The planner emits a bounded skill program; the monitor resolves it into one concrete SkillCmd for the low-level policy.
• Tracks its own progress. A progress prediction from the skill policy tells the execution monitor when a skill is effectively finished, so handoff happens at a decided moment rather than on a timeout.
• Thinks while it moves. Plan chunking lets the VLM propose the next bounded step while the current skill is still executing, so slow reasoning never stalls the 200 Hz control loop.
• Keeps belief across episodes, which is why the second run of the drawer task is short.

Take out any one of these and the task degrades in a specific way: without belief the agent re-searches drawers forever, without replanning it stalls once the plan stops matching what it knows.

The problem

Control may tick every few milliseconds while a VLM takes several seconds. Between them, perception, memory, planning, and learned skills update as the world changes. The behavior depends on which data crosses each rate boundary and when. Most robot stacks spread those decisions across callbacks, queues, and scheduler behavior. Retriever makes them part of the program.

Compose temporal modules

A Retriever agent is a graph of reusable temporal modules. A Flow owns local state, a Clock decides when it runs, and a Sync policy on each edge selects which upstream history it consumes. These declarations keep a slow planner, a learned skill, and a fast controller independent without pushing timing logic into callbacks.

Retriever-0: a concrete closed-loop agent

Retriever-0 is a demonstration agent for long-horizon manipulation under partial observability. It closes the perception–reasoning–action loop by combining camera observations, belief memory, VLM planning, VLA skills, execution monitoring, and high-rate control. As the task unfolds, the agent jointly updates its memory, plan, and action without interrupting execution.

Read the graph through three elements:

1. Blocks — what computes. Each Flow is a stateful module with one defined role, such as perception, memory, planning, skill execution, or control.
2. Clocks — when it runs. The badge at the upper-right of each block declares its clock. A Flow emits only when that clock fires, whether periodically or on a trigger. Before it runs, its incoming histories are synchronized to that clock; its outputs then follow the same clock. In the animation, one source Flow keeps one color, while periodic clocks have one motion frequency.
3. Edges — what connects. Each directed edge identifies the upstream and downstream Flows, its label names the information being exchanged, and its synchronization policy determines which upstream history is consumed.

Together, the blocks, clocks, and labeled edges form a complete, decomposable description of the agent’s temporal computation graph.

Closed-loop by design

Feedback is part of Retriever-0’s graph. New observations change the belief used by planning. The execution monitor reads plan chunks and skill progress to decide when to replan or switch skills, while the controller keeps running.

In Python, these loops are ordinary Pipeline edges declared with the same Flow, Clock, and Sync vocabulary.

What a Retriever program looks like

The same graph fits in a short Python program. Flows define computation, clocks define run conditions, sync policies define what crosses each edge, and a Pipeline composes the pieces.

# 1) Define Flows (what computes) and clocks (when they run)
top_cam   = CameraSource(id=0)       @Rate(hz=30)
wrist_cam = CameraSource(id=1)       @Rate(hz=30)
belief    = BeliefMemoryFlow()       @Trigger("inspection_done")
monitor   = ExecutionMonitorFlow()   @Trigger("belief_updated", "progress_prediction")
planner   = VLMPlanFlow("gemini")    @Trigger("replan")
vla       = VLASkillFlow("pi05")     @Rate(hz=2)
robot     = ControllerFlow(id=0)     @Rate(hz=200)

# 2) Compose the graph and declare sync policies on edges
pipe = Pipeline("Closed-loop Agent")
with pipe:
    wrist_cam.then(vla, sync=Latest()) \
             .then(robot, sync=Chunking())

    top_cam.then(belief, sync=Latest()) \
           .then(planner, sync=Latest()) \
           .then(monitor, sync=Latest()) \
           .then(vla, sync=Latest())

# 3) Debug locally or deploy asynchronously
pipe.step(dt=0.1)
pipe.run(backend="dora")

The graph, code, and timeline are three views of the same temporal contract. Retriever can step the graph locally or run it asynchronously without changing what the program means.

Deterministic replay

Each Flow samples inputs by timestamp. Given the same recorded history and a fixed order for equal timestamps, replay produces the same behavior regardless of live scheduling. The trace shows exactly what each Flow consumed, so a run can be inspected during debugging or reused as training data.

The technical blog covers the full formulation, synchronization semantics, runtime mapping, and proof sketch.

Acknowledgments

We thank Sebastian Castro, Toby Huang, Siyuan Huang, Haoyan Lin, Shoukang Yu, and Liyun Zhang for helpful technical discussions and contributions to implementation and experimentation. We thank Hao Zhou and Jason Wu for their support. Also, we thank Will Shen, Wenlong Huang, Leslie Kaelbling, Jeannette Bohg, and other colleagues for helpful discussions and feedback.