Key Takeaways:

• Researchers developed a method for robots to learn tool use by analyzing ordinary two-angle videos.
• The approach focuses on tool behavior rather than mimicking human movement, enabling broader skill transfer.
• The system showed 71% higher success rates and 77% faster data collection compared to traditional training.
• Potential applications range from cooking to construction, with scalability through billions of existing online videos.
• Challenges remain around pose estimation, tool rigidity, and camera view synthesis.

Robotics researchers have long sought efficient ways to teach machines how to interact with the physical world, and a recent breakthrough suggests that the solution might already be in billions of videos captured by humans every day. A team from the University of Illinois Urbana-Champaign, working with Columbia University and the University of Texas at Austin, has unveiled a method that enables robots to learn how to use tools simply by watching ordinary video clips. Their findings, published this month, represent what the group describes as a step toward robots learning from the same observational cues that guide human children.

Traditionally, teaching robots to perform tasks such as hammering a nail or flipping food in a pan has required complex manual demonstrations, expensive sensors, or specialized training environments. The new system sidesteps much of that complexity. Using as little as two smartphone recordings from different angles, researchers feed the footage into a vision model called MASt3R. This model reconstructs a 3D scene using a process known as 3D Gaussian splatting, a technique that builds a detailed digital representation of how the tool interacts with its environment.

To prevent robots from simply mimicking human arm movements, which can be inefficient or impractical for robotic hardware, the team developed a process to digitally remove the human from the recording. By using a tool called Grounded-SAM, the system isolates the tool itself, leaving only its movement and interaction with objects behind. This “tool-centric” view allows robots to concentrate on the physics of the tool rather than the anatomy of the human.

“Instead of teaching robots to copy us, we’re teaching them to understand what the tool is doing,” the researchers explained in their presentation. “That shift enables the knowledge to transfer more broadly, since the robot doesn’t need to replicate our arms or hands—it just needs to replicate what the tool achieves.”

The difference in results is notable. When tested, robots trained with this approach showed a 71% higher success rate than existing methods and collected usable training data 77% faster. In practice, this means a robot could master a simple cooking task, like scooping meatballs from a pan, in far fewer training cycles than would previously have been required. Other examples included hammering, balancing a wine bottle on a tray, or even kicking a soccer ball.

The implications are significant. By focusing on the tool rather than the human, robots can potentially learn a vast array of tasks by analyzing video libraries already in existence. YouTube alone contains millions of hours of footage showing people using tools in kitchens, workshops, and outdoor environments. With the right processing, these videos could serve as training data for robots worldwide, accelerating the pace at which they can be taught new capabilities.

Still, challenges remain. The researchers pointed out that ensuring tools are represented as rigid objects is not always straightforward, especially in low-quality video. Accurately estimating six-degree-of-freedom poses, which describe how an object moves and rotates in space, continues to be a technical hurdle. Additionally, synthesizing realistic new camera views from limited footage is an area where improvements will be necessary to make the system more robust.

Despite those obstacles, the research community took notice. The project received the Best Paper Award at the ICRA 2025 Workshop on Foundation Models and Neural-Symbolic AI for Robotics. Recognition at such a venue suggests growing interest in shifting robotic training away from tightly controlled lab settings and toward real-world observational learning.

The comparison to how children learn is a recurring theme in the team’s description of their work. Human children often master tool use not through direct instruction, but by watching adults perform tasks repeatedly in everyday settings. Applying a similar principle to robotics suggests a way forward that could make training less resource-intensive.

The approach also raises broader questions about scaling. If robots can learn to use a spatula by watching a short cooking video, could they one day learn to use power tools by analyzing construction footage? Could training libraries be built from crowdsourced recordings, where people film themselves performing specific tasks for the purpose of robot education? While the researchers did not suggest immediate commercial applications, they acknowledged that the potential is vast.

For now, the system remains in the research phase. The focus is on refining the accuracy of the models and ensuring that the tool-centric perspective continues to generalize across different types of tasks and physical contexts. However, the trajectory appears clear. By leveraging a massive pool of human-captured videos, robots may move closer to performing diverse real-world tasks without the need for labor-intensive training setups.

As one researcher summarized, “Our work shows that robots don’t need to watch how we move—they need to watch what we do.” That distinction could reshape how robotics evolves, making the machines more adaptable and capable of handling the complexity of human environments.

In the near future, teaching robots might not require coding new instructions or manually guiding their arms. Instead, it could involve uploading a video, letting them watch, and trusting the system to translate tool use into robotic capability. The success of this approach could mark the beginning of a new phase in human-robot collaboration, built not on direct programming but on shared observation.