Researchers show robots how to identify an object's properties through handling

US and Canadian researchers have developed a method for robots to guess the weight, softness and other physical properties of an object just by picking it up.

A human clearing junk out of an attic can often guess the contents of a box simply by picking it up and giving it a shake, without the need to see what’s inside. Now, researchers from MIT, Amazon Robotics and the University of British Columbia have taught robots to do something similar.

They developed a technique that enables robots to use only internal sensors to learn about an object’s weight, softness or contents by picking it up and gently shaking it. With their method, which does not require external measurement tools or cameras, the robot can accurately guess parameters like an object’s mass in a matter of seconds.

This low-cost technique could be especially useful in applications where cameras might be less effective, such as sorting objects in a dark basement or clearing rubble inside a building that partially collapsed after an earthquake. Key to their approach is a simulation process that incorporates models of the robot and the object to rapidly identify characteristics of that object as the robot interacts with it.

The researchers’ technique is as good at guessing an object’s mass as some more complex and expensive methods that incorporate computer vision. In addition, their data-efficient approach is robust enough to handle many types of unseen scenarios.

“This idea is general, and I believe we are just scratching the surface of what a robot can learn in this way. My dream would be to have robots go out into the world, touch things and move things in their environments, and figure out the properties of everything they interact with on their own,” said Peter Yichen Chen, an MIT postdoc and lead author of a paper on this technique.

Sensing signals

The researchers’ method leverages proprioception, which is a human or robot’s ability to sense its movement or position in space. For instance, a human who lifts a dumbbell at the gym can sense the weight of that dumbbell in their wrist and bicep, even though they are holding the dumbbell in their hand. In the same way, a robot can ‘feel’ the heaviness of an object through the multiple joints in its arm.

“A human doesn’t have super-accurate measurements of the joint angles in our fingers or the precise amount of torque we are applying to an object, but a robot does. We take advantage of these abilities,” said fellow MIT postdoc and co-author Chao Liu.

As the robot lifts an object, the researchers’ system gathers signals from the robot’s joint encoders. Most robots have joint encoders within the motors that drive their moveable parts, which makes their technique more cost-effective than some approaches because it doesn’t need extra components like tactile sensors or vision-tracking systems.

To estimate an object’s properties during robot–object interactions, their system relies on two models: one that simulates the robot and its motion and one that simulates the dynamics of the object.

“Having an accurate digital twin of the real world is really important for the success of our method,” Chen said.

The researchers’ algorithm ‘watches’ the robot and object move during a physical interaction and uses joint encoder data to work backward and identify the properties of the object. For instance, a heavier object will move more slowly than a light one if the robot applies the same amount of force.

Differentiable simulations

The researchers utilise a technique called differentiable simulation, which allows the algorithm to predict how small changes in an object’s properties, like mass or softness, impact the robot’s ending joint position. The researchers built their simulations using NVIDIA’s Warp library, an open-source developer tool that supports differentiable simulations.

Once the differentiable simulation matches up with the robot’s real movements, the system has identified the correct property. The algorithm can do this in a matter of seconds and only needs to see one real-world trajectory of the robot in motion to perform the calculations.

“Technically, as long as you know the model of the object and how the robot can apply force to that object, you should be able to figure out the parameter you want to identify,” Liu said.

The researchers used their method to learn the mass and softness of an object, but their technique could also determine properties like moment of inertia or the viscosity of a fluid inside a container.

Plus, because their algorithm does not need an extensive dataset for training like some methods that rely on computer vision or external sensors, it would not be as susceptible to failure when faced with unseen environments or new objects. In the future, the researchers want to try combining their method with computer vision to create a multimodal sensing technique that is even more powerful.

“This work is not trying to replace computer vision. Both methods have their pros and cons. But here we have shown that without a camera we can already figure out some of these properties,” Chen said.

Image credit: iStock.com/onurdongel