ABB's YuMi robot conducts opera

To mark the First International Festival of Robotics, ABB’s collaborative robot YuMi will be conducting the Lucca Philharmonic Orchestra in Pisa tonight, performing along with soloists Andrea Bocelli and Luigia Borsi. Conductor Andrea Columbini reflects on the process of teaching the robot to conduct an orchestra.

“Conducting is much more than standing up and waving your arms around in front of an orchestra,” said Columbini. “The conductor’s role is to share a vision of a work of music, to set the tone and to shape the diverse voices of the musicians into a single expression in service to the composer. Conducting combines scholarship, technique, interpretation and charisma.

Columbini took an active role in teaching ABB’s YuMi dual-arm collaborative robot to conduct for its debut performance as conductor at the Teatro Verdi in Pisa, Italy, for the gala of the First International Festival of Robotics. YuMi is to accompany Italian tenor Andrea Bocelli in a program of Verdi with the Lucca Philharmonic Orchestra.

“Working to master the nuanced technique of a human orchestra director with YuMi has surely been one of the most satisfying, albeit challenging, tasks of my professional career,” he said. “YuMi’s sophisticated technology excited me and revealed many possible ways to realise art and music through the robot. Setting up the interaction between the elbow, forearm and wrist of the robot, making use of its versatility in repeated and demanding attempts to break down the upbeats and downbeats, was very successful.

“The robot’s performance was developed in two steps,” he explained. “First, my movements were captured with a process called ‘lead-through programming’, where the robot’s two arms are guided to follow my motions with great attention to detail; these movements are then recorded. The second step involved fine-tuning the movements in ABB’s RobotStudio software, where we made sure the motions were synchronised to the music. Of course this took some technical expertise from ABB, but the lead-through programming let me focus on naturally doing what I do best, bringing the music to life. It is amazing to me that this could be accomplished so smoothly.”

Columbini was surprised by the outcome.

“I must say, the final outcome is incredible. The gestural nuances of a conductor have been fully reproduced at a level that was previously unthinkable to me. But we did it: YuMi achieves a very high level of fluidity of gesture, with an incredible softness of touch and expressive nuancing. This is an incredible step forward, given the rigidity of gestures by previous robots,” he said.

But of course, YuMi is not human, so there are some limitations.

“Of course, YuMi is good when it comes to technique but is ultimately not gifted with human sensitivity,” explained Columbini. “The robot uses its arms, but the soul, the spirit, always come from a human. I imagine the robot could serve as an aid, perhaps to execute, in the absence of a conductor, the first rehearsal, before the director steps in to make the adjustments that result in the material and artistic interpretation of a work of music.

“Once we understood each other, YuMi and I became good friends and I am very much looking forward to the performance.”

At the gala event on 12 September, Andrea Bocelli will sing as YuMi directs La Donna è Mobile, the famous aria from Verdi’s Rigoletto. Soloist Maria Luigia Borsi will sing the classic soprano aria O mio babbino caro from Gianni Schicchi by Puccini. YuMi will also conduct a passage from Mascagni’s intermezzo from the opera Cavalleria Rusticana.