Meet Shimon

Shimon - The Perceptual and Improvisational Robotic Marimba Player

Shimon, the robotic marimba player, is designed to create meaningful and inspiring musical interactions with humans, leading to novel musical experiences and outcomes. The robot, which was developed with the support of the National Science Foundation, is designed to listen like a human and improvise like a machine. It combines computational modeling of music perception, interaction, and improvisation with the capacity to produce melodic and acoustic responses in physical and visual manners. The goal of the project is to create real-time musical collaborations between human and robotic musicians that would capitalize on the combination of their unique strengths (human emotions and expressions on one side and processing power and mechanical skills on the other) to produce new and compelling music.

Most computer-supported interactive music systems are hampered by their inanimate nature, which does not provide players and audiences with physical and visual cues that are essential for creating expressive musical interactions. Such systems are also limited by the electronic reproduction and amplification of sound through speakers, which cannot fully capture the richness of acoustic sound. Shimon is designed to overcome these limitations, while also serving as an educational tool, introducing learners not only to music but also to the mathematics, physics, and technology behind it in an interactive, hands-on manner.

Shimon is not your ordinary marimba-playing robot. It uses its melodic and harmonic perception and improvisation modules to create surprising and inspiring musical responses. It does so using rich acoustic sound and communicative social cues with its human counterparts. The robot's head provides visual cues that represent social-musical elements, from beat detection through tonality, to attention and spatial interaction. Just imagine the head bob of a jazz drummer or a DJ spinning a hip hop record, and you have a picture of Shimon's personality.

It is amazing to see Shimon improvise and interact with human musicians. Imagine a pianist playing a musical phrase followed by Shimon, who builds on this input with a new improvised sequence. A fellow guitar player can then enhance Shimon's ideas, leading to new responses that could inspire the humans to play in ways they have never played before. The result is not only novel and expressive human-robotic interaction, but also great new music.