Faculty
Cornelia Fermüller, Shihab Shamma, Irina Muresanu, School of Music (co-PI)Funding Agency
National Science FoundationYear
2023Descriptions
"Emphasizing Explanation in AI Augmented String Instrumental Education" is a three-year, $900K NSF collaborative research grant. ISR-affiliated Research Scientist Cornelia Fermüller (UMIACS) is the principal investigator; Professor Shihab Shamma (ECE/ISR) and Associate Professor Irina Myresanu (School of Music) are the co-PIs.
Music is an integral part of our lives and there is a lot of evidence that giving students opportunities to instrumental music education is important, because of its advantages for developing cognitive, social, and physical skills, and its benefits for mental health. Learning to play a musical instrument, however, constitutes a lengthy and complex process. Private music instruction is costly, afforded only by a privileged part of the world population. The recently developed tools in AI and online education provide an opportunity to create new technology for music education which can reach a very diverse student population. Little is known, however, on how to do intelligent online teaching of musical instruments, and how to keep students engaged through individualized instructions in a discipline that involves motor instructions. In this project, a team with expertise in violin pedagogy and visual and auditory signal analysis aims to develop an AI platform to analyze a student's playing during individual practice time. Video and audio input recorded by cameras and microphones on handheld devices are used to provide feedback on posture, bow movements, sound quality and selection of curricular materials. This project advances our understanding of the learning and teaching of motor tasks, and provides new insights for computational perception of human movements from multi-modal data. It supports digital humanities in the field of music; from a humanity perspective, it democratizes music education, and provides the benefits of instrumental instruction to a large and more diverse population.
Inspired by cognitive studies on educational feedback intervention, this project develops a prototype AI system that acts as a virtual assistance to violin students, teachers, and supervising parents, and like a human teacher it also provides explanations. This is accomplished through two unique components: 1) a feedback system that provides advice based on visual and auditory analysis and causal relationships of errors, i.e., why a movement error caused non-ideal sound; and 2) assignment of educational materials specific to the students based on the perceptual evaluation and a corpus of recorded materials. The technical work involves the collection of multi-modal data from violin players, the development of machine learning algorithms for analyzing the players? performance, the recording and curation of a corpus of educational music pieces from videos and created sheet music, the design of a gamified user interface, and feedback instructions. The combination of developed AI software, data collected through observation of violin students, and the music corpuses to be digitized, recorded, and categorized constitute a major step forward into the 21st century for the field of music pedagogy and innovation of tools for studying human motor learning through perception.
Top