Exploration and training of music performance skills by means of adaptive monitoring and visual feedback (NNN.6301)

Research
Learning to perform music is about much more than just 'playing the right
notes'. After acquiring this basic skill, in many subsequent years of intensive teaching and practice the complex skill of musical expression and style is developed. These important aspects of music performance are not prescribed in the score and have to be decided on by the musician. But even more important than choosing a musical interpretation is the amount of motor control needed for achieving the chosen expressive timing, dynamics, intonation, articulation, etc. These motor skills are hard to learn. For example, it is not unusual for the timing of notes (in the contexts of a fluctuating tempo and expressive delays) to be accurate in the order of around a hundredth of a second. This kind of precision is needed in order to perceive a fluent performance.

Correct interpretation of music of a certain genre is learned usually by imitation of master musicians and by reinforcement (i.e., following a teachers critics) without explicit instruction about the physical parameters of the sound. Today, with the advances in computer technology, it is feasible to provide detailed real-time feedback about these physical parameters such as duration, pitch level or timing deviations. However in a self practice or teaching situation, such detailed information has limited value, first of all because it is not directly a part of the conceptual system of a musician, and, more importantly, because there are so many complex interactions between the parameters that they become meaningless in isolation.

In this project a system will be developed that monitors a student during practice and provides feedback on the success of imitation in an integrated visual way. It allows teachers to build libraries of exercises in the form of recorded audio fragments, examples and counter examples of a specific style or aspect (e.g. 'swing', agogic accent, vibrato) instead of explicit instructions. From the sound fragments (examples and counter examples) the features are deduced that are relevant to explore and learn, and these are extracted in real time while the student is practicing. Feedback is provided as a mapping from these relevant features to parameters of a visual form (shape, texture, orientation, color, position etc.). This helps in exploring through the space of possible performances. How the method helps in learning to control the instrument will be evaluated in several experiments. The best design choices will be put in a prototype for music students that (depending on available technology) may be accessible globally via the Internet or on
computer game consoles.

Utilization
A system aimed at learning the high-level skills of controlling a musical performance more quick and efficient than can be done with feedback from a teacher alone will be constructed. Teachers contribute their lessons in the form of performed examples and the success of the student.s imitation of the various features is displayed online as dynamic visual shapes. After the prototype is fine-tuned in a real educational environment (with conservatory students) and the courseware is established (by conservatory teachers), the application can be commercially exploited by educational publishers, either as software or via the internet.

Using a similar approach, characterizing proper patterns of stress and intonation for correct pronunciation of a foreign language is also possible. Such a education aid has a broader market and is of interest of parties that are involved in language education.