〈 Previous Article ¦ List of all Papers ¦ List of all Authors ¦ Next Article 〉

Real-time estimation of instrument controls with marker based IR cameras.

Perez-Carrillo, Alfonso and Wanderley, Marcelo 

Scientific analysis and understanding of musical performances is an ambitious challenge at the intersection of a wide array of disciplines ranging from motor-learning and cognitive sciences to music pedagogy. Recently, the availability of technology and methods to measure many aspects of the musical performances allow for a better understanding of the mechanisms behind musical practice. Among these aspects, it is of special interest in this work the measurement of instrumental controls. Several methods have been reported in the last years adapted to a specific kind of instrument. Most of these methods are generally intrusive and in many cases they need for data post-processing so that instrumental controls can not be computed in real-time, which in some applications is crucial. We present a method based on high speed video cameras that track the position of reflective markers. The main advantages with respect to previous solutions are that 1) the degree of intrusivity is very low, while 2) it is able to compute the instrumental parameters in real-time based on the geometrical position of the markers and 3) it allows for the measurement of several instruments and performers. The main problem with such optical systems is marker occlusion. Each marker needs to be identified by at least three cameras placed at different angles and planes in order to correctly determine its 3D coordinates. Marker identification is made robust by the use of rigid bodies (RB), a six degrees-of-freedom (6DOF) rigid structure defined by the position of a set of markers and associated with a local system of coordinates (SoC). The position of the markers is constant relative to the local SoC and their global coordinates can be obtained by a simple rotation and translation from the local to the global SoC. Even if some of the markers are occluded, their position can be reconstructed from the others. The method has been successfully applied to bowed strings by tracking the position of bow and strings and it is being adapted to the guitar, which presents extra difficulties as the hands of the performer are flexible skeletons rather than rigid bodies.