fMRI-compatible musical instruments

My good friend and colleague Avrum Hollinger created some amazing fMRI-compatible musical instruments while working in the Input Devices and Music Interaction Lab at McGill University. They recently released a video explaining parts of the project and featuring a few of his creations:


Article published in MTAP

Now available online: Malloch, J., Sinclair, S., M. M. Wanderley. Distributed tools for interactive design of heterogeneous signal networks. Multimedia Tools and Applications, 73(2),  2014. DOI: 10.1007/s11042-014-1878-5

We introduce libmapper, an open source, cross-platform software library for flexibly connecting disparate interactive media control systems at run-time. This library implements a minimal, openly-documented protocol meant to replace and improve on existing schemes for connecting digital musical instruments and other interactive systems, bringing clarified, strong semantics to system messaging and description. We use automated discovery and message translation instead of imposed system-representation standards to approach “plug-and-play” usability without sacrificing design flexibility. System modularity is encouraged, and data are transported between peers without centralized servers.

Ph.D. dissertation

My Ph.D. dissertation A Framework and Tools for Mapping of Digital Musical Instruments is available – contact me if you would like an electronic copy.

Digital musical instruments (DMIs) are typically composed of an interface using some type of sensor technology, and real-time media synthesis algorithms running on a digital computer. The connections between various input signals from performer interaction and the parameters of synthesis must be artificially associated – this mapping of gesture to sound or other media defines the behaviour of the system as a whole. Mapping design is a challenging and sometimes frustrating process.

In this dissertation, the design and implementation of an open-source, cross-platform software library and several related tools for supporting the mapping task are presented. These tools are designed to provide discovery and interconnection between parts of DMIs and other interactive systems, and to achieve compatibility through translation and transformation of data representations rather than imposing representation standards. The control parameters of software and hardware devices compliant with libmapper can be freely interconnected without requiring any intended mutual compatibility.

Among the unique features presented is support for mapping between systems that include entities with multiple instances with dynamic lifetimes, systems which would usually require bespoke programming. A formalization of the problem is described, and several examples of real-world applications are outlined.

Finally, use-cases for the mapping tools are presented in-depth: the design, development and use of novel digital musical instruments for live performance.

The Spine

Spine Sketches

The Spine is a “prosthetic” digital musical instrument developed for the collaborative project Les Gestes, in which we endeavoured to design new instruments for dancers.  The new instruments would extrapolate from the T-Stick, which we had already used in the performance Duo pour un violoncelle et un danseur with the same collaborators. Starting with foam prototypes, the Spine and its companion instruments the Rib and the Visor were developed iteratively using participatory design through frequent workshops, parallel problem solving, and digital fabrication methods. The current models are fabricated from laser-cut transparent acrylic, transparent PVC tubing, and PETg rods. The entire structure is assembled using interference fitting rather than any glues or fasteners.

The Spine tracks and reports it’s orientation and shape in real-time, accomplished through the use of inertial and magnetic-field sensing at each end of the structure. Sensor-fusion algorithms run on-board the instrument.

I have previously blogged some teaser photos and a couple of videos showing a demonstration of the orientation and deformation sensing I developed for the Spine and a promo for the upcoming shows.