Recent developments in Augmented Reality (AR) and Virtual Reality (VR) are opening up new, unprecedented ways of representation and interaction.

Virtual bodies, with an arbitrary behavior, not necessarily depending on physical laws, can be linked to arbitrary functions, spanning from forms of real-time graphic representations to control interfaces and virtual instrument design. Furthermore, those aspects don't have to be assigned to different objects; they can reside in the same body. Therefore, notation, sound, gesture and their interaction can be collapsed together in the same entity, providing new forms of musical expression.

I am exploring those possibilities essentially through three different perspectives:

LINEAR: an environment for real-time composition and performance based on the possibilities offered by the combination of AR and machine learning. The application, used by an iPhone or HTC Vive performer, allows to draw permanent gestures in the air. Those gestures are linked to specific sounds and musical profiles (the results can range from simple melody to advanced real-time sound synthesis). The AR drawings are mirrored to a projector and read as a form of graphic notation by other performers.

SYNESTHESIZER: The SYNESTHESIZER is a timbre synthesizer based on color recognition and Machine Learning. It is inspired by the principle of synesthesia, i.e. when the perceptions through one sense are involuntary extended to other senses. More specifically, we are addressing chromosthesia (also known as sound-to-color synesthesia), i.e. the involuntary experience of a color provoked by a heard sound. The main idea of the SYNESTHESIZER is to use the continuum spectrum of colors in order to explore a continuum spectrum in sounds. Its implementation in VR space will also allow a more engaging interaction, with creation of virtual objects in real-time. The tool is extremely simple and intuitive and open to be used by untrained musicians.

ARScore: This study explores a new way to notate extended techniques on instruments like piano or percussions such as gong, tam-tam and thundersheet. The application allows the representation of gestures (through holograms projected over the instrument’s surface) addressed to specific results. The application allows an easier and extremely more specific way of representing gestures on instruments, avoiding the overcomplex technicality of the notation on paper for what concerns some specific kind of extended techniques.

More information at: https://www.giovannisantini.com/research

Additional Project Images
2018