Creating embodied hybrid experiences

The music and visual arts communities have been working for the past two decades on creating hybrid physical digital music compositions, visual art works and installations. However, human experience of the world is embodied and multi-modal. Arts can best investigate and depict our current hybrid reality through multi-modal, interactive works that include movement. The main drawback for such works has been the difficulty of digitizing and analyzing movement.

Sampling, digital analysis and digital synthesis of sound and image are already fairly advanced. Complex processing of sound and image can be realized with good accuracy in real time. User-friendly software for interactive analysis and synthesis of sound and image are readily available.

Some of those packages (i.e. Max/MSP/Jitter) allow for the simultaneous processing of both image and sound. The tools for coherent, digital, sound-image interactive compositions are available and increasingly used in all types of settings--from installations in galleries, to jam sessions in music bars, symphonic concerts, educational settings and biodesign experiments.

Documentation and analysis of motion has until recently relied on 2D video capture that provides limited information on the spatial and kinesthetic aspects of movement and has limited capabilities in real time use.

The development of motion capture technology in the past two decades has opened up many possibilities for advanced digitization and analysis of movement. The motion^e project is developing techniques for advanced real time capture and analysis of movement. These motion analysis systems are being integrated with real time systems for image and sound analysis and synthesis to create unified platforms for the creation of multimodal physical-digital works.