SMALLab - Situated Multimedia Arts Learning Lab

Software Architecture
	Six interrelated software modules drive the interactive system. This diagram illustrates the bi-directional data flow between each system, and each component is briefly described below.
Vision based sensing
3 firewire cameras mounted above the space	Three Point Grey Dragonfly2 (VGA resolution at 60FPS) video cameras are mounted around the active space. These are connected serially on the same IEEE1394 bus to the PC through a single port. The videos from these three cameras are automatically synchronized. The tracking system uses color information to locate the 3D positions of multiple objects in the space. 2D candidate object locations in each camera view are first obtained using the Continuously Adaptive Mean Shift (CAMSHIFT) algorithm provided in OpenCV. Triangulation is then used to find out the 3D object locations. Before each session, the external camera calibration parameters are computed using the multi-camera calibration toolbox. The current system is capable of tracking up to five objects at forty frames per second. This three-dimensional position data, along with velocity and magnitude, is sensed and broadcast to other modules. We are experimenting with various scenarios that combine student and object tracking in order to detect movement patterns that arise from collaboration between students. For example, the sensing system is capable of extracting patterned movements such as primitive shapes that emerge over short time spans.
Sound Feedback
4.1-channel spatialized audio projection system and software	Sound plays a critical role in SMALLab and many of our learning exercises depend on the immersive, three-dimensional nature of sound. Four raised speakers and one subwoofer surround the space. We have developed software to project spatialized, reactive sound into the space. An extensible database of soundfiles supports this module, and through in-classroom and web based interfaces, both students and teachers can contribute sound content. Our current work in this area extends prior research in the development of interactive installations, and borrows techniques from musique concréte and concatenative music composition. We have designed specialized learning modules that allow students to record their own sounds from the environment and then discuss, share, and interact with those sounds in SMALLab. During classroom activities and via the Edulink website, these collected sounds can be annotated and auditioned by students and teachers. These annotations inform our models for interaction and allow for the delivery of sonic feedback that is adaptable to individuals and groups of students. The image to the left shows a screenshot of the interface to sound feedback. Please click here or on the waveform image to hear an example of a student creating sound in real time in SMALLab.
Visual Feedback
Top-mounted video projector displays on the floor	A top-mounted video projector displays interactive visual content on the floor of SMALLab. In contrast to related work CAVE environments, we have sought to develop an architecture that promotes social interaction and collaboration among groups of students. The absence of projection screens surrounding the space subverts many biases of screen-based media, and creates an open physical environment. Our feedback frameworks utilize still images and video clips that are collected and annotated by the students. In addition, we have developed a three-dimensional graphics engine using OpenGL within the Max/MSP/Jitter programming environment. Interactive graphics modules are coupled with specialized learning exercises to assist in the development of students’ understanding of spatial relationships, movement dynamics, and activity patterns. The images to the left show animations of students' movement during learning exercises. The top image shows a histogram of 3D movement (please click to see a rotated animation). The bottom image shows a virtual ball simulating the real movement in the space.
Data Archival and Annotation
MySQL database archives data and shares between locations	We have developed a software module with a MySQL database backend to archive all sensing and feedback data in real time. This stored data can be accessed for a number of purposes. First, during a given learning session, students can recall and replay movement passages to reflect on their activities. Second, this data is used to update real time context models that can inform our feedback mechanisms. Finally, this data can be used for evaluation and assessment by providing a detailed view of students’ activities over multiple time scales. For example, we are currently examining the relationships between student movement patterns and sonic feedback to better understand how sound can be used to influence movement in service of more efficient learning.
Context Models
Software for analysing and modeling all data	Informed by domain knowledge from the arts, sciences, education, and psychology, we are developing computational models of context. Currently this work is focused on the understanding of students’ sound and movement interactions. The model first extracts the elements and principles underlying the artifacts of each student’s interaction. Second, it tracks how this context develops over time and is individuated amongst different students. The context consists of lexica representing the students’ modes of interaction within the space, plus a syntax describing how lexica relate across different times or modalities. For example, in one of our developing learning modules, students are asked to express an abstract idea, such as “joy”, using their voices and interactive sounds in SMALLab. Corresponding sound-oriented lexica include “swooping” (up-down) pitch contours in the students’ vocalizations, identical or contrasting contours in the sounds projected by the feedback apparatus, and an idea that the student is expressing through sound and movement. A corresponding syntactic rule is that, given a set of sounds tagged by a particular student as related to “joy”, the pitch contour of the student’s vocalization is more likely to match that of the feedback immediately preceding the vocalization provided that both share “up-down” patterns. Conversely, they might be more likely to contrast if the feedback contains strictly “up” or “down” gestures. Such complex syntactic relations may differ from student to student, but they nevertheless form an integral part of how each student constructs the concept, “joy,” through interaction. In a broader sense, syntax may encompass the patterns in sound that emerge from the relationship of three-dimensional position and the dynamics of movement during various learning modules.