AME Mixed Reality System for Stroke Rehabilitation

Foundational Principles and Structure

Our team has developed an adaptive, mixed reality rehabilitation (AMMR) system to train reaching and grasping movements of stroke survivors. Mixed reality therapy, combining training in virtual and physical environments, was developed to connect virtual learning to physical reality, and thus better facilitate the transfer of strategies from therapy to activities of daily living. The AMRR system integrates rehabilitation and motor learning theories with high-resolution motion capture and sensing technologies, smart physical objects, and interactive computer graphics and sound.

The AMRR system uses kinematic measurements, derived from motion capture data, to evaluate a participant’s performance and generate multiple streams of customizable audio and visual feedback. A participant’s movement is simplified into an action representation composed of key kinematic features of a reaching and grasping action. A simplified action representation is necessary to reduce the reach and grasp movement into a manageable number of measurable kinematic attributes and to provide general relationships among those attributes relative to accomplishing the action goal. The simplified representation is derived from principles within rehabilitation practice, motor learning research, and phenomenological approaches to interactive technology. Within the AMRR System, kinematic features that comprise the action representation are extracted from the 3-D positions of reflective sensors placed on the participant’s torso, affected arm and hand, which are tracked by optical motion capture cameras during movement.

The AMRR system uses digital audio and visual feedback to intuitively communicate to the stroke survivor levels of his performance and direction for improvement. Individual audio and visual feedback mappings correspond to different kinematic attributes within the action representation. While each feedback mapping communicates performance of an individual kinematic attribute, all feedback mappings integrate into one audiovisual narrative in order to communicate the stroke survivor’s overall performance. The design of the interactive feedback is based on perceptual and design principles used within film, music, dance, and media arts in order to facilitate intuitive communication to, and encourage self-assessment by, the participant.

Our system integrates adaptive virtual feedback with adaptable physical elements, which increases the customization possibilities for each individual subject. It allows for the clinician to (1) increase the virtual elements in order to recontextualize the reaching task within the feedback environment, and (2) decrease (fade) feedback to facilitate transfer of performance knowledge gained through virtual elements to unassisted physical action. Because the system provides a customizable way to map functional features of a participant’s movement to real-time interactive feedback, the system can provide training scenarios appropriate for stroke survivors of varying impairment and throughout different stages of recovery. Further details on the system and its original development work can be found in the list of papers given below. Information on current trials and ongoing research is given in the paragraphs below.

Computational Assessment

A stroke survivor’s movement performance during the reach and grasp task is evaluated by using a novel computational measure developed by our team – the Kinematic Impairment Measure (KIM). This evaluation is a real-time, standard measure that maps an individual’s movement to a normalized value between zero (idealized movement) and one (maximal deviation from the idealized movement). The idealized movement reference was derived from a sample of unimpaired individuals performing the reaching task. An attribute-specific continuous model, based on data collected from stroke survivors of varied impairment, is then used to map the raw kinematic values to a KIM value between zero and one. The maximal deviation from unimpaired movement, a KIM measure of one, refers to the estimated greatest possible impairment, while still being able to physically attempt the movement. This estimate and the models for computing each attribute KIM are constantly updated and improved as more data is gathered. The KIM measure may be used to assess individual kinematic features (e.g. peak speed, torso flexion compensation), categories of features (e.g. compensation, temporal profile), or overall movement performance across all categories of kinematic features to create a composite KIM score.

recovery of physical space

Analysis of individual kinematic features and categories of features, with respect to the composite KIM score, allows for identification of how each movement feature or kinematic category contributes to the participant’s functional impairment. The KIM measure thus can be used to measure the difference between a stroke survivor’s movements and idealized, unimpaired movements as a standard way to calculate and compare performance between and across participants, while also tracking rehabilitation progress quantitatively over time and across multiple kinematic dimensions. Our emerging data set indicates that the KIM measure is highly correlated with clinical scales as well as clinical observations and is robust to variations in impairment and performance within and between participants. When used within the AMRR system, the KIM provides detailed, real-time information about the participant’s movement and progress for the clinician, and can be used to inform the clinician’s adaptation decisions.

System Adaptation

The AMRR system is adaptable to maintain a level of challenge and engagement appropriate for each stroke survivor’s impairment and progress. Reaches are preformed in sets of 10, and the clinician can adapt each set in a number of ways. The clinician can focus the training on any set of kinematic features by activating the system’s related explicit feedback mappings (or group of mappings for implicit feedback). The individual feedback sensitivities can also be adjusted to gradually increase or decrease the challenge of the task, depending on the participant’s progress. The clinician decides how many sets will focus on a specific task or kinematic feature, and whether the training of the selected aspects will be continuous or intermittent. The clinician may make any of these adaptations after each set of 10 reaches, and his or her decision is based on the kinematic impairment measures (KIMs), graphical visualization of kinematic values (e.g. trajectory, velocity, joint angles) and direct observation of the stroke survivor’s performance.

AMRR training can utilize four target locations, each of which require the use of a unique combination of joints, ranging from a simple to a more complex joint space. The physical target may either be a cone or a large button to be pressed, both of which can sense the user's touch though sensors that measure contact and force. Three types of training environments may be used: a purely physical environment (no audio or visual feedback with a physical target), a purely virtual environment (audio and/or visual feedback provided on the movement without a physical target), or a mixed environment (audio and/or visual feedback provided on the movement with a physical target).

Current Research

Clinical Study At Banner Baywood

In the Spring of 2009, we began a partnership with the Rhodes Rehabilitation Institute at Banner Baywood Medical Center. The AMRR system was installed at the Institute and we initiated a clinical research study that compares our mediated therapy to traditional physical therapy. This study will further inform development of a home based training system that will be used in partnership with Banner Baywood Medical Center as well. As of December 2010, we have recruited 12 stroke survivors to participate in the study and aim to recruit a total of 25 participants. The participants in the intervention group experience 4 weeks of mediated training (training utilizing the AMRR system) and the participants in the control group experience four weeks of traditional therapy. Training using the mediated system includes a low-cost motion tracking system, full feedback paradigm, and various tangible objects. Results from the clinical study to-date show that our system has significantly improved performance of a reaching and grasping task for stroke survivors with different severity of stroke, after one month of short-term training. Improvement of participants using our system also compares favorably to improvements of our control group that is undergoing traditional physical therapy.

Below is a panorama of the system currently in place at Banner Baywood. Click and drag to scroll around the room!

Home System


Our team is also developing a scaled-down version of the AMRR system for home-based rehabilitation. The home rehabilitation system attempts to bring the training and assessment of clinician-led therapy sessions to a stroke survivor’s home in an affordable and low cost manner. The first version of the home rehabilitation system consisted of a table top game that featured three fixed targets of varying difficulty for the participant to reach towards and grasp. The 3-D position of the participant’s wrist was tracked by using the infrared technology of two Nintendo Wii Remotes. This preliminary assessment tool was used by two stroke survivor’s, from which basic kinematic calculations and usage patterns were collected. The overall response from both participants was very positive, as both utilized the system well above the minimum requirements.

The main design goal of the second version of the home system currently under development is to advance beyond a pure assessment tool and develop an active and engaging training device that can integrate within the home environment. In order to promote thinking about and understanding appropriate strategies to complete the task, the design of this system incorporates a shift in paradigm from emphasizing real-time feedback to summary feedback of overall activity. Feedback is provided across different timescales, including real time and after one or multiple sets of tasks, to create an evolving experience over the multiple months that a participant uses this device.

The second version of the home system also begins to address how to best facilitate unsupervised training. As a clinician will not be present while the participant uses the system, an algorithmic adaptation component is being developed to help the participant progress through therapy and to adapt to the participant’s dynamic improvement. Currently, data from the AMRR hospital-based system is informing the development of the first implementation of the home-based system’s adaptation algorithm. The goal is to create a meaningful therapy experience over multiple months, with only intermittent intervention from the clinician, who reviews the participant’s progress once a week.

The second version of the home system will be tested in user-based studies in Fall 2012. Beyond this next iteration, future developments with regards to home therapy are also being explored including: (1) development of interactive components that can be moved outside of a motion capture space, thus integrating more with activities of daily living and (2) an improved tele-rehabilitation component for the clinician to remotely interact with the participant as needed.

Brain Imaging


Experiments are currently underway incorporating functional MRI (fMRI) and electroencephalography (EEG) to observe changes in neural activity associated with training in a mixed reality rehabilitation system. The fMRI study looks at long-term plasticity of the brain from training in a mixed reality system by comparing fMRI scan results before and after training. These scans are conducted at the Barrow Neurological Institute of St. Joseph's Health and Medical Center. Current EEG experiments at ASU using a 32 channel EEG system are examining the effects of the feedback environment on neural activity in non-impaired subjects. Through innovative EEG analysis techniques drawing from chaos theory and information theory, combined with time-synchronized electromyography (EMG) and kinematic data, we are answering questions about the neural and muscular activity involved in the preparation and performance of reaching and grasping in a mixed reality environment. Following up on our success with non-impaired subjects, an EEG study monitoring brain activity in stroke subjects utilizing the rehabilitation system is scheduled to begin in 2011.

Project Publications

  • Nicole Lehrer, Suneth Attygalle, Steve L. Wolf, and Thanassis Rikakis. "Exploring the bases for a mixed reality stroke rehabilitation system, Part I: A unified approach for representing action, quantitative evaluation, and interactive feedback." accepted for publication, Journal of NeuroEngineering and Rehabilitation, 2011. PDF
  • Nicole Lehrer, Yinpeng Chen, Margaret Duff, Steve L. Wolf, and Thanassis Rikakis. "Exploring the bases for a mixed reality stroke rehabilitation system, Part II: Design of Interactive Feedback for upper limb rehabilitation." accepted for publication, Journal of NeuroEngineering and Rehabilitation, 2011. PDF
  • Diana Siwiak, Nicole Lehrer, Michael Baran, Yinpeng Chen, Margaret Duff, Todd Ingalls, and Thanassis Rikakis. "A Home-based Adaptive Mixed Reality Rehabilitation System", ACM Multimedia 2011 PDF
  • Michael Baran, Nicole Lehrer, Diana Siwiak, Yinpeng Chen, Margaret Duff, Todd Ingalls, and Thanassis Rikakis. "Design of a home-based adaptive mixed reality rehabilitation system for stroke survivors." 33rd Annual International IEEE EMBS Conference, Boston, Massachusetts, August 30 - September 3, 2011. PDF
  • Thanassis Rikakis, “Utilizing Media Arts Principles for Designing Effective Neurorehabilitation Systems”, 33rd Annual International IEEE EMBS Conference, Boston, Massachusetts, August 30 - September 3, 2011. PDF
  • Aaron Faith, Yinpeng Chen, Thanassis Rikakis and Leo Iasemidis, “Interactive Rehabilitation and Dynamical Analysis of Scalp EEG”, 33rd Annual International IEEE EMBS Conference, Boston, Massachusetts, August 30 - September 3, 2011. PDF
  • Yinpeng Chen, Margaret Duff, Nicole Lehrer, Sheng-Min Liu, Paul Blake, Steve L. Wolf, Hari Sundaram, and Thanassis Rikakis, "A Novel Adaptive Mixed Reality System for Stroke Rehabilitation: Principles, Proof of Concept and Preliminary Application in Two Patients", Topics in Stroke Rehabilitation 2011, 18(3):212–230 PDF
  • Yinpeng Chen, Margaret Duff, Nicole Lehrer, Hari Sundaram, Jiping He, Steve L. Wolf, and Thanassis Rikakis T, “A Computational Framework for Quantitative Evaluation of Movement during Rehabilitation”, International Symposium on Computational Models for Life Sciences, Tokyo, Japan, 22-24 June 2011. PDF
  • Hari Sundaram, Yinpeng Chen, and Thanassis Rikakis. "A Computational Framework for Constructing Interactive Feedback for Assisting Motor Learning." PDF
  • Margaret Duff, Yinpeng Chen, Suneth Attygalle, Hari Sundaram, and Thanassis Rikakis. "Mixed Reality Rehabilitation for Stroke Survivors Promotes Generalized Motor Improvements." 32nd Annual International IEEE EMBS Conference, Buenos Aires, Argentina, August 31 - September 4, 2010. PDF
  • Jeffrey Boyd, Hari Sundaram, and Aviral Shrivastava. "Power-accuracy tradeoffs in human activity transition detection," Design, Automation & Test in Europe Conference & Exhibition, Dresden, Germany, 8-12 March 2010. PDF
  • Yinpeng Chen, Nicole Lehrer, Hari Sundaram and Thanassis Rikakis. "Adaptive Mixed Reality Stroke Rehabilitation: System Architecture and Evaluation Metrics". 1st Multimedia Systems Conference (MMSys 2010). Feb. 2010. Scottsdale, AZ. PDF
  • Margaret Duff, Yinpeng Chen, Suneth Attygalle, Janice Herman, Hari Sundaram, Gang Qian, Jiping He, and Thanassis Rikakis. "An Adaptive Mixed Reality Training System for Stroke Rehabilitation." IEEE Transactions on Neural Systems & Rehabilitation Engineering, 2010. 18: p. 531-541. PDF
  • Jeffrey Boyd and Hari Sundaram. "A framework to detect and classify activity transitions in low-power applications," IEEE International Conference on Multimedia and Expo, 2009 (ICME 2009). New York, June 28 - July 3 2009. PDF
  • Yinpeng Chen, Weiwei Xu, Hari Sundaram, Thanassis Rikakis, and Sheng-Min Liu. "A Dynamic Decision Network Framework for Online Media Adaptation in Stroke Rehabilitation" ACM Transactions on Multimedia Computing, Communications, and Applications. Oct. 2008 PDF
  • Suneth Attygalle, Margaret Duff, Thanassis Rikakis, and Jiping He. "Low-cost, at-home assessment system with Wii Remote based motion capture". Virtual Rehabilitation 2008, Vancouver, Canada, 2008. [second best student paper award] PDF
  • Margaret Duff, Suneth Attygalle, Jiping He, and Thanassis Rikakis. "A Portable, Low-cost Assessment Device for Reaching Times". Engineering in Medicine and Biology Conference (EMBC2008), Vancouver, Canada, 2008. PDF
  • Yinpeng Chen, Hari Sundaram, Thanassis Rikakis, Loren Olson, Todd Ingalls, and Jiping He. "Experiential Media Systems - The Biofeedback Project, in Multimedia Content Analysis: Theory and Applications", A. Divakaran (eds.), Springer Verlag, Oct. 2008. PDF
  • Yufei Liu and Gang Qian. Projector-Camera Based Fast Environment Restoration of a Biofeedback System for Rehabilitation, Projector and Camera Systems Workshop, at IEEE Conference on Computer Vision and Pattern Recognition (CVPR'07), Minneapolis, 2007. PDF
  • Isaac Wallis, Todd Ingalls, Thanassis Rikakis, Loren Olson, Yinpeng Chen, Weiwei Xu, and Hari Sundaram. "Realtime Sonification of Movement for an Immersive Stroke Rehabilitation Environment." International Conference on Auditory Display (ICAD 2007), Montreal, Canada, 2007. PDF
  • Weiwei Xu and Hari Sundaram. Information Dense Summaries for Review of Patient Performance in Biofeedback Rehabilitation, Proceedings of the 15th annual ACM international conference on Multimedia (paper), ACM Press, Sep. 2007, Augsburg, Germany. PDF
  • Weiwei Xu and Hari Sundaram. Information Dense Summaries for Review of Patient Performance in Biofeedback Rehabilitation, Proceedings of the 15th annual ACM international conference on Multimedia (demo paper), ACM Press, Sep. 2007, Augsburg, Germany. [best paper finalist] PDF
  • Yinpeng Chen, Weiwei Xu, Hari Sundaram, Thanassis Rikakis, and Sheng-Min Liu. "Media Adaptation Framework in Biofeedback System for Stroke Patient Rehabilitation", ACM Multimedia 2007 PDF
  • Yinpeng Chen, He Huang, Weiwei Xu, Richard Isaac Wallis, Hari Sundaram, Thanassis Rikakis, Todd Ingalls, Loren Olson, and Jiping He. "The Design of a Real-Time, Multimodal Biofeedback System for Stroke Patient Rehabilitation." ACM Multimedia 2006 PDF
  • Yinpeng Chen, He Huang, Weiwei Xu, Richard Isaac Wallis, Hari Sundaram, Thanassis Rikakis, Todd Ingalls, Loren Olson, and Jiping He. "A Real-Time, Multimodal Biofeedback System For Stroke Patient Rehabilitation" (demo paper) ACM Multimedia 2006 [best demo award] PDF
  • Weiwei Xu, Yinpeng Chen, Hari Sundaram, and Thanassis Rikakis. "Multimodal Archiving, Real-Time Annotation and Information Visualization in a Biofeedback System for Stroke Patient Rehabilitation.", CARPE'06, 2006. PDF
  • He Huang, Todd Ingalls, Loren Olson, Kathleen Ganley, Thanassis Rikakis, Jiping He. “Interactive, Multimodal Biofeedback System for Task-Oriented Neural Rehabilitation”. IEEE-EMBC 2005, Shanghai, China. PDF
  • He Huang, Jiping He, Thanassis Rikakis, Todd Ingalls, Loren Olson. “A new framework of biofeedback system for neural rehabilitation.” Biomedical Engineering Society Fall meeting. 2004.
  • He Huang, Jiping He, Thanassis Rikakis, Todd Ingalls, Loren Olson. “Design of biofeedback system to assist the robot-aided movement therapy for stroke rehabilitation.” Proceeding of Society for Neuroscience's 34th Annual Meeting, 2004.