Developing A Real Time Activity Recognition for Children with Communication and Social Interaction Disabilities




  • Swinburne Univrsity of Technology Sarawak




According to Thorsen (2013), children with social interaction and communication impairments have three challenging behaviors: 1) destructive behaviors like aggressive, self-Injurious, property destruction, injury to others, throw, push, harmful behavior with hands and feet; 2) disruptive behaviors like tantrums, loud noise/screaming/crying, running, repetitive noises, talking out, negative comments; 3) interfering and/or Irritating behaviors like self-stimulation, repetitive and perseverative speech/questions, argumentative, poor task completion. The key to lock these behaviors is a positive behavioral support which requires a lot of real time care, monitoring and social instruction as appropriate. According to the study on 26 patients with ASD, 30.8% presented suicidal ideation and 7.7% committed suicide (Raja et al., 2011).

For children with social interaction and communication impairments, self-injurious behaviors or activities like head banging (on floors, walls or other surfaces), hand or arm biting, hair pulling, eye gouging, face or head slapping, skin picking, scratching or pinching and forceful head shaking could invite some involuntary injuries that need prevention (The National Autistic Society, 2013). The trigger was mainly due to inability to express their thoughts and be understood by others (Mandasari, 2012).  CALL7 (2011) reported a patient was found dead in Colorado Mental Health Institute after overconsumption of drug, also contributed by the lack of real time activity monitoring. Currently, the focus on assistive technology for children with social interaction and communication impairments is mainly on learning (language and social skills) and speech therapy (Parker & Kamps, 2011; Gaylord et al., 2005; Tobii, 2013; Autism Speaks, 2013). As to date, there is still a lack of real time intelligent ICT based tool to monitor, detect, recognize and alert in case of any critical behavior occurrence.

Nowadays, with the state-of-the-art technologies like motion sensors, depth sensors, wearable sensors, wireless sensor network, Wi-Fi, tablet computers and smartphones, non-intrusive and intuitive real time nursing and caring for the children with social interaction and communication is not impossible. As to date, all these technologies have yet been made intelligent by soft computing to deliver a total solution that can assist to recognize behavior/activities of the children with social interaction and communication impairments, provide them real time social instruction to avoid destructive actions while alerting the caretakers.

In general, wearable sensors are used to provide real time feeds on physiological parameters of interest to consumers and clinicians such as heart rate, physical activity, fall and sleep patterns (Sustainable Nonwovens, 2012; Kepski & Kwolek, 2012) at low cost and high efficiency. Marschollek et al. (2012) highlighted that wearable sensors are essential components as they are an integral part of future pervasive, ubiquitous and person-centered health care delivery. However, the integration into sensor-enhanced health information systems and rigorous evaluation studies appear to be a future challenge. The most vital contribution of wearable sensors and systems is to achieve early detection of changes in user’s status requiring intervention (Patel et al. 2012). On the other hand, smartphones with gyroscope, accelerometer and magnetometer are also considered as wearable sensors that most of the people have them and do not perceive them as intrusive. On the other hand, indoor positioning uses Wi-Fi to locate the user within five to 10 meters through Wi-Fi access points (The Age, 2012). Wi-Fi has become the standard for companies such as Google to perform indoor positioning. Researchers at University of New South Wales have utilized other sensors on the phone such as the gyroscope, accelerometer and magnetometer to bring the accuracy down to within three meters. Currently, Google has indoor maps in countries like the United States, Japan, Canada, Switzerland, Britain, France, Denmark and Sweden with more than 10,000 mapped floor plans for airports, shopping centers, hotels, universities and libraries. Abuzz focused on mapping 80 shopping centers in Australia in high details for indoor positioning while Smarttrack RFID has developed indoor positioning technology for museums and art galleries (The Age, 2012).

Multi depth motion sensors have been adapted in various low cost and high efficiency vision based monitoring (Stone & Skubic, 2011). For instance, fall prevention, post-stroke rehabilitation (Lange, et al., 2011; Saini, et al., 2012; University of Southampton, 2012; Polytechnic University, 2012), early warning system for forthcoming health issues of residents in apartments (Keene, 2011), remote health monitoring for patients who suffer from multiple Sclerosis and chronic ailments which require periodic physiotherapy and monitoring (Cognizant, 2012) and tracking movement in meter/second of at-risk children (Ravishankar et al. 2012). In Ong et al. (2013), a single rotatable multi depth sensor was used to recognize the gestures of falling, inactivity for a period of time and seeking for help by waving hand(s) in an indoor setting, the application was meant for assisting people with disabilities. Prior to submitting this proposal, the applicants modeled and tested some algorithms to recognize daily routines like eating, drinking, reading, writing, standing, walking, running, bending, sitting, brushing teeth and combing hair, shaking head, waving hand on top of the falling, inactivity and seeking help that we have published (Ong et al, 2013). Those algorithms were only tested with volunteers without any disabilities. However, it enlightened us with the idea of utilizing the soft computing and multi sensors to assist our targeted community i.e. children with social interaction and communication through activity recognition to challenging behaviors.

For more information about autism and for information regarding autism support groups for parents, visit :


  • Engineers without Borders (EWB) at the University of Johannesburg

Contact Us


Ashley Moran
445 Hoes Lane
Piscataway, NJ 08854, USA


+1 732-562-6552