Autism, iPads, & Interaction – Design for Independence & Inimitable Minds

Many years ago, I sat helpless as a friend of mine voiced her frustration. The three of them had been dealing with Autism Spectrum Disorder (ASD) and other cognitive and behavioural challenges for years with very little help. While listening to her talk about the various tactics and techniques she’s tried, my concern grew.

She has fought so hard just to get the boys to school on a regular basis. What will happen when the boys get older and want the things all young men want? What will happen when my friend becomes too old to care for them? She admitted these were things she worried about, too.

The transition from living at home and going to school, to holding a job and living independently is also the source for many concerns of parents for the more than 1.2 million children and teens in the US (1 in 88 in the US, 1 in 68 in Canada) estimated to be diagnosed with the condition.

Background

According to one of North America’s leading experts on ASD, Bennet Leventhal,

“Children with ASD grow up. Transitioning these folks is extremely difficult…It’s virtually unstudied, and there are no reliable resources available.”

There isn’t much I can do in terms of resources. And even if there were, what kind of  resource could be provided that people would trust and feel confident with? When you think about it, parents of ASD youth need something they can feel confident in. In fact, I think the need to reassure parents is almost more important than the need to reassure the ASD youth in some ways. What if technology and iPads could help?

I quickly made a list of tasks the boys and other youth with ASD struggled with. This was easy particularly since I was already working on a similar project for seniors in the early stages of dementia and Alzheimer’s disease. The more I thought about it, however, the more I realized it wasn’t the features and functionality that were the real problem. The real problem was creating software that worked with the way their minds worked. This was going to require some serious research and testing. And much to my surprise, much of this research hasn’t been done.

The Autistic Mind

Autism isn’t like a rash with a defined set of causes, symptoms, and treatments. While some traits are common across the spectrum, they vary in degree and are accompanied by a whole host of other symptoms and challenges. To address this, I generated a list of the most common features and characteristics (repetitive/restrictive behaviours, language deficits, and social behaviour insufficiencies) and began to research and address these.

Once complete, I hope to build software with enough flexibility and features to make it adaptable and customizable. So, with that in mind, let’s get into my findings and exploration. (To address the needs of the poster session at the I-CYS Conference at the University of Lethbridge, and make them easy for attendees to find, all references can be found at the bottom of the page.)

Designing for Users with Autism for Independent Living

Implicit Processing Errors

Implicit memory — the knowledge we use subconsciously — is a significant problem for ASD individuals particularly in the areas of emotional processing and social cognition (Kaufman, 2010; Peñagarikano & Geschwind, 2012; Carmody & Lewis, 2010). Generally procedural memory, this information is encoded in the same way it was gathered (Kolb, Teskey, & Whishaw, 2016). (See Scoville’s bilateral medial temporal lobe resection of Henry Molaison (H.M.) for more insight.)

In the brain, implicit memory runs on a circuit, which includes the basal ganglia, ventral thalamus, substantia nigra, and premotor cortex (Mishkin et al., 1997). The amygdala also has a role in the process, and when you consider the many common challenges ASD individuals face, this makes sense. The amygdala, along with the medial temporal cortex, brainstem, hypothalamus, periaqueductal gray matter, and basal ganglia, controls emotional memory, which is a significant problem for many with ASD (Kolb, Teskey, & Whishaw, 2016).

A review of all the literature referenced below suggests to me that the information is picked up, but segments of it are lost during encoding, recall, and use, which leads to the wide range of complications with social, emotional, and cognitive processes. (Think of it like packet loss when you have a shoddy internet connection — you can use it, but there are interruptions in the connection. It’s slow, and bits of information get lost.)

Sensory Processing and Disrupted Neural Synchronization

Research from Dinstein et al (2011) and others suggest much of the synchronization and sensory processing errors are caused by issues with the corpus collosum and how the two hemispheres of the brain communicate. In neuroscience speak, this is the collection of neuronal axons that send potentials from one side of the brain to the other. For the rest of us, this is the connection of telephone wires that tells the left side of the brain what the right side is doing and vice versa. For those on the spectrum, it means trouble.

When these connections fail to work properly, language, vision, tactile feedback, auditory information, and other processes get confused. The cells that should fire up may be delayed or receive partial information. There may be a delay in the signals that normally turn those thoughts and feelings off. And, we may not become sensitive or habituated to the things we’re supposed to. Here’s an example:

Normally, our skin and brain becomes habituated to our clothing. This is how we can ignore the feeling of clothing on our skin unless it begins to hurt us in some way. For those with ASD, however, this isn’t always the case. They might go into a full meltdown over a tag in their shirt or a lint ball inside their sock. The sensation overwhelms them so much that it’s all they can think about. (It’s the equivalent of stepping on a sharp pine needle that’s stuck in your sock.)

In terms of language abilities, they may hear you and understand the words you’ve used, but are unable to process it to get the meaning of what you have said and form an appropriate response. Or, they can form a response, but can’t seem to think of the sounds they need to make to say them. Why? Language processing, use, and speech require multiple areas of the brain in both hemispheres.

Poor Social Skills

Those diagnosed with autism have trouble picking up on social cues and responding in the expected manner. They may be aware of cultural and social norms, but are unable to understand how to respond in a way that fits with those expectations. They may be unable to process implicit information such as body language and non-verbal communication. Or, they are unable to complete the actions after processing (connect the stimuli and response with the appropriate sequence of events). The result is that they have difficulty making friends, are easily frustrated, and become easily overwhelmed. And with all the added challenges they have to deal with, who wouldn’t struggle with cognitive overload?

In many instances, one-on-one support and the inclusion in educational institutions can help significantly. Unfortunately, that doesn’t last forever. ASD youth have to leave school eventually. And when they do, all of that support, their friends, their routines, and the life they’ve come to depend on disappears.

They find it difficult to make new friends. The idea of trying new things is a nightmare, and social interaction is awkward. Dating is nearly impossible. As a result, many of those with autism suddenly start suffering with a shrinking social circle, and eventually, social isolation. This causes additional stress as well as the potential for emotional issues such as depression and anger. We’d have the same reaction, so this is no surprise.

Unable to “See the Big Picture” or “Think Ahead”

If you turn the stove on and leave it for a few minutes, you know not to touch it because it will burn you, hurt for days, and likely require an emergency trip to the hospital. Many young adults with autism just know they’re not supposed to touch it. Or, they know they’ll get burned, but don’t realize it will hurt and require emergency medical care. This is an oversimplified example, but you get the idea.

Abstract thought is not a strength for many individuals with autism. And it’s not just the ability to think ahead. Planning, sequencing, and other cognitive challenges also come into play. While most of the time, this results in nothing more than upsetting someone, it can have dire life-and-death consequences.

Many young people with autism will wander away or run off, for example, without considering how they’ll get back or what might happen to them while they’re wandering around alone. They may be so focused on their own thoughts that they fail to pay attention to things such as traffic, which brings me to my next point…

Attention and Sensory Overload

While ASD youth are generally thought of as being very inattentive, the opposite is true. Those on the autism spectrum generally have amazing attention. Where the difficulty comes in, however, is directing their attention and keeping it there.

When they’re interested in something, those living with ASD become hyper-focused. They become fixated on a particular topic or object and ignore the rest of the world around them. They thirst for knowledge on their chosen topic and become almost a savant. In a stimuli-rich environment, however, their hyper-focus (and what may be a lack of synaptic pruning (Tang et al., 2014)) causes what sounds like normal background noise to you or me to sound like a jet engine to someone with autism. They become overwhelmed, unable to focus, and what I imagine to be a full-on panic not unlike the feeling some people get when walking in a terrible storm.

A disinterest in something can also make them highly susceptible to distraction. This can become extremely frustrating for caregivers and the individual. When on their own however, it also makes it difficult for them to complete daily tasks.

Emotions and Expression

One of the most heartbreaking symptoms of autism for parents is the youth’s ability to express affection or identify emotions. It makes communicating in other ways difficult as well. Social interaction becomes further complicated, and life goals such as finding a spouse and living a “normal” life seem impossible.

I believe much of this is also as much of a side effect of the challenges associated with autism as it is a result of the disorder itself. After all, how would you feel if you were never able to truly connect with another human being?

If you were ostracized from society because you failed to follow the unwritten rules you weren’t aware of?

If you’re so frustrated and stressed out with basic daily living that you can’t even begin to think about the “fluffy” things that make life special and fulfilling?

Cognitive Overload

Individuals with autism have a simple set of requirements in my experience. They need to be able to have:

  • Comfort
  • Expectations and fulfillment
  • Simplicity
  • Regular and timely reinforcement
  • Predictability

As soon as these needs are not met, they experience cognitive overload and understandably become agitated and discontent. And when you think about it, we all would feel that way as those are the same things we all want. There is one big difference between ASD and non-ASD individuals, however: the ability to adapt.

When something doesn’t go according to plan, those of us not on the autism spectrum are able to think the situation through and adapt. For those with developmental and cognitive challenges, however, it becomes a disaster. They are unable to figure out how to get from where they are now in their day to the next step because, for them, everything has changed. Think of it like this: You’re playing a fun game of Scrabble. Then, you turn your head, and when you look back, someone has exchanged the Scrabble board for a Monopoly board. You have no idea what to do next. This is why routine, reinforcement, simplicity, predictability, expectations, and familiarity are so important.

iPads and Software as a Solution

I am a firm believer in the idea that people shouldn’t conform to technology. I believe that technology should be developed to work the way we do to help us with the things we are unable to do. And if we can do it, it should make us more efficient. Many of the devices we carry already attempt to do that, but what if it could do the same for those with autism or Alzheimer’s?

Artificial Intelligence and Machine Learning

Donald Norman is a firm believer in the idea that design can (and should) fill in the knowledge and processing gaps between the physical world and our minds (2009). In this instance, I don’t think design is enough by itself. I think it will also require machine learning and artificial intelligence. For example, iPads can deliver alerts, notifications, prompts, and reminders to help those with ASD stay on schedule. I also think these technologies and techniques could keep them connected with their healthcare providers, caregivers, friends, family, and communities.

For example: Let’s say an ASD user enjoys painting and there happens to be a free painting class at the local art gallery. The software, knowing he enjoys painting, could suggest the event. When the individual agrees to go, it would notify his caregiver, arrange for transportation both ways, tell him when it’s time to get ready to leave, and give him a checklist to make sure he doesn’t forget anything.

In short, I see no reason why a properly designed and developed piece of software couldn’t learn routines and either prevent the disruption of those routines, or help them get back on track. If this person’s normal routine includes watching a certain television program every day and suddenly doesn’t one day, the software could send a caregiver or friend a notification. This way, someone could check up on him to make sure everything is ok. If it is able to learn, it could slowly adapt and change to include new experiences in that routine while eliminating the psychological trauma that can come with it.

Sensory Processing, Affordances, and Feedback

iPads and Apple, Inc. already have several cleverly designed accessibility options and apps to help with sensory deficiencies, but what if we took that further? Haptics (tactile feedback), visual, and auditory affordances could make software much easier to use? And by allowing the user to change the kind of feedback provided to the user, it could also be adapted to suit a multitude of needs. So, for example, if someone is easily overwhelmed by auditory stimuli, the sound could be turned off and the information normally provided through auditory affordances could be delivered visually instead.

Because of the way people with autism and Alzheimer’s process and use information, I also suspect that traditional affordances aren’t always as effective as we think. So, by testing this theory, and adjusting these details as necessary, the software would become easier to use. And the more easier it is to use, the more likely someone is to use it on a regular basis and even try new things.

I also feel simplicity is important here. By decentralizing the software (essentially removing the desktop), and making sure features or options only appear if, when, and where they’re needed, it will be possible to reduce distractions, errors, and confusion. Further testing would be required, of course, but this concept is important in standard design, so I can’t see this not holding true for autism/Alzheimer’s-specific design. Research to date also suggests those with ASD have difficulty with rich multimedia interfaces (Grynszpan, Martin, and Nadel, 2008).

Emotions and Emotional Well-Being

Research suggests that those with autism and autism-like challenges are better able to identify emotive words than facial expressions (Han, Yoo, Kim, McMahon, and Renshaw, 2014). However, educational research has found face and emotion flashcards have been helpful to learn emotion identification as well as encourage expression (Macfarlane, 2010). One suggestion was that this may be caused by prosopagnosia. (The inability to recognize faces. Also known as face blindness or facial agnosia.) This will require further study regardless of the cause, but I think the results would be well worth it from a technological and non-technological standpoint.

Once this has been determined, software can offer digital versions of face and emotion flashcards. It could then be used to help them communicate with others as well as learn facial expressions. This could also be used in the software design itself. One of the ways I’d like to do this is through an emotional scorecard, machine learning, and artificial intelligence.

By assessing an individual’s emotional state implicitly and explicitly, it could be tracked and matched to an emotional scorecard. (This could also include general information such as food consumption, medications, whether the individual has socialized, etc.) Then, the software could offer suggestions such as music or ambient sound to help address the emotional state. It could also act as a journal for caregivers and physicians as well as an alert system if things go badly.

Fear and Uncertainty

For those with autism and their loved ones, any software that would allow someone to live independently must be reliable and trustworthy. This doesn’t mean the software needs to be perfect. It means it must handle all situations in a predictable manner. If it is going to allow the user to take control, the transition between machine and human needs to be flawless and predictable. Lastly, the software must be able to find a balance between safety and freedom, which means different things for different people. That’s a tall order, but I believe it’s completely possible.

Designing for someone with autism, or anyone really, requires experts to design for the exceptions rather than the rule. This way, no matter what happens, the software will behave in the way the user expects even if it can’t solve the problem. There are other features to consider outside the basic design. Feedback, for example, needs to be timely, informative, and delivered in the right way. Mistakes either need to be eliminated or mitigated by making it virtually impossible to cause irreparable damage. Or, at least delaying any harmful actions until the user has had time to carefully consider his or her actions.

Conclusion

This seems like a tall order for a simple iPad, but I feel the ends more than justify the means. After all, individuals diagnosed with autism want the same things we all want: to lead a fulfilling life they can control and call their own. And if technology such as tablet software can achieve that then so be it. Even if it only ever helps one person, I think it’s worth it. Additionally, I hope the lessons I learn here can be applied to other objects, disorders, and design practices.

References:

McKillop, K. B. (2016). The Impact of Affordance Types on User Interface Usability Given Varying Aesthetics [Scholarly project]. On KevinMcKillop.ca. Retrieved from http://kevinmckillop.ca/Affordances.pdf

Tang, G., Gudsnuk, K., Kuo, S., Cotrina, M., Rosoklija, G., Sosunov, A., . . . Sulzer, D. (2014). Loss of mTOR-Dependent Macroautophagy Causes Autistic-like Synaptic Pruning Deficits. Neuron, 83(5), 1131-1143. doi:10.1016/j.neuron.2014.07.040

Kaufman, S. B. (2010, March 5). Can People with Autism Learn Implicitly? Retrieved April 01, 2017, from https://www.psychologytoday.com/blog/beautiful-minds/201003/can-people-autism-learn-implicitly

Dinstein, I., Pierce, K., Eyler, L., Solso, S., Malach, R., Behrmann, M., & Courchesne, E. (2011). Disrupted Neural Synchronization in Toddlers with Autism. Neuron, 70(6), 1218-1225. doi:10.1016/j.neuron.2011.04.018

Lartseva, A., Dijkstra, T., & Buitelaar, J. K. (2014). Emotional language processing in autism spectrum disorders: a systematic review. Frontiers in Human Neuroscience, 8, 991. http://doi.org/10.3389/fnhum.2014.00991

Peñagarikano, O., & Geschwind, D. H. (2012). What does CNTNAP2 reveal about autism spectrum disorder? Trends in Molecular Medicine, 18(3), 156-163. doi:10.1016/j.molmed.2012.01.003

Carmody, D. P., & Lewis, M. (2010). Regional white matter development in children with autism spectrum disorders. Developmental Psychobiology, 52(8), 755-763. doi:10.1002/dev.20471

Mishkin, M., Suzuki, W. A., Gadian, D. G., & Vargha-Khadem, F. (1997). Hierarchical organization of cognitive memory. Philosophical Transactions of the Royal Society B: Biological Sciences, 352(1360), 1461-1467. doi:10.1098/rstb.1997.0132

Kolb, B., Teskey, G. C., & Whishaw, I. Q. (2016). An introduction to brain and behavior (5th ed.). NY, NY: Worth .

Favaro, A. (2014, March 27). U.S. and Canadian autism rates on the rise, studies show. Retrieved April 01, 2017, from http://www.ctvnews.ca/health/u-s-and-canadian-autism-rates-on-the-rise-studies-show-1.1749020

Leventhal, B. (2016, August 27). Retrieved April 01, 2017, from https://www.youtube.com/watch?v=gQtqysYH_YU

Han, D. H., Yoo, H. J., Kim, B. N., McMahon, W., & Renshaw, P. F. (2014, March 12). Brain Activity of Adolescents with High Functioning Autism in Response to Emotional Words and Facial Emoticons. Retrieved April 01, 2017, from http://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0091214

Leo, G. D., & Leroy, G. (2008). Smartphones to facilitate communication and improve social skills of children with severe autism spectrum disorder. Proceedings of the 7th international conference on Interaction design and children – IDC ’08, 45-48. doi:10.1145/1463689.1463715

Grynszpan, O., Martin, J., & Nadel, J. (2008). Multimedia interfaces for users with high functioning autism: An empirical investigation. International Journal of Human-Computer Studies, 66(8), 628-639. doi:10.1016/j.ijhcs.2008.04.001

Boyd, B. A., Mccarty, C. H., & Sethi, C. (2014). Families of Children with Autism: A Synthesis of Family Routines Literature. Journal of Occupational Science, 21(3), 322-333. doi:10.1080/14427591.2014.908816

DiGennaro Reed, F. D., Hyman, S. R., & J. M. (2011). Applications of technology to teach social skills to children with autism. Research in Autism Spectrum Disorders, 5(3), 1003-1010. http://dx.doi.org/10.1016/j.rasd.2011.01.022

Escobedo, L., Nguyen, D. H., Boyd, L., Hirano, S., Rangel, A., Garcia-Rosas, D., . . . Hayes, G. (2012). MOSOCO: a mobile assistive tool to support children with autism practicing social skills in real-life situations. In CHI ’12 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2589-2598). Austin, TX: ACN. doi:https://doi.org/10.1145/2207676.2208649

Rodger, M. (2014, June). Https://www.researchgate.net/publication/272161190_The_Nature_of_Auditory_Affordances. Lecture presented at European Workshop on Ecological Psychology in Fermanagh, UK, Lusty Beg.

Mustafa, M., Arshad, H., & Zaman, H. B. (2013). Framework Methodology of the Autism Children — Vibratory Haptic Interface (AC-VHI). 2013 International Conference on Advanced Computer Science Applications and Technologies, 201-206. doi:10.1109/acsat.2013.47

Lofland, K. B. (n.d.). The Use of Technology in Treatment of Autism Spectrum Disorders. Retrieved April 01, 2017, from https://www.iidc.indiana.edu/pages/the-use-of-technology-in-treatment-of-autism-spectrum-disorders

Callenmark, B., Kjellin, L., Rönnqvist, L., & Bölte, S. (2014). Explicit versus implicit social cognition testing in autism spectrum disorder. Autism, 18(6), 684-693. doi:10.1177/1362361313492393

Macfarlane, J. R. (2010). Teaching emotion recognition to children with autism: effects of two computer displayed interventions (Master’s thesis, Simon Fraser University, 2010). Burnaby, BC: Simon Fraser University. Retrieved March 31, 2017, from http://summit.sfu.ca/item/11883

Norman, D. A. (2002). The design of everyday things. New York: Basic Books.

Norman, D. A., & Rumelhart, D. E. (1975). Explorations in cognition. San Francisco: W.H. Freeman and Company.

Norman, D. A. (2009). The design of future things. New York: Basic Books/Perseus Book Group.

Comments are closed