BETA Project

Background

The number of digital technologies to support the autistic community has been exponentially increasing the past decade [1], whereas no methodological tool is available to assess the evidence base of their efficacy. While autistic people show a strong affinity for digital technologies [2], carers are often concerned about their suitability [3] and practitioners find it difficult to choose the most appropriate ones to support their work [4].

Existing models of evidence-based practice (EBP) [5] are not adequate to address the specific needs of ASD technological interventions [6]. Our aim is, together with the autistic community, to co-develop an accessible EBP framework to enable an understanding of the evidence base for such interventions.

[1] Grynszpan O. et al. (2014) Innovative technology-based interventions for autism spectrum disorders: A meta-analysis. Autism: The International Journal of Research and Practice. 18, 346-361.
[2] Goldsmith, T. R., & LeBlanc, L. A. (2004). Use of technology in interventions for children with autism. Journal of Early and Intensive Behavior Intervention, 1(2), 166.
[3] Fletcher-Watson, S. (2014) A Targeted Review of Computer-Assisted Learning for People with Autism Spectrum Disorder: Towards a Consistent Methodology. Review Journal of Autism and Developmental Disorders, 1, 87-100.
[4] Fletcher-Watson, S., et al. (2014) Parent attitudes to using technology with children with autism in the home: a comparison of the UK and Spain. ITASD, Paris, October 2014.
[5] Mesibov, G.B. and Shea, V. (2011) Evidence-Based Practices and Autism. Autism: The International Journal of Research and Practice, 15, 114-133.
[6] Kasari, C. and Smith, T. (2013) Interventions in schools for children with autism spectrum disorder: methods and recommendations. Autism: The International Journal of Research and Practice, 17, 254-267.

Project aim

Our aim is to put together evaluation guidelines to help people make smart choices when buying or recommending technology for autism. We are based upon the scientific literature valuing at the same time the involvement of the autism and autistic communities in decision-making processes that affect them. At the end of the BETA project we aspire to offer an evidence-based framework to all parties involved in the design and usage of technology for autism, namely autistic people, families, carers, educators, practitioners, developers and researchers.

Project Stages

1. Systematic review of current state of evidence supporting digital interventions for people with autism spectrum disorders (ASD). We assessed the methodological rigour of relevant studies using Reichow’s et al. evaluative method for evidence-based practices in autism [1] and our own novel evaluation scale called UCDI (User-Centred Design for Interventions).
Our systematic review protocol is now officially on University of York’s register.

2. Online surveys using the Delphi [2] methodology with
– autistic people, their families and carers, practitioners and teachers
– researchers with a wide range of expertise: psychology, neuroscience, computer science

We conducted a Delphi study, a methodology that allows experts in a domain to work towards a mutual agreement on a specific topic. BETA investigated what people in the autism and autistic communities look for when selecting a suitable piece of technology. The novelty of our approach was that we treated autistic people and the community around them as experts on the subject.

At the final stage of the study the expert panel was also asked to recommend specific sources of information they use when looking for evidence for autism-related technology. You can find the resources they recommended grouped here.

3. Workshops with expert groups of professionals in software design and development to adapt and refine our evaluation criteria.

One such workshop took place at ISIR, Paris where product managers and designers commented on the feasibility of the emerged guidelines and gave insights on how they can be adapted to be meaningful and useful to tech developers.

4. Implementation of our new evaluation framework on existing international digital technology databases such as AppyAutism, DART and Autism Speaks.

[1]Reichow, B., Volkmar, F. R., & Cicchetti, D. V. (2008). Development of the evaluative method for evaluating and determining evidence-based practices in autism. Journal of autism and developmental disorders, 38(7), 1311-1319.
[2]Boulkedid, R., Abdoul, H., Loustau, M., Sibony, O., & Alberti, C. (2011). Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PloS one, 6(6), e20476.

Conferences, seminars and presentations

BETA featured in seminars around the world and stirred the discussion about the inclusion of the autistic community and and the evidence base of autism-specific technology development

- « Innovative Technologies for Autism Spectrum Disorders » ITASD conference. Slides of introductory project presentation

- Spanish association of autism professionals AETAPI seminar. Slides of the BETA project presentation (English/Spanish)

- « Social Interaction, Intelligent systems and Health » seminar organised by Institute of Engineering in Health at Sorbonne University.

- « International Society for Autism Research » INSAR 2018. Team delegates presented tech demos of their work and designed the first phase of the project’s online survey which will be conducted in summer 2018.

- Centre for Applied Autism Research CAAR seminar series in March 2019. We presented a sample of our findings and gave a taste of the tools we developed during the literature review and the collaborative study. The audience was interested to know how all stakeholders can be well informed and work together to ensure the quality and suitability of tech products.
- « International Society for Autism Research » INSAR 2019. Two posters were presented in the Technological Approaches session explaining the tools developed for evaluation of technology throughout the two years of the project. Read the abstracts here: Poster 1 and Poster 2 (links to INSAR website). Representatives form the academe and the industry showed great interest in applying BETA’s framework to products under design or review.
- Workshops at McGill University in Canada and at the Universities of Edinburgh and Bath in the UK. Presentation slides (pptx)

Associate partners

Richard Mills

Aspergirl

Our online participatory study

1. Participants

For the first two rounds of our Delphi study we recruited

• Adults with a diagnosis of Autism/Asperger
• Family members or carers of autistic people of any age or intellectual ability
• Professionals working with autistic people of any age or intellectual ability
Those categories were not mutually exclusive as one could be, for instance, an autistic individual with an autistic child.

Our participants were also required to have a keen interest in and aptitude for digital technologies.

Their participation played a key role in understanding what is important for the autism community when choosing a piece of technology. We believe that autistic people, their families and people working with them have invaluable expertise and experience to offer to a research process.
Researchers in autism and technology joined the rest of the panel in the third and fourth round. We needed their expertise and scientific judgement in order to check and guarantee the quality of emerging guidelines and make sure we were not inadvertently perpetuating popular myths or poor judgements

2. The process

We split the survey into four phases which took place during the summer of 2018. In the first two phases only community members were invited to share their experiences when selecting pieces of technology. In the following rounds community members together with autism researchers were asked to rank and comment on statements according to their importance and justify their preferences. Seeing how the rest of the panel had evaluated the statements, they had the opportunity to change their previous scores or argue in favour of their choices. At the end of the four rounds the panel reached a consensus and co-created a set of guidelines addressing different aspects of a tech product.

3. The moderator of the study and her role

The moderator of the study organised and facilitated the online study. She distributed scores, anonymous comments and summaries of the results to all participants. Throughout the study she collected and analysed the participants’ responses identifying common and conflicting viewpoints.

4. Anonymity

Participants could read others’ comments and knew that theirs might be read by other participants. Our methodology, however, was based on anonymity. We needed all participants to have an equal say on the matters discussed without being influenced by the identity of the other participants. Similarly, they could not contact the author of a comment directly, but they could respond or elaborate on comments and justifications of other members of the panel.

5. Rules and requirements

1) Participants had a deadline to submit their responses at every round. After that, their responses could not be considered and they could no longer be involved in the study.

2) A very important component of our methodology was anonymity. Everyone was treated equally and for that reason we discouraged sharing personally identifiable information. Otherwise, the moderator had the right to censor participant input at her discretion.

3) A reasonably good level of standard English was required, although our panel was international and not everyone’s native language was English.

4) Arrangements for internet access needed to be made by the participants.

6. Benefits for the participants

1) Their input will help researchers and developers design products that better meet the needs of autistic people.

2) They had the opportunity to see how (other) autistic people, their families and those working with them make choices about technology and what is important to them.

3) They were offered a 30€ Amazon voucher at the end of the study as a thank you for their time and commitment.

7. Ethical approval

Our study has been approved by the Ethical Review Board for Health Research (CERES) based in Paris.

8. Dissemination of study’s results

We can email the final results of the online study, upon request, as well as the results of our whole project once those are published. Alongside papers in scientific journals, we are publishing accessible online documents with the guidelines derived from our whole research process.

Facilitating inclusion in co-design through online communication

Upon completion of our Delphi study, we wanted to share some observations and conclusions about best practice when conducting online research with members of the autism community. Below you can download in PFD format our set of recommendations in English and French.

What did we do?
We conducted a discursive consultation in an online platform together with autistic people, carers, practitioners and researchers with a view to creating a mutually acceptable checklist to help consumers select suitable and high-quality technology for autistic people.

Who is this set of recommendations for?
Based on our methodological choices, observations and informal feedback from participants, we put together a set of recommendations for facilitating inclusion in co-design through online communication. We hope our recommendations will prove useful and open a dialogue with other researchers conducting online participatory studies with autistic participants.

What is the Delphi method?
The Delphi method is a structured communication technique via which experts in a domain anonymously participate in an online study to achieve consensus on a specific topic within their area of expertise. The study takes place in rounds during which participants’ statements are scored and commented upon anonymously until consensus is achieved.

Who took part in this study?
Two subgroups participated in our study: a community group consisting of autistic people, their families/carers and professionals/practitioners working with autistic people. The other group consisted of researchers in the areas of computer science, psychology and education.

Facilitating inclusion in co-design through online communication – Recommendations (PDF – English)

Faciliter l’inclusion dans la co-conception par le biais de la communication en ligne – Conseils pratiques (PDF – Français)

Sources of evidence

The panel members of our online participatory study were asked to recommend to the community sources of evidence they use when they make purchase decisions or recommendations for digital products for autistic people. Sources of evidence (PDF with interactive links)