INTRODUCTION
Children with developmental disabilities (DDs) are characterized by impairment in areas of physical, learning, language, or behavior. DD includes autism spectrum disorder (ASD), intellectual disability (ID), and language disorder (LD), all of which are described as neurodevelopmental disorders in the Diagnostic and Statistical Manual of Mental Disorders, 5th edition (DSM-5) [1]. Generally, studies conducted in children with DD or their families suggest that intensive interventions from an early age are important [2,3]. However, most of the interventions require a high educator-to-child ratio because they are delivered to individuals or small groups of children. There are more barriers to general usage of intensive interventions, including high cost and uneven geographic accessibility [4]. As a result, only a limited proportion of children who are initially diagnosed with DD have the chance to receive appropriate individualized interventions.
During the last decade, mobile devices such as smartphones and tablets became widely available, and the potential of their use has also increased. Before the widespread availability of mobile devices, cognitive training programs were based on personal computers and their efficacy was studied in patients and healthy participants [5]. As mobile devices rapidly proliferated, mobile-based cognitive training programs gained popularity. Lumosity and BrainHQ are an example of mobilebased general educational apps that are developed to promote cognitive function [6,7], and are marketed to be effective in improving cognitive outcomes in typically developing children. However, studies on those apps have methodological issues such as the absence of active controls and inclusion of insufficient number of participants [8]. Despite these issues, mobilebased apps are easy to access and mostly do not require trained educators, and thus can be an alternative to children with DD who cannot receive in-person interventions.
Recently, efforts have been made to develop cognitive training programs specifically for children with DD in addition to children with typical development [9,10]. DoBrain is a mobilebased cognitive training program designed for children with DD [11], and targets areas of primary cognitive capacity (attention, orientation, memory), higher-level thinking abilities (problem-solving, reasoning, concept formation), and metaprocessing abilities (executive function, self-awareness). The program consists of multiple levels of games with different levels of difficulty, and only requires simple screen touches as an input. Although DoBrain is commercially available and actively used in countries including Korea and the United States, the efficacy of DoBrain has not been fully examined yet.
Therefore, in a community-based sample of preschool children, we compared the efficacy of DoBrain with Junior Naver and Kakao Kids (JNKK), which are the most popular general educational apps used in Korea for children with typical development. In addition, we examined whether the efficacy of DoBrain was different between children with DD and those without.
METHODS
Participants and procedure
A power analysis showed that with 100 children in each group, the study would have 80% power to detect a moderate effect (Cohen’s d=0.4, p<0.05, two-tailed). A total of 200 children were enrolled upon consent from their parents (Figure 1). The children were enrolled at community-based daycare centers, kindergartens, and special education centers from May 1, 2020 to July 31, 2020. The enrolled children were between 34 and 77 months of age. Exclusion criteria included 1) history of neurologic diseases such as cerebral palsy, 2) any sensory disturbances (i.e., vision, hearing, taste, or smell), or 3) severe gross or fine motor problems that prevented them from participating in psychometric tests. Children who were receiving language or behavioral intervention were eligible if no changes in treatment were planned during the study duration of 24 weeks. Of the 200 children who were enrolled, 81 children were diagnosed with DDs and 119 children showed typical development. DDs included ASD, ID, and LD, and the diagnosis of each disorder was confirmed by board-certified child and adolescent psychiatrists based on DSM-5 and relevant psychometric tests including Psychoeducational Profile-Revised (PEP-R), Preschool Receptive-Expressive Language Scale (PRES) or Sequenced Language Scale for Infants (SELSI), and Korean version of the Childhood Autism Rating Scale (K-CARS). When a diagnosis of ID was confirmed, a diagnosis of LD was not applicable. If a child did not have any diagnosable DD, the child was classified as having normal development. The study was approved by the Institutional Review Board of Asan Medical Center (2020-0386). Trial registration: Clinical Research Information Service KCT0007096 (https://cris.nih.go.kr/cris/index/index.do).
Study design
DoBrain, which is a mobile-based cognitive training app designed for children with DD, was used in the DoBrain group (Supplementary Figure 1 in the online-only Data Supplement) [12]. Junior Naver [13] and Kakao Kids [14], which are general educational apps designed for typically developing children, were used in the JNKK group. The Junior Naver app includes educational videos, audiobooks, and games that are designed to stimulate imagination and promote development. The Kakao Kids app includes educational contents such as language games, mathematical quizzes, videos on social relationships, and fairy tales that are based on the national educational curriculum for preschool children. No difficulty adjustment was made for the general educational apps.
The present study was a single-blinded, randomized clinical trial. Children participating in the study were assigned to the DoBrain group or JNKK group in a 1:1 ratio using the block randomization method. The children and their parents were aware of the group assignment, but the investigators and evaluators were blinded to group assignment. Only the study coordinator was aware of the group assignment, who did not take part in evaluation or data analysis. To ensure the blinding, children and parents were instructed not to discuss the intervention method while meeting with the investigators and evaluators.
Children in the DoBrain group were instructed to use the DoBrain app for 30 minutes every day for 24 weeks. Children in the JNKK group were instructed to use the Junior Naver app for 30 minutes every day in the first 12 weeks and the Kakao Kids app for 30 minutes every day in the next 12 weeks. Parents were instructed to help the children with the app usage. App compliance was monitored via the central server system. When the expected time was not fulfilled, a text message encouraging additional usage was sent on a weekly basis. The app was delivered through smartphones or tablets.
Assessment and measures
The primary outcome of the present study was cognitive development, and the secondary outcomes were adaptive functioning, language skills, ASD symptoms, behavioral problems, and caregiver stress and depression. Relevant psychological tests and questionnaires were administered once at baseline after randomization and once more at post-test after the end of the 24-week study duration.
Cognitive development
Cognitive development was assessed by the PEP-R [15] that assesses seven domains of development: imitation, perception, fine motor, gross motor, eye-hand coordination, cognitive performance, and cognitive verbal performance. Previous studies reported that the PEP-R shows good internal consistency, test-retest reliability, and high concurrent validity with the Stanford-Binet Intelligence Scale and the Vineland Adaptive Behavior Scale [16,17]. The Developmental quotient (DQ) was calculated to assess the overall developmental level ([developmental age/chronological age]×100), where a lower DQ indicates more delay in cognitive development.
Adaptive functioning
The Vineland Adaptive Behavior Scale, 2nd edition (VABS), which is a semi-structured interview, was administered to estimate adaptive functioning [18]. Overall standard Adaptive Behavior Composite Score (ABCS) and standardized scores for 4 domains were obtained. Standardized scores have a mean value of 100±15 and higher scores indicate a higher level of functioning. The VABS demonstrated good-to-excellent splithalf and test-retest reliability and modest concurrent validity [18].
Language skills
The PRES, which was standardized and validated in Korean children 2-to 6-years of age, was used as the primary measure for language skills [19]. When PRES was not applicable due to inadequate language abilities or other child factors, the SELSI was administered [20]. SELSI is an indirect test for language skills where caregivers are questioned instead of children, which was standardized in 1,090 typically developing children. The Receptive and Expressive Vocabulary Test (REVT) was also administered whenever possible [21]. Speech therapists administered the PRES, SELSI, and REVT and the results were supervised by a senior speech therapist with more than 10 years of experience. A significant delay was defined as when the language age confirmed by PRES or SELSI was delayed by more than 12 months. Language quotient was calculated ([developmental age/chronological age]×100) to compare language development in children of different ages.
ASD symptoms
The K-CARS was administered by child psychologists to assess the ASD symptoms [22]. The K-CARS consists of 15 items rated on a 7-point scale with scores ranging from 1 to 4 (half points), in which higher total scores indicate higher severity. In addition, the Social Responsiveness Scale (SRS), a parentcompleted 65-item questionnaire, was used to measure the frequency of ASD-related behaviors [23].
Behavioral problems
Behavioral and emotional problems were measured by the Child Behavior Checklist for ages 1.5-5 (CBCL) [24] and the Aberrant Behavior Checklist (ABC) [25]. CBCL is a 99-item scale rated by parents with scores ranging from 0 to 2 for each item. The scores for the following seven subscales were calculated: emotionally reactive, anxious/depressed, somatic complaints, withdrawn, sleep problems, attention problems, and aggressive behaviors. CBCL demonstrated good reliability and validity in clinical and non-clinical populations, and showed good cross-informant agreement [24]. ABC, which is a rating scale used to assess behavioral problems in individuals with DD, was also administered. The ABC consists of 58 items answered on a 4-point scale from 0 to 3 based on problem severity. The items are further categorized into the following five domains: irritability, lethargy/social withdrawal, stereotypic behavior, hyperactivity/noncompliance, and inappropriate speech. The psychometric properties of ABC have been assessed, and the subscales showed high internal consistency, adequate reliability, and established validity [25].
Caregiver stress and depression
Parenting stress and depression were measured using the Parenting Stress Index, 4th edition (PSI) [26] and the Center for Epidemiologic Studies Depression Scale-Revised (CESD-R) [27], respectively. The PSI is a 120-item measure with two domains: 1) the child characteristics domain and 2) the parent characteristics domain. Both domains showed excellent internal consistency, and the PSI displayed predictive validity in studies performed in other countries [26]. The CESD-R consists of 20 items related to depression rated on a 5-point scale based on symptom frequency. CESD-R showed good psychometric properties, including high internal consistency, and consistent convergent and divergent validity [27].
Problematic smartphone use
The Smartphone Overuse Scale (SOS) was completed by parents to assess smartphone overuse in children. The scale was developed by the National Information Society Agency, and consists of 9-items that are rated in a 4-point scale [28]. The items are categorized into three subscales. The self-control failure subscale assesses whether controlling smartphone usage is possible; the salience subscale assesses whether smartphone use is the most important activity; and the problematic consequences assesses whether smartphone use is related to impairment in functioning. Psychometric properties of the SOS were explored in a preliminary survey sample of 201 children, and main survey sample of 2,348 children. Confirmatory factor analysis was conducted for SOS (Tucker-Lewis index=0.91, comparative fit index=0.95, root-mean-square-error of approximation=0.06), and the reliability ranged from 0.58 to 0.75. The receiver operating characteristic analysis was conducted conducted using the internet gaming disorder criteria in Section III of DSM-5 as a golden standard. As a result, a total score of ≥24 in the SOS indicated potential risk (area under the curve [AUC]=0.668, sensitivity=70.4%, and specificity=63.3%), and ≥28 indicated high risk (AUC=0.590, sensitivity=28.7%, and specificity=89.2%).
Statistical analyses
Descriptive statistics were calculated and the results were compared between the DoBrain group and the JNKK group. Psychological test and questionnaire outcomes were compared between the DoBrain group and the JNKK group. First, to estimate the change after app use in each group, paired ttest or Wilcoxon signed-rank test was conducted. Next, to compare the effect of the app between the DoBrain and the JNKK group, t-test or Wilcoxon rank-sum test was conducted. Test for normality was conducted using the Shapiro-Wilk test. Identical statistical analyses were performed to compare the effect of DoBrain in children with and without DD. Subscales were adjusted for multiple comparisons using Bonferroni’s method. Statistical significance was set at p=0.05. All statistical analyses were performed using SAS software, version 9.4 (SAS Institute, Cary, NC, USA) and the R Statistical Software, version 4.0.2 (R Foundation for Statistical Computing, Vienna, Austria).
RESULTS
Of the 200 children who were enrolled, 34 children were lost to post-test and 166 children were included in the final analysis of the study (Figure 1). There were no significant differences between the study children and those who were lost to post-test in terms of age and sex. However, the proportion of children diagnosed with ASD (final analysis group, 25.9% vs. lost-to-post-test group, 52.9%; p=0.002) or ID (final analysis group, 27.7% vs. lost-to-post-test group, 50.0%; p=0.011) was higher in the lost to post-test group. The reason for posttest loss included situations related to COVID-19 (n=17), unable to visit due to personal reasons (n=8), problems with using the app (n=3), concerns related to media exposure (n=1), and unknown (n=4).
The overall demographic characteristics of the study participants are summarized in Table 1. Children in the DoBrain group and the JNKK group did not show significant differences in the age at baseline, sex, diagnosis, number of siblings, birth order, paternal and maternal education, and family income.
Psychological test results were compared between the Do-Brain and JNKK groups (Table 2). Mean change from baseline (Δ) was calculated, and the between-group differences in Δ were assessed. PEP-R domains including imitation (adjusted p=0.049 and p=0.022), perception (adjusted p=0.004 and p<0.001), and gross motor (adjusted p=0.003 and p=0.002) showed significant declines from baseline in both the DoBrain and JNKK groups, but no significant difference in Δ was observed between the two groups. The ABCS of the VABS showed improvement from baseline only in the DoBrain group (adjusted p=0.005), with no significant between-group difference in Δ. The communication subscale of the VABS showed improvement from baseline in both the DoBrain and JNKK groups (adjusted p<0.001 and p=0.006), but no significant difference in Δ was observed between the two groups. In the JNKK group, receptive language assessed by the SELSI showed a significant decline from baseline (adjusted p=0.046), whereas both receptive (adjusted p=0.022) and expressive (adjusted p=0.004) language assessed by the REVT showed improvement from baseline, with no significant between-group differences in Δ. The K-CARS score showed a significant difference in Δ between the two groups (adjusted p=0.045).
Similarly, the questionnaire results were compared between the DoBrain and JNKK groups (Table 2). The SRS total score (adjusted p=0.040) and the aggressive behavior subscale score of the CBCL (adjusted p=0.013) improved from baseline only in the DoBrain group, and there were no significant differences in Δ between the two groups. The irritability subscale of the ABC showed significant improvement from baseline in the DoBrain and JNKK groups (adjusted p=0.001 and p=0.007), with no significant between-group difference in Δ. The lethargy/social withdrawal subscale (adjusted p=0.017) and the hyperactivity/noncompliance subscale (adjusted p=0.005) of the ABC showed significant improvement from baseline only in the DoBrain group, and there were no significant differences in Δ between the two groups. The caregivers of the Do-Brain group reported significant improvement in the child characteristics subscale of PSI (adjusted p=0.012), whereas the caregivers of the JNKK group reported significant improvement of their CESD-R score (adjusted p=0.004), with no significant between-group differences in Δ.
The DoBrain and JNKK groups were compared in a subset of children with DDs (Supplementary Table 1 in the onlineonly Data Supplement). In the JNKK group, the perception domain of PEP-R (adjusted p=0.008) and daily living skills of the VABS (adjusted p=0.020) showed significant decline from baseline. In addition, in the JNKK group, the K-CARS score showed significant aggravation from baseline (adjusted p=0.022), with significant difference in Δ between the two groups (adjusted p=0.005). In the DoBrain group, the irritability, lethargy/social withdrawal, and hyperactivity/noncompliance subscales (adjusted p=0.001, 0.017, and 0.017, respectively) of the ABC showed significant improvement. The caregivers of the JNKK group reported significant improvement of their CESD-R score (adjusted p=0.015).
In parallel, the DoBrain and JNKK groups were compared in a subset of children with normal development (Supplementary Table 2 in the online-only Data Supplement). PEP-R domains including imitation (adjusted p<0.001 and p=0.002), perception (adjusted p<0.001 and p<0.001), and gross motor (adjusted p<0.001 and p<0.001) showed significant declines from baseline in both the DoBrain and JNKK groups. The ABCS of the VABS showed improvement from baseline only in the DoBrain group (adjusted p=0.017), and the communication subscale of the VABS showed improvement from baseline in both the DoBrain and JNKK groups (adjusted p=0.002 and p=0.005). The caregivers of the DoBrain group reported significant improvement of their CESD-R score (adjusted p=0.043).
No significant side effects were reported in the study children. Problematic smartphone or tablet use was monitored by the SOS (Table 2). When the mean change from baseline (Δ) was calculated, the Problematic Consequences subscale showed a significant increase in the DoBrain group (adjusted p=0.012), and the salience subscale showed a significant increase in the JNKK group (adjusted p=0.006). However, when the DoBrain and JNKK groups were compared, no significant differences were observed in the subscales of the SOS.
A subset of children who used DoBrain were selected to compare the efficacy of DoBrain according to the presence of DD (Table 3). The results of the psychological tests and questionnaires were compared; the mean changes from baseline (Δ) were calculated, and differences in Δ were assessed between children with DD and those without. The eye-hand coordination domain of the PEP-R showed significant differences in Δ between children with DD and those without (adjusted p=0.047). Perception (adjusted p<0.001) and fine motor (adjusted p<0.001) domains of the PEP-R showed a significant decline from baseline in children without DD, but no significant difference in Δ was observed between children with DD and those without. The SRS total score showed significant improvement from baseline in children with DD (adjusted p=0.001), and a significant difference in Δ was observed between children with DD and those without (adjusted p=0.001). In children with DD, significant improvements from baseline were observed in irritability (adjusted p=0.001), lethargy/social withdrawal (adjusted p=0.017), and hyperactivity/noncompliance (adjusted p=0.017) subscales of the ABC. Significant differences in Δ between children with DD and those without were observed in irritability (adjusted p=0.006), lethargy/social withdrawal (adjusted p=0.001), and stereotypic behavior (adjusted p=0.008) subscales of the ABC. The salience subscale of the SOS showed a significant increase from baseline in children with DD (adjusted p=0.030).
DISCUSSION
In this study, we found that preschool children who used the DoBrain app for 24 weeks did not show significant differences in overall cognitive development compared with children who used general educational apps such as JNKK. Compared with the JNKK group, the DoBrain group showed significant improvements in the K-CARS assessment for ASD symptoms. When the efficacy of DoBrain was compared according to the presence of DD, children with DD showed greater gains in the eye-hand coordination domain of PEP-R, SRS total score, and irritability, lethargy/social withdrawal, and stereotypic behavior subscales of ABC than did those without DD. Compared to baseline, cognitive domains related to imitation, perception, and gross motor declined in both the DoBrain group and the JNKK group, while SRS total score and behavioral problems as measured by the aggressive behavior subscale of CBCL, lethargy/social withdrawal and hyperactivity/noncompliance domains of ABC, and child characteristics domain of PSI improved only in the DoBrain group.
To our concern, we found that cognitive domains related to imitation, perception, and gross motor declined in the Do-Brain group and the JNKK group alike. Increased screen time may be one possible explanation for this observation. During the study period, smartphones and tablets were regularly used, and problematic screen use assessed by SOS was increased. Excessive screen time negatively affects social interaction [29], physical activity [30], and perception [31], which can result in the decline of the imitation, gross motor, and perception domains of the PEP-R. More screen time can lead to less time engaged in other activities that promote normal development. As such, mobile-based cognitive training is not free from the negative effects of screen use. In the same context, a recent study that used computer-assisted intervention reported that increased program use was associated with worse receptive language outcomes [32]. Further research is necessary to evaluate the detrimental effect of mobile-based cognitive training.
Contrary to our expectations, the effect of DoBrain on overall cognitive development was not superior to general educational apps. This result is in line with other studies that have questioned the effect of cognitive training in children [33,34]. While some studies proposed that cognitive training leads to cognitive improvement [35,36], methodological issues should be taken into account, including the absence of an active control group and appropriate blinding. The strength of this study lies in the use of comparison with general educational apps, Junior Naver and Kakao Kids, which served as an active control. In addition, although double-blinding was not possible due to the nature of this study, the single-blinded design of the study would have minimized the assessor bias. In contrast to previous studies that included older children or adolescents [35-37], our sample included children with a mean age younger than 5 years, and may better represent the efficacy of mobile-based cognitive training in preschool children.
ASD symptoms measured by K-CARS showed a significant improvement in the DoBrain group than in the JNKK group. This difference between two groups was observed in a subset of children with DD, but no difference was observed in children with normal development. Furthermore, ASD symptoms measured by SRS showed a significant decline from baseline in the DoBrain group but not in the JNKK group. When a subgroup of children using DoBrain were assessed separately, the SRS total score showed a significant improvement in children with DD compared with those without. Existing evidence suggests that cognitive training may improve cognitive function, and thus may have beneficial effects on ASD symptoms [9]. However, in our study, the improvement of ASD symptoms was not accompanied by the improvement of cognitive function. Therefore, the improvements in ASD symptom observed in our study require careful interpretation.
Language skills were assessed by PRES, SELSI, and PRES, which yielded heterogeneous results. Compared with baseline, no significant differences were observed in the DoBrain group, whereas the JNKK group showed a significant decline in the SELSI receptive language domain and improvements in REVT receptive and expressive language domains. This inconsistent finding can be explained in part by the different dimensions of language that the two tests assess. SELSI assesses overall language abilities including semantics, phonology, syntax, and pragmatics through parental report, while REVT only assesses vocabulary abilities [20,21]. In this study, receptive and expressive vocabulary improved in the JNKK group, which is consistent with the characteristics of the JNKK app that present diverse vocabularies for practice. However, this improvement was confined to vocabulary. In contrast to the JNKK group, the DoBrain group did not show significant changes in language skills; the DoBrain app is more focused on problemsolving and reasoning, and thus minimal language stimuli are provided, which can be the reason for the overall null effect. In line with previous studies [38,39], our results suggest that a welldesigned software that uses multimodal stimuli may be helpful in improving vocabulary skills.
Children in the DoBrain group showed significant improvements in behavioral problems as measured by the aggressive behavior subscale of CBCL, irritability, lethargy/social withdrawal, and hyperactivity/noncompliance domains of ABC, and child characteristics domain of PSI. In particular, irritability, lethargy/social withdrawal, and hyperactivity/noncompliance domains of ABC showed significant improvement in children with DD, but not in children with normal development. This finding is in line with a recent meta-analysis in which behavioral measures showed significant improvements after using cognitive training programs [36]. Behavioral problems may have improved as a consequence of using DoBrain, considering that neural and behavioral plasticity is greater in younger children [40]. However, it should be taken into account that CBCL, ABC, and PSI are completed by caregivers of the children, and that placebo effects may also have played a part. In addition, children who spent more time using DoBrain may simply have had less time to engage in problematic behaviors.
We used the SOS to monitor problematic smartphone or tablet use, which is a possible side effect of a mobile-based cognitive training program. Compared with baseline, the problematic consequences subscale increased in the DoBrain group, and the salience subscale increased in the JNKK group. Although the total score did not exceed the cutoff point and the increase in the total score was less than one point, a significant increase suggests that monitoring the side effects of screen use is necessary. To the best of our knowledge, no other studies assessed the impact of mobile-based cognitive training programs on problematic screen use. Further studies should be performed while accounting for the changes in total screen time.
Several limitations of this study warrant discussion. First, the participants of the study were informed of which group they were allocated to. Parents, who completed the questionnaires, were aware of the group status, and this may have resulted in bias. In particular, questionnaires such as SRS, CBCL, ABC, PSI, and caregiver CESD-R rely on parental evaluation, and parents in the DoBrain group may report more positive outcomes. However, the JNKK group were active controls for this study, which may result in less placebo effect when compared with passive controls. Second, children with DD were diagnosed with heterogeneous disorders including ASD, ID, and LD, which exhibit distinct clinical characteristics. Therefore, the effect observed in children with DD as a whole may not be applicable to children with each disorder. Third, among the children who were lost to post-test, there were significantly more children who were diagnosed with ASD or ID. Although the majority of the reasons for follow-up loss were related to COVID-19, some of the children had difficulties in continuing the app. Therefore, there may be a bias in the sample included in the final analysis. Fourth, parents were instructed to help the children with the app usage, but the exact extent of parental help was not assessed, which may have an impact on study outcomes. Fifth, the study population was not sufficiently large to represent the general population. Sixth, for assessments that are not standardized for age, the effect of natural developmental trajectory on improvement must be considered. Finally, no passive control group was included in the study.
Despite these limitations, the strength of our study comes from a relatively large sample of children who were included in a randomized trial. The need for mobile-based cognitive training for children with or without DD is rapidly growing, with diverse apps being readily available. Our current results indicate that DoBrain has no superior effect over general educational apps on overall cognitive development. In the domains of cognitive development that are frequently practiced inside the DoBrain app including eye-hand coordination, possible improvement was implicated in a subset of children with DD, which requires careful interpretation. DoBrain was more effective in reducing ASD symptoms compared with the general educational apps, but the effect size was small. Further study with a better-designed, personalized mobilebased cognitive training app is required to verify its effect on cognitive development. In addition, careful monitoring of the detrimental effect of mobile-based cognitive training on specific domains of development is necessary.
PDF Links
PubReader
ePub Link
XML Download
Full text via DOI
Download Citation
Supplement1
Print
