Combination of Two Primary Screening Instruments (K-SCQ and K-SRS-2) and Setting of New Cutoff Values to Improve Diagnostic Accuracy of Autism Spectrum Disorder in Young Children
Article information
Abstract
Objective
This study aimed to explore the diagnostic validity of primary screening instruments (Korean version of the Social Communication Questionnaire [K-SCQ] and Korean version of Social Responsiveness Scale second edition [K-SRS-2]) in Korean children aged 10–60 months and to examine patterns of validity across age. Additionally, we aimed to propose new cutoff values specific to age subgroups.
Methods
The study included 1,326 children (autism spectrum disorder [ASD], n=822, M=41.79, SD=10.28; non-ASD, n=504, M=32.48, SD=10.88) divided by age (10–17, 18–29, 30–41, 42–53, and 54–60 months) who completed the instruments and underwent clinical best-estimate diagnostic evaluation. An optimal screening strategy was sought by calculating sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) based on instrument combinations. Receiver operating characteristic (ROC) analysis was conducted to determine the optimal novel cutoff values for the instruments in each age subgroup.
Results
The validity of individual instruments varied significantly across age subgroups. However, there was some improvement in validity when applying “either K-SCQ or K-SRS-2,” especially in children aged 30 months and older (sensitivity, 83.3%–94.9%; specificity, 58.1%–90.9%; PPV, 21.7%–98.5%; NPV, 65.3%–96.2%). Estimated cutoff for K-SCQ were 13.5, 9.5, 10.5, 7.5, and 9.5 for ages 10–17, 18–29, 30–41, 42–53, and 54-60 months respectively (sensitivity, 82.4%–92.2%; specificity, 74.8%–90.9%). Estimated cutoffs for K-SRS-2 were 58.5, 54.5, 55.5, 55.5, and 52.5 for ages 10–17, 18–29, 30–41, 42–53, and 54–60 months, respectively (sensitivity, 50.0%–94.1%; specificity, 80.3%–97.7%).
Conclusion
In children aged 30 to 60 months, the combination of either K-SCQ or K-SRS-2 allowed for accurate screening of ASD. To further improve accuracy, adjusted cutoff values can be applied based on age subgroups.
INTRODUCTION
Autism spectrum disorder (ASD) is a neurodevelopmental disorder characterized by early deficits in social skills, communication, and repetitive behaviors [1]. These symptoms can greatly affect daily functioning, affecting both children and their caregivers. Early diagnosis and intervention are crucial to improve long-term outcomes [2].
Although research indicates that ASD can be diagnosed as early as 14 months of age [3], the diagnosis often occurs around the age of 4 years in clinical practice worldwide [4,5]. Therefore, clinicians should utilize time-efficient ASD screening instruments to identify at-risk children and initiate early interventions before confirming the diagnosis.
Research has focused on the development of new instruments and strategies to improve early ASD screening. One approach is to develop ASD screening instruments that consider each country’s cultural context as cultural factors can influence the assessment of ASD symptoms [6,7]. Cultural factors can alter the assessment criteria for ASD symptoms [8]. Efforts have emerged to create culturally adapted tools, especially in Asia, to complement the many ASD instruments developed in the U.S. For example, South Korea developed the Behavior Development Screening for Toddlers-Play/Interview (BeDevel-P/I) for young children [9], Hong Kong introduced the Classroom Observation Scale for preschoolers [10], and India developed the Chandigarh Autism Screening Instrument [11]. Although these culturally tailored tools are beneficial, their widespread adoption in clinical practice is time-consuming. The second strategy is to translate and adapt existing instruments, revalidate them for local populations, and adjust the cutoff points. This method saves time and allows for cross-population comparisons. However, adapting instruments can lead to qualitative differences in assessment owing to variations in the original version, sampling, and psychometric properties [7]. Although efforts have been made to identify culturally appropriate cut-off points, few studies have thoroughly validated these adaptations, especially in East Asia.
Even with standardized screening instruments, their validity often varies when applied in clinical settings, which is also true in South Korea. The effectiveness of the instruments can vary depending on the sampling method [7]. In particular, when there is a qualitative difference between the sample in which the instrument’s validity is confirmed and the sample in which the instrument is actually applied, the comprehensibility of the instrument tends to decrease [12]. In South Korea, although the Social Communication Questionnaire (SCQ) and the Social Responsiveness Scale (SRS) have been validated and are commonly used in clinical settings for ASD screening, the use of these instruments alone is limited in terms of accuracy.
This study aimed to explore ways to enhance the diagnostic yield using the Korean version of the SCQ (K-SCQ) and Korean version of SRS second edition (K-SRS-2), which are parent-report screening instruments for ASD that have been validated for reliability and are commonly used in clinical practice in South Korea. Given the potential benefits of such combinations, we verified the screening accuracy of combining these two instruments in various ways. Additionally, as a strategy to further enhance screening accuracy, we divided the children into age groups and determined the optimal cutoff for each age group.
METHODS
Participants
The study enrolled children who participated in ongoing or completed studies, including those that recruited children with ASD and their families for genomic analysis, as well as studies that involved toddlers aged ≥9 months to develop an ASD screening instrument. The inclusion criteria required the child to cooperate to a certain extent with the necessary assessments and the absence of severe medical or neurological conditions, as well as sensory and motor issues.
Children were recruited through multiple channels, including Seoul National University Bundang Hospital, local primary clinics, online and offline bulletin boards of public institutions, and caregiver self-help communities. Written informed consent was obtained from all caregivers and children at the time of data collection. This study was approved by the Institutional Review Board (IRB no. B-2309-851-103) at the authors’ institution. The requirement for written informed consent was waived because of the retrospective analysis of anonymized data from the pooled dataset (IRB no. B-2003-603-301, B-1607-353-005, B-1703-388-303).
Procedures
Data were collected from children aged 10–60 months, whose K-SCQ and K-SRS-2 results were preserved. The children’s age groups were subdivided based on the starting age of each stage of the Korean Developmental Screening Test for Infants and Children. Participants underwent diagnostic evaluations through parent-report questionnaires on ASD as well as the Autism Diagnostic Observation Schedule and Autism Diagnostic Interview-Revised assessments. To ensure interrater reliability, parent interviews and child observational evaluations were recorded and reviewed during scoring to confirm the diagnoses. Additionally, basic developmental history, family history, and clinical information were collected. This information was not disclosed to the evaluators prior to assessment. The best estimated diagnoses were made by child psychiatrists and special education professionals who integrated and clinically assessed the test results using the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition criteria. In this study, the participants were categorized into ASD and non-ASD groups based on their diagnostic outcomes.
Additional information included children’s IQ, Vineland Adaptive Behavior Scales (VABS), and Korean Childhood Autism Rating Scale (K-CARS) scores. BeDevel-P/I scores were collected from children aged 10–42 months.
The K-SCQ “lifetime form” was used in this study. While the established cutoff was 15 points, this study referred to Kim’s research [13] on Korean infants and adolescents and used a cut-off of 10 points. The t-scores of the K-SRS-2 were used to derive screening results, with a cutoff value of 60 points [14]. ASD screening results were evaluated for each screening instrument and their combinations. The first combination method screened for ASD if a child was identified as ASD by either one of the two screening tools, denoted as “either K-SCQ or K-SRS-2.” The second method, denoted as “both K-SCQ and K-SRS-2,” required a child to be screened as ASD by both tools. Consequently, the validity of a total of four screening combinations (K-SCQ alone, K-SRS-2 alone, either K-SCQ or K-SRS-2, both K-SCQ and K-SRS-2) was examined across different age groups.
Additionally, the optimal cutoff points for the age groups were determined for the K-SCQ and K-SRS-2 based on the study population, and differences from existing cutoffs were assessed. Finally, the concordance between the screening results of the two tools using the existing cutoffs and the level 2 screening tool, BeDevel-P/I, was examined.
Measures
K-SCQ
The SCQ is comprises items that assess a child’s communication skills, social interactions, and repetitive behaviors and is commonly used for screening ASD [15]. It consists of 40 questions, and responses are provided as “yes” or “no” by parents, caregivers, or teachers familiar with the child. Higher scores indicate a greater likelihood of developing ASD. The SCQ has two forms: the “current form,” which assesses social communication difficulties over the past few months, and the “lifetime form,” which evaluates developmental history and social skills over the child’s lifetime. In this study, we used the K-SCQ lifetime form, which has been translated and validated for use with Korean individuals [13,16].
K-SRS-2 preschool form
The SRS-2 is a 65-item questionnaire designed to evaluate an individual’s social communication and behavior over the past 6 months [17]. It is commonly used when ASD or social communication disorder is suspected and is completed by caregivers or teachers familiar with the child. There are four forms based on age, with the preschool form designed for children aged 2.5–4.5 years. The questionnaire yielded a raw score, which was converted to a t-score adjusted for sex. The t-score is more commonly used and is reliable for screening and assessing the severity of ASD. Higher t-scores indicated more severe ASD symptoms. In this study, we used the K-SRS-2, which was back-translated and validated with the authors’ approval [18].
BeDevel-P/I
The BeDevel is a level-2 instrument developed for the early screening of ASD, evaluating various aspects of a child’s development, social skills, communication, and motor skills [9]. BeDevel comprises two versions based on the assessment method. BeDevel-P involves the observation and interaction with children in a structured play environment. The BeDevel-I involves interviewing the child’s caregivers to collect information on the child’s developmental milestones and behaviors, supplementing observations from play sessions. This tool was designed for children aged 9–42 months and has been validated for ASD screening accuracy. In this study, data analysis was conducted only for children aged ≤42 months for whom BeDevel results were available.
K-CARS second edition
The CARS is an instrument used to diagnose and assess ASD [19]. It is administered by trained professionals who evaluate a child through direct observation or by compiling information obtained from interviews with caregivers. The scale consists of 15 items that assess various aspects such as social interaction, communication, repetitive behaviors, and sensory sensitivity, resulting in a total score. Higher total scores indicated more severe ASD symptoms. In this study, we used K-CARS-2, which has been standardized for use in Korea [13,20].
VABS 2nd edition
The VABS 2nd edition involves conducting a semi-structured interview with a child’s caregiver to assess the child’s adaptive behavior and living skills [21]. The subdomains of the assessment include communication, daily living skills, socialization, and motor skills. Based on the caregivers’ responses, standard scores were obtained according to the child’s sex and age. Standard scores were calculated with a mean of 100 and a standard deviation of 15, with higher scores indicating higher levels of development and adaptation compared to peers.
Statistical analyses
The t-test was used to identify differences in characteristics between the ASD and non-ASD groups in the demographic analysis. To determine the screening accuracy of each combination of screening instruments, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated for each age group. Additionally, receiver operating characteristic (ROC) analyses were performed to identify the optimal cutoff values for the K-SCQ and K-SRS-2 within each age group, with the cutoff determined at the point where the area under the curve (AUC) was maximized. Finally, Cohen’s kappa was used to assess the agreement between the K-SCQ, K-SRS-2, and BeDevel-P/I. All analyses were conducted using SPSS Statistics 27 (IBM Corp.).
RESULTS
General characteristics of participants
A total of 1,326 children aged 10–60 months, with a mean age of 38.3 months (SD=11.4) were included in this study. The sample consisted of 957 boys (72.2%) and 369 girls (27.8%). Based on the final diagnosis, 822 children (62.2%) were classified as having ASD, whereas 504 (38.0%) were classified as non-ASD. The mean age of the ASD group was 41.8 months (SD=10.28), which was higher than that of the non-ASD group at 32.5 months (SD=10.88).
The age distribution was as follows: 49 children (3.7%) were aged 10–17 months, 265 children (20.0%) were aged 18–29 months, 535 children (40.3%) were aged 30–41 months, 319 children (24.1%) were aged 42–53 months, and 158 children (11.9%) were aged 54–60 months, with the highest number of children in the 30–41 month age group. Table 1 details the proportion of children diagnosed with ASD and non-ASD within each age group, along with sex and the results of the K-SCQ and K-SRS-2.
General participant characteristics (N=1,326)
There was a trend of an increasing prevalence of ASD diagnoses with advancing age. The K-SCQ, K-SRS-2, and K-CARS scores, which indicate the severity of ASD symptoms, were significantly higher in the ASD group than in the non-ASD group across all age ranges. The VABS scores, which reflect adaptive functioning, were significantly higher in the non-ASD group than in the ASD group across all age ranges, except for the 42–53 month range. The IQ was significantly higher in the non-ASD group than in the ASD group in all age ranges, except for the age group of 10–29 months, where there were many missing values.
Accuracy of each screening instruments and combination of the instruments
The accuracy of the combination of screening instruments (K-SCQ alone, K-SRS-2 alone, either K-SCQ or K-SRS-2, both K-SCQ and K-SRS-2) in identifying children with ASD was assessed across different age groups using sensitivity, specificity, PPV, and NPV (Table 2).
Sensitivity, specificity, PPV, and NPV of each instrument combination
The sensitivity of the K-SCQ ranged from 82.4%–86.8% across all age groups; however, its specificity varied widely, from 58.1%–90.9%. For the age groups of 30–60 months, both the sensitivity and specificity were above 80 years, demonstrating excellent validity. The PPV of the K-SCQ ranged from 21.7%–98.3%, and the NPV ranged from 52.6%–96.2%.
The K-SRS-2 had a sensitivity range of 33.3%–89.0% and a specificity range of 88.5%–100%. The K-SRS-2 showed excellent sensitivity and specificity only in the 54–60 month age group (sensitivity, 89.0%; specificity, 95.5%). The PPV of K-SRS-2 ranged from 87.9%–100%, and the NPV ranged from 38.8%–91.5%.
The combination of “either K-SCQ or K-SRS-2” had a sensitivity range of 83.3%–94.9% and a specificity range of 58.1%–90.9%, showing the highest sensitivity across all age groups compared to other combinations. The specificity was excellent in the age groups of 30–60 months (30–41 months, 80.5%; 42–53 months, 80.3%; 54–60 months, 90.9%). The PPV for the “either K-SCQ or K-SRS-2” ranged from 21.7%–98.5%, and the NPV ranged from 65.3%–96.2%.
The combination of “both K-SCQ and K-SRS-2” had a sensitivity range of 33.3%–80.9%, showing the lowest sensitivity across all age groups compared to other combinations. However, it had the highest specificity, ranging from 91.8%–100%, across all age groups. The PPV ranged from 88.5%–100% and the NPV ranged from 40.9%–91.5%.
Adjusted cutoff values for the K-SCQ and K-SRS-2 in each age subgroup
ROC analysis revealed that the AUC for the screening instruments ranged from 0.85–0.96, indicating excellent diagnostic accuracy when applying new cutoffs, except for the K-SRS-2 in the 10–17 month age group (AUC, 0.68).
For the K-SCQ, the most significant deviation from the previously known cutoff (10) was observed in the 10–17 month age group, showing the highest cutoff value across all age groups (cutoff, 13.5). Conversely, the cutoff for the 42–53 month age group was lower than that previously established at 7.5, whereas the other age groups had cutoffs similar to or slightly higher than the previous cutoff (18–29 months, 9.5; 30–41 months, 10.5; 54–60 months, 9.5). When applying the adjusted cutoffs for each age group, the sensitivity of the K-SCQ ranged from 0.82–0.92, and specificity ranged from 0.75–0.91, indicating good performance.
For the K-SRS-2, the optimal cutoff values were lower than the previous cutoff values across all age groups. The 10–17 month age group showed a low AUC, suggesting that this cutoff may not be optimal. The lowest optimal cutoff was observed in the 54–60 month age group (18–29 months, 54.5; 30–41 months, 55.5; 42–53 months, 55.5; 54–60 months, 52.5). When applying the adjusted cutoffs, the sensitivity of the K-SRS-2 ranged from 0.50–0.94, with lower values observed in children under 30 months. Specificity ranged from 0.80–0.98, indicating excellent performance across all age groups (Table 3).
Adjusted cutoff values for the K-SCQ and K-SRS-2 in each age subgroup
Agreement with the level-2 screening instruments: BeDevel-P/I
Based on Cohen’s kappa values, the agreement between the K-SCQ, K-SRS-2, and BeDevel-P/I was satisfactory (Table 4). Owing to the insufficient sample size, agreement could not be assessed for the 10–17 month age group. However, in the 18–60 month age age groups, K-SCQ and K-SRS-2 showed significant agreement with BeDevel-P/I, with adequate k-values. For K-SCQ, the agreement with BeDevel-P in children aged ≥18 months ranged from 75.4%–84.2% (k-value, 0.50–0.64), and with BeDevel-I, the agreement ranged from 77.6%–81.6% (k-value, 0.52–0.62). K-SRS-2 showed an agreement with BeDevel-P of 70.1%–86.8% (k-value, 0.33–0.72) and with BeDevel-I, the agreement ranged from 72.6%–78.9% (k-value, 0.44–0.51) in children aged ≥18 months.
Agreement between BeDevel-P and BeDevel-I
DISCUSSION
This study is the first to attempt to combine two level 1 screening instruments for the early detection of ASD and demonstrated that the combination of two primary screening instruments can help reduce delays in ASD screening in community settings.
In children aged ≥30 months, application of the “either K-SCQ or K-SRS-2” resulted in a sensitivity, specificity, and PPV exceeding 80%, demonstrating high validity. The NPV was not as high (64.5%–81.2%) in this age range, which likely reflects the higher proportion of children with non-ASD children without ASD. Thus, for children aged ≥30 months, it appears reasonable to begin early intervention if they are screened as being at risk for ASD using either of the two screening instruments. Conversely, in children under 30 months of age, combining primary screening instruments resulted in high false positives, limiting their use for reliable ASD determination. For the adjusted cutoffs, except for the 10–17 months age range, both screening instruments showed lower values than the originally recommended cutoffs. The age range with the most significant difference from the original cutoff for the K-SCQ was 42–53 months (cutoff, 7.5), and for the K-SRS-2, it was 54–60 months (cutoff, 52.5).
The findings of this study align with those of previous research, highlighting the need to be cautious of overdiagnosis in younger children and suggesting that a more stable diagnosis of ASD can typically be made after 24 months [22,23]. These findings may be partly explained by the inclusion of younger children in the study sample, some of whom were below the recommended age range for the SCQ and SRS when they were originally developed. The SCQ is recommended for children aged 4 years and older, while the SRS-2 is recommended for children aged ≥2.5 years. Therefore, using only primary screening instruments for children under 30 months of age may be inaccurate and lead to high false positives. Previous studies have also suggested that applying the SCQ to children under the age of four may lead to an increased rate of false positives [24]. Incorporating additional level 2 assessments could be a viable solution to improve screening accuracy and reduce false positives in this age group. Caution is necessary when applying screening instruments to an age range broader than originally intended.
Among the children in this study, the validity of the K-SRS-2 was lower than that of the K-SCQ. Additionally, when examining agreement with the Korean-developed level 2 screening instruments BeDevel-P/I, the K-SCQ consistently showed higher agreement than the K-SRS-2. This discrepancy may be attributed to qualitative differences in how the SCQ and SRS-2 assess ASD or to the varying impacts of sociocultural factors on each instrument in the Korean context. Sociocultural factors, emphasized in previous studies as having a significant influence on ASD diagnosis, could also play a role. These factors not only affect the adaptation process of the instruments but also influence how parents perceive and report their children’s autistic traits through screening items [25,26]. Therefore, when applying screening instruments developed in other countries to a domestic context, it is crucial to consider how well these instruments align with the cultural context of the target country. Additionally, considering that previous studies have shown variability in cutoff scores, the fact that this study also found differences in cutoff scores compared to established values, as well as variations across different age groups, suggests that the interpretation of screening instrument items may be influenced by the cultural context in Korea [7].
This study has several strengths. First, it included a large sample size, and ASD diagnosis for participants was conducted by child psychiatrists and other specialists using standardized diagnostic instruments. Additionally, the study utilized two screening instruments commonly used in the Korean healthcare setting that were easily completed by parents through questionnaires. By examining methods to enhance the diagnostic accuracy of ASD and verifying the validity of these tools, this study provides valuable insights. Aligning the screening process with the existing schedule of Korean early childhood health checkups for children aged ≥30 months is expected to significantly aid in the early diagnosis of ASD.
However, this study has some limitations. A method to improve the screening accuracy for children ≤30 months without increasing the risk of misdiagnosis has not yet been proposed. In particular, the sample size for the 10–17-month age group was insufficient, limiting the statistical power and generalizability of the findings within this age range. Moreover, given the substantially higher proportion of participants aged 30–60 months compared to those under 30 months, and the increased stability of their diagnostic processes and outcomes, caution is warranted when interpreting the diagnostic accuracy findings for the younger age group. Another limitation is the insufficient elucidation of the qualitative differences between the two screening instruments. Previous research has shown that parent-reported screening tools can be effectively developed for children aged 12–36 months with significant validity, indicating a potential area for further research to address the limitations of this study [27]. The Early Screening for Autism and Communication Disorders demonstrated high validity, even in children younger than 24 months, by presenting different cutoff values for different age groups and specific items. Because parent-reported screening instruments are convenient and accessible, further research is needed to explore methods to enhance screening accuracy using primary screening instruments in children under 24 months of age. Overall, while this study demonstrates a strong potential for improving early ASD diagnosis through commonly used screening tools in the Korean clinical context, further investigation is needed to refine screening methods for younger children and explore the qualitative aspects of the tools used.
In summary, for children under 30 months of age, using only the primary screening instrument may increase the likelihood of false positives; therefore, employing a secondary screening instrument in conjunction may reduce false positives and improve accuracy. For children aged 30–60 months, both K-SCQ and K-SRS-2 demonstrated good validity. Combining these two qualitatively different tests and considering a child as having ASD if they screened positive in either test resulted in higher sensitivity without significantly increasing false negatives. To improve the diagnostic accuracy of ASD, it is recommended that different cutoff values be applied according to age group. Future research should include further analyses of the K-SCQ and K-SRS-2 to enhance diagnostic accuracy across a wider age range. Specifically, additional analyses should be conducted on other factors influencing ASD screening (such as sex, age, and developmental level) and item-level analysis.
Notes
Availability of Data and Material
The datasets generated or analyzed during the study are available from the corresponding author upon reasonable request.
Conflicts of Interest
The authors have no potential conflicts of interest to disclose.
Author Contributions
Conceptualization: all authors. Data curation: all authors. Formal analysis: Yoojeong Lee. Funding acquisition: Heejeong Yoo. Investigation: Yoojeong Lee. Methodology: Yoojeong Lee, Guiyoung Bong, Heejeong Yoo. Software: Yoojeong Lee, Da-Yea Song, Heejeong Yoo. Validation: Yoojeong Lee, Guiyoung Bong, Heejeong Yoo. Writing—original draft: Yoojeong Lee. Writing—review & editing: Heejeong Yoo.
Funding Statement
This study was supported by Institute for Information and Communications Technology Promotion (IITP) grant funded by the Korean government (MSIT) (No. RS-2019-II190330).
Acknowledgments
None