Development and Validation Study of the Internet Overuse Screening Questionnaire
Article information
Abstract
Objective
Concerns over behavioral and emotional problems caused by excessive internet usage have been developed. This study intended to develop and a standardize questionnaire that can efficiently identify at-risk internet users through their internet usage habits.
Methods
Participants (n=158) were recruited at six I-will-centers located in Seoul, South Korea. From the initial 36 questionnaire item pool, 28 preliminary items were selected through expert evaluation and panel discussions. The construct validity, internal consistency, and concurrent validity were examined. We also conducted Receiver Operating Curve (ROC) analysis to assess diagnostic ability of the Internet Overuse Screening-Questionnaire (IOS-Q).
Results
The exploratory factor analysis yielded a five factor structure. Four factors with 17 items remained after items that had unclear factor loading were removed. The Cronbach’s alpha for the IOS-Q total score was 0.91, and test-retest reliability was 0.72. The correlation between Young’s internet addiction scale and K-scale supported concurrent validity. ROC analysis showed that the IOS-Q has superior diagnostic ability with the Area Under the Curve of 0.87. At the cut-off point of 25.5, the sensitivity was 0.93 and specificity was 0.86.
Conclusion
Overall, this study supports the use of IOS-Q for internet addiction research and for screening high-risk individuals.
INTRODUCTION
Since the beginning of the 1990s, the evolution of information technology (IT) has spawned concerns about technological addiction [1], particularly concerning internet addiction [2]. However, the operational definition of internet addiction remains contentious. Whether internet addiction is even real is still debatable. While Young [2] described the concept of internet addiction, Griffiths [3] argued that the internet is just the place where people engage in specific behavior. That is, excessive users use the internet as a medium to fuel other addictions while not being addicted to the internet itself [4,5]. A Second point of contention concerns the diagnostic criteria of internet addiction, that is what conditions should be satisfied for internet addiction diagnosis [6,7]. For example, although Tao et al. [8] and Griffiths [9] indicated that tolerance is needed to identify addictive behavior, some researchers have argued that it is difficult to objectively define or measure tolerance for internet addiction, and there is a lack of grounds of tolerance even for substance use disorders [10]. Also, it is still unclear whether internet addiction is an isolated disease entity or whether it is a manifestation/subset of other underlying mental disease such as depressive disorder [11,12], anxiety disorder [13], social phobia [14], attention-deficit hyperactivity disorder [15], or impulse control disorder [16,17].
To clarify concepts and diagnoses of internet addiction, there have been attempts to explore the causes of the phenomenon. Some researchers argued that internet addiction can be resulted as a response or coping strategy to stressful event [18,19], since the addictive behavior tends to be triggered when one cannot achieve a sufficient satisfaction from natural rewards [20]. Another study focused on individual sensation seeking tendency or impulsivity based on the unique feature of the internet that enables immediate satisfaction with minimal delay [19]. Other studies suggested that demographic factors like gender [6], education [21], or socioeconomic status could be risk factors for internet addiction [22,23]. Other underlying mental disorders, personality traits, or low self-esteem might also lead to internet addiction [24,25]. Recent studies have suggested that similar to other substance use disorders, internet addiction is likely to be related to neurobiological abnormalities [10] or dysfunctions of dopaminergic brain systems [26]. However, these studies seem to offer explanations only about vulnerabilities or the pathway to the disease state.
The reason why internet addiction studies are complicated is because internet addiction presents unique characteristics and results. For internet addiction, direct physical effects of substances do not affect the brain. However, sleep patterns are often disrupted by the extended nocturnal overuse of the internet. Snoring, teeth grinding, and sleep apnea can be prevalent in individuals considered to be internet addicts [27]. This leads to fatigue during daytime, reduces academic and occupational functioning, and negatively affects physical homeostasis and immune system function [2,28]. The case of a South Korean male who died due to excessive internet gaming after a few sleepless nights has caused awareness about the serious results of internet addiction [29]. Also, internet addiction is likely to cause problems with family or friends, which can lead to secondary psychological and emotional problems [2].
With concerns over the negative consequences of excessive internet use, many studies have tried to explore the current status of internet addiction by developing appropriate screening or diagnostic tools. However, the tools devised so far have failed to reach agreement. Criticisms of the tools include differences in the criteria between tools and lack of appropriate standardization process [17,30]. Moreover, research on screening tools for at-risk internet users is insufficient with a few exceptions including the Internet Addition Test (IAT) [2] or Chinese Internet Addiction Scale (CIAS) [31]. Even though the IAT and CIAS are widely used, these tools also have limitations. Since the IAT was intended for adults, it may not be appropriate to measure the problem of youth. The revised Korean version for adolescent was suggested [32], but the translation and modification process was not presented precisely. Also, the study did not provide diagnostic information such as cut-off point because the study was conducted with community-dwelling students without diagnostic interview. Although Young (1998) suggested cut-off point, it would be inappropriate to use it without considering the cultural differences. For CIAS, the cut-off point was well investigated [29]. The study, however, did not suggest factor structure or concurrent validity of the instrument. Also, CIAS has largely been used in the Taiwan and was not standardized in Korea. Although the Korean Scale for Internet Addiction (K-scale) also has been widely used in Korea [33], it contains some ambiguous expressions that make it difficult to interpret results. For example, item 2 for adolescents (There are more people who recognize me online than offline) or adults (I become more confident during using the internet) do not seem to reflect internet addiction properly.
Since the internet is essential for academic or occupational functioning in modern society, it is difficult to determine between essential-use and overuse with only a few brief questions. In addition, the problem is easily concealed if the subject denies the symptoms. Therefore, questions that reflect various aspects of lifestyle and fulfill an unmet need of the filed are needed. Of all internet users, an estimated 3% to 15% might be considered addicted, with those at high-risk of becoming addicted estimated as 10% to 40% [7,17,18,34]. Internet addiction is likely to become a serious public health issue in the near future, considering that even young children use the internet these days and it can easily be accessed anytime and anywhere through smartphones. Although internet addiction was not included in the 5th edition of the Diagnostic and Statistical Manual of Mental Disorders [35], internet gaming disorder was included in the section ‘condition for further study’. In this situation, a systematically developed and validated screening tool could assist in identifying current status, preventative intervention, and countermeasures.
METHODS
Scale development
A preliminary questionnaire was compiled through a comprehensive review of published research papers about internet addiction, interview material with addiction center visitors, and diagnostic criteria for both pathological gambling and substance use disorders. Next, an expert group consisting of a psychiatrist, psychologist, social worker, and sociologist determined 36 preliminary items after a series of discussions to refine the contexts. For the questions, 50 members of an addiction society rated the adequacy and importance of each item using a 5-point (1–5) Likert scale.
The average score rated by 50 experts was 3.94 for adequacy and 3.90 for importance. Based on the results of the assessment, an item was removed if any of the adequacy or importance scores were below 3.6 (items 7, 12, 18, 25, 28, and 33). Next, if adequacy or importance were below the mean, the expert panel discussed whether the item needed to be included in the final questionnaire considering its clinical value. Items 1 and 4 were deleted because they were similar to items 2 and 5, respectively. Items 3 and 8 were included because they were considered to have high validity for preoccupation to internet. Items 6, 9, 13, and 15 were included because these were the only items representing academic neglect, tolerance, emotional relief, and attempt to conceal a problem respectively. In addition, items 27, 29, and 32 were included because these questions suggested interpersonal problems caused by excessive internet usage. Item 35, which indicates willingness to change, was considered to have predictive value for therapeutic prognosis. Item 17 was included because it reflects evaluation from others. Finally, 28 questions were confirmed (Table 1).
Preliminary item development, and adequacy and importance rated by 50 addiction experts
Participants
The subjects were recruited through six ‘I-will’ centers located in Seoul, South Korea. The centers are operated for the management and intervention for internet, game, or smartphone addiction under the management of Seoul city. Most visitors are teenagers who attended at the request of family members or schools, although some visit voluntarily. The visitors of the center were given the full explanation of the study, and 158 people voluntarily signed the consent form. They included 88 men and 70 women, with an average age of 22.12 (SD=7.56) years. For all participants, the psychologist at each center collected information on the reason for referral or voluntary visit, and explored internet usage habits in a face-to-face interview. Based on the interview material, a psychiatrist and two clinical psychologists evaluated whether a person could be classified as an problematic internet user or not, through overall assessment of internet usage time, dependence, tolerance, withdrawal symptoms, perceived controllability, subjective discomfort, and functional impairment. The Samsung Medical Center Institutional Review Board approved the study, and all participants were paid $30 for their participation (IRB No. 2014-08-114-016).
Measures
Internet addiction screening-Questionnaire (IOS-Q)
Through the aforementioned procedure, the IOS-Q was finalized. The questionnaire was designed to help clinicians to explore the internet usage habits of respondents and identify at-risk internet addiction individuals. Each respondent was asked to evaluate the frequency of the statements on a four-point scale (Not at all, Sometimes, Often, or Always).
Young’s internet addiction scale (IAT)
To verify the concurrent validity of the SOS-Q, the IAT was used. The IAT is the most frequently used measure of internet addiction. We used a Korean translated version contained in a mental health screening survey scale book published by Seoul Child, Adolescent Mental Health Center [36]. Subjects rated their internet usage habits on a 5-point Likert scale. The psychometric properties of the IAT have been verified in South Korea [32]. Cronbach’s alpha for this study sample was 0.95.
Korean Scale for internet addiction (K-Scale)
We used the short-form, 15-item K-Scale for children, adolescents, and adults [33]. The first K-Scale was developed by the Korea National Information Society Agency and Seoul National University in 2002 to measure internet addiction in adolescents [37]. It is composed of 40 questions revised from Young’s 20 item Scale. The K-Scale for adults was published in 2005 [38]. The measure requires individuals to rate on a 4-point scale about their internet usage habits. Based on a total score, users are classified as high-risk, potential risk, or general user groups. Cronbach’s alpha coefficient was 0.78 in this study.
Smartphone scale for smartphone addiction (S-Scale)
Based on the K-Scale, the S-Scale was developed to measure smartphone addiction. The term ‘internet’ was changed to ‘smartphones’. The S-Scale consists of 15 items, on a 4-point scale like the K-Scale. Respondents are categorized according to the total score. It has adequate reliability and validity [39]. Cronbach’s alpha coefficient was 0.81 in our sample.
Statistical analyses
The Statistical Package for Social Sciences version 21 (SPSS ver. 21; IBM Corp., Armonk, NY, USA) was used for analyzing group statistics, internal consistency, inter-item correlation, test-retest reliability, and concurrent validity. T-test and chi-squared test were used to compare the results between addicts and non-addicts. To examine the factor structure of IOS-Q, we conducted Exploratory Factor Analysis (EFA) using Comprehensive Exploratory Factor Analysis (CEFA) Version 3.04 [40]. The factor analysis method used maximum likelihood extraction with oblique direct Quartimin rotation. Oblique rotation was used because correlations between factors were expected. Also, diagnostic ability of the IOS-Q was assessed by investigating sensitivity, specificity of each cut-off score using Receiver Operating Curve (ROC) analysis. If the value of Area Under the Curve (AUC) is <0.5, it is considered random guess, 0.7–0.8 is acceptable, and ≥0.8 is considered excellent [41].
RESULTS
Demographics
Table 2 presents the basic demographic information and internet usage habits by the addiction/non-addiction groups. Differences in age, gender, and internet usage habits among addicts and non-addicts were significant. In addition, the internet addiction group highly assessed the internet’s influence on academic/occupational and household function. For internet addicts, the main purpose of the internet usage was social media, internet games, web-surfing, watching broadcasting, comics, shopping, and blogging in descending order of frequency. On the other hand, the non-addiction group used internet mainly for web-surfing, social networking, watching broadcasting events, and to play internet games.
Descriptive statistics of basic characteristics and internet usage habits of participants
Construct validity
The significance of Bartlett’s test of sphericity [χ2 (df=378)=2453.23, p<0.001] and the Kaiser-Meyer-Olkin (KMO) results (KMO=0.094) suggested that factor analysis was appropriate for these data. The Kaiser criterion [42], scree test [43], and the root mean square of approximation (RMSEA) [44] was used to determine the appropriate number of factors. The Kaiser criterion, which assesses the number of factors with eigenvalues >1.0, suggested 6-factors. The drop rate of the scree plot suggested a 5- or 6-factor structure. According to these results, 4- to 6-factor models were examined. The RMSEA of the 4-, 5-, and 6-factor model was 0.080 [90% confidence interval (CI): 0.070–0.089], 0.073 (90% CI: 0.063–0.084), and 0.062 (90% CI: 0.050–0.073), respectively, which demonstrated that the 5- or 6-factor model would be reasonable (i.e. RMSEA <0.08) [44]. Although the RMSEA value of the 6-factor model was greater, the 5-factor structure was selected since 4 items of 6-factor structure showed under-factoring, in which the factor loading was <0.30.
After accepting the 5-factor solution, each item’s factor loading was examined (Table 3). In order to clarify the factor structure, items that had factor loading less than 0.3 (item 3) and items that showed cross loading (factor loading >0.3 to more than 2 factors; items 2, 4, 6, 11, 12, 14, 17, 18, 24, and 26) were deleted. As a result, total of 17 items remained, and factor 3 was removed because there were no items that remained. Items that belonged to each factors are: Factor 1=Items 1, 13, 20, and 23, Factor 2=Items 16 and 25, Factor 4=5, 7, 8, 9, 10, 15, 27, and 28, and Factor 5=19, 21, and 22). Each factor was named loss of control, preoccupation, craving, and neglect of other areas, respectively. All the sub-factor scores were higher in the addiction group compared to the non-addiction group at the p<0.001 level.
Quartimin rotated factor loadings of preliminary 28 items
Internal consistency and test-retest reliability
Coefficient alphas and corrected item-total correlations were computed for the IOS-Q. The Cronbach’s alpha was 0.91 for total 17 items and the corrected item-total correlations ranged from 0.34 to 0.78. Cronbach’s alpha if item deleted were 0.90 for all items. The Cronbach’s alpha for sub-factors are: Loss of control=0.79, Preoccupation=0.54, Craving=0.86, and Neglect of other areas: 0.76. The correlations between items of the IOS-Q ranged from 0.07 to 0.71.
The test-retest correlation was calculated with the exception of 30 cases which missed the date of second visit. The test-retest reliability for an average time lapse of 10.29 days was 0.72 (p<0.001).
Concurrent validity
Inter-correlations between IOS-Q and other self-reported measures are presented in Table 4. Although correlations between all the measures were significant, the correlation between IOS-Q and the internet addiction related measures, IAT and K-Scale, were higher compared to correlations between the IOS-Q and S-Scale. Also, all the sub-factors of IOS-Q showed higher correlations with IAT and K-Scale compared to S-Scale. These results support convergent and discriminant validity of the IOS-Q.
Correlations between other related measures and IOS-Q total and sub-factor scores
ROC analysis
Two clinical psychologists and one psychiatrist independently reviewed the interview material of all participants of I-will center and decided normal or problematic user. The diagnostic concordance rate was 94.94% among raters, and the discrepancy occurred when functional impairment, self-controllability, or excessive use was unclear. In these cases, the individual was assigned a group in which two or more experts agreed after case discussion. As a result, 28 out of 158 people were classified as internet addicts. Figure 1 shows the results of the ROC analysis of the final 17 items. The AUC value was 0.87 (95% CI: 0.80–0.95), and the optimal cut-off score was 25.5 considering the sensitivity and the specificity determined by Youden’s J statistic. The point determined by the Youden’s index represents the furthest point from the diagonal line, where the sum of the sensitivity and specificity can be maximized [45]. At the cut-off point of 25.5, the sensitivity and the specificity was 0.93 and 0.76, respectively. Results of ROC analysis of sub-factors are presented in Table 5.
Receiver Operating Characteristic (ROC) Curve analysis of the 17-item IOS-Q. IOS-Q: Internet Overuse Screening Questionnaire, Sn: sensitivity, Sp: specificity, J: Youden’s J Statistics.
Results of Receiver Operating Curve Analysis of IOS-Q
DISCUSSION
This study identified the developmental process of the IOS-Q and explored the psychological properties of the scale. Internal reliability was superior, except for one sub-factor (preoccupation). Inter-item correlation and test-retest reliability were adequate. Moreover, the correlations between the IOS-Q and the internet-related measures supported concurrent validity of the scale. EFA revealed a 5-factor structure, with all sub-factor scores being higher in the addiction group compared to the non-addiction group, supporting content validity. ROC analysis revealed that addiction and non-addiction group can be effectively distinguished based on a cut-off score of 25.5. Regarding the result of ROC analysis, it should be noted that the cut-off score and the total mean score were almost similar. Given that it is usually considered abnormal from 1.5 standard deviation above the mean based on normal distribution, it is likely that the cut-off score of this study was somewhat lower. This could be due the under-reporting of problems by the affected individuals. Since internet addicts often underreport their symptoms, this cut-off score is plausible for screening purposes, assuring adequate sensitivity of the instrument.
The average daily internet usage time was <4 hours even in those classified as internet addicted. Considering that initial study of internet addiction defined internet addiction as ‘use of internet more than 38 hours per week’ [19], this seems less than expected. Similar results have been reported in several previous studies. For example, in a review study by Dowling and Quirk [46], there were no significant differences in internet usage time and psychological distress between internet addicts and non-addicts. The authors discussed the possibility of limited internet use due to management of parents or schools. Although the influence of the internet to academic/occupational and household function was highly estimated by internet addiction group compared to non-addiction group in our sample, it was normal range on a Likert scale of 1 to 10. Internet over-users might have thought that there is no problem with that amount of usage time. Also, there were cases in the sample complaining of craving symptoms even though actual usage time was average, or who cannot remember their own usage time exactly. Given this, the intent of the IOS-Q to identify problematic internet users through various internet usage habits seems reasonable.
There are several limitations in this study. Although environmental factors such as social economic status might be related to internet addiction [18,21], we could not examine these variables since most of our samples were single students. In addition, the question whether internet addiction is an isolated disease entity or related to other comorbid mental disorders cannot be answered because we did not carry out a structured clinical interview. There are other limitations related to methodological issues. First of all, it is likely that the respondents were disingenuous or insincere because all the measures were self-report. Also, due to the fact that the participants were recruited through the I-will center, there is a possibility that the sample have been biased. Finally, given the number of items included in the factor analysis (n=28), a sample of 158 participants might not be able to provide sufficient power to support the analysis. The 5-factor structure revealed in this study should be re-examined with confirmatory factor analysis in the future study.
Despite these limitations, the IOS-Q underwent a systematical development and standardization process: i.e. item rating by 50 addiction experts, and group classification and case discussion by independent clinicians. The overall psychological properties of the questionnaire turned out to be favorable. Research on internet addiction is still in development, and this instrument would be worthwhile to identify the status and characteristics of the phenomenon. However, in the absence of consensus on the definition of internet addiction, it should be noted that we cannot solely rely on this instrument for screening or diagnosis purposes. The IOS-Q should serve as the basis for in-depth discussions.
Acknowledgements
This study was supported by a grant of the Korean Mental Health Technology R&D Project, Ministry of Health & Welfare, Republic of Korea (No. HM14C2567; PI, HJJ). This work was partly supported by Institute for Information & communications Technology Promotion (IITP) grant funded by the Korea government (MSIP) (No. B0132-15-1003, the development of skin adhesive patches for the monitoring and prediction of mental disorders), and by the Original Technology Research Program for Brain Science through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology (No. NRF-2016M3C7A1947307; PI HJJ), and the Bio & Medical Technology Development Program of the NRF funded by the Korean government, MSIP (No. NRF-2017M3A9F1027323; PI HJJ).