A Validation Study of Mental Health Monitoring Through a Mobile Application
Article information
Abstract
Objective
Face-to-face evaluation is the most important in psychiatric evaluation, but smart healthcare, including non-face-to-face evaluation, can be beneficial considering the situation in which face-to-face evaluation is limited or the preventive aspect of mental illness. In this paper, we aimed to check whether mental health screening tests have the same significance as paper-based tests even when collected through mobile applications.
Methods
A smart mental healthcare screening test was conducted on the 1,327 community subjects. We measured two indicators of depression (Patient Health Questionnaire 9-item scale, PHQ-9) and anxiety (Generalized Anxiety Disorder 7-item scale, GAD-7) to check mental health conditions.
Results
The average Cronbach’s alpha value of the PHQ-9 questionnaire was good at 0.870. As a result of PHQ-9’s principal component analysis, one component with an eigenvalue of 1 or more was identified, which is suitable to be described as a single factor. The average Cronbach’s alpha value of the GAD-7 was 0.919. The structural validity of the GAD-7 was confirmed through principal component analysis.
Conclusion
Our results show that PHQ-9 and GAD-7 scales performed through mobile applications can have the same meaning as paper-based tests. Surveys using a tablet PC, or smartphone application can monitor residents’ mental health and accumulate data. Based on these data, smart mental health management can check the mental health of residents and treat mental illness in connection with medical services.
INTRODUCTION
Psychiatric treatment has developed and improved the symptoms of many patients. Increasingly, attention is shifting to preventing diseases and early treatment as healthcare services expand. Screening and monitoring are essential processes in the prevention and treatment of diseases [1]. Unlike other conditions, psychiatric symptoms are challenging to quantify or visually reveal. It is not easy to check the mental health of people who do not visit hospitals because many diagnostic parts rely on face-to-face diagnosis by experts. Self-survey is not available for diagnostic purposes because it is less accurate to diagnose diseases. Still, it can be used to check the possibility of illness before visiting the hospital and nonmedical institutions such as state and local mental health welfare centers.
It is still essential for doctors and patients to meet, observe, and talk directly for accurate diagnosis and effective treatment. But repeated pandemics face the limitations of the classical care system. Most recently, pandemics threatening the world, such as coronavirus disease-19, severe acute respiratory syndrome, and middle east respiratory syndrome, have repeatedly emerged. In situations such as pandemics, where contact between doctors and patients can be minimal, mobile application tests can be helpful in that non-face-to-face evaluation is possible [2].
The trend of modern medicine is moving toward preventing diseases and shifting toward individuals knowing and managing their health conditions. Several researchers also point out that mobile devices can be helpful tools for patients to keep their daily health on their own. The growing number of smartphone users worldwide also shows that mobile devices have the potential to be valuable tools for health care [3-5].
A mobile application can be an attractive alternative in terms of cost-effectiveness without visiting the institution in person. The cost reduction effect is even more remarkable because it is easy to perform the test and collect and analyze data. Therefore, in this paper, we aimed to confirm whether mental health screening tests, which can be the basis for smart mental health care, have the same significance as the paper-based test even when collected through mobile applications [6].
The questionnaires used for psychiatric screening in this study are the Patient Health Questionnaire 9-item scale (PHQ-9) and the Generalized Anxiety Disorder 7-item scale (GAD-7). Depressive and anxiety disorders are the most common psychiatric diseases, and we used these two scales to measure depression and anxiety [7
METHODS
Subjects
Smart mental care screening tests were conducted on the local community population in Busan, Korea, and the subjects were recruited by opening posters. After explaining the purpose and method of this study, we proceeded with written consent. A total of 1,327 subjects participated, with a mean age of 41.47 (±19.15) years recruited. The sample size of this study was 1,327, which is relatively much larger than the sample size of previous papers that collected data using mobile devices [8,9].
Procedure
This study was conducted with the approval of the Institutional Review Board at Pusan National University Hospital (H-2110-011-108). We collected demographic and basic medical data such as age, gender, educational background, psychiatric treatment history, and family history from the subjects. In addition, we measured four indicators of depression (PHQ-9), anxiety (GAD-7), stress (Perceived Stress Scale, PSS), and quality of life (abbreviated version of the World Health Organization Quality of Life, WHOQOL-BREF) to check mental health medical status. All information, including questionnaires, was collected on tablet PCs and mobile phones.
Outcome measures
Depression (PHQ-9)
PHQ-9 is a survey developed by Spitzer et al. [10] in 1999 to help detect and diagnose major depressive disorders in primary clinical scenes. PHQ-9 consists of 9 questions, and each item was created using the diagnostic criteria of Diagnostic and Statistical Manual of Mental Disorders, 4th Edition [11] depression illustration. Each item must respond according to the frequency of symptoms experienced over the past two weeks, from “not at all” to “near every day,” on a 4-point Likert scale. After calculating the total score for each item, 0–4 points can be evaluated as minimal, 5–9 points as mild, 10– 14 points as moderate, 15–19 points as moderate-high, and 20 points or more as high depression [12]. In this study, we used the Korean version of the depression screening tool translated by An et al. [13] The internal consistency and test-retest reliability were 0.95, 0.91, and more than 9 points out of 27 were set as the cutting point for depression (sensitivity: 88.5%, specificity: 94.7%).
Anxiety (GAD-7)
GAD-7 is a screening tool developed to identify individuals with Generalized Anxiety Disorder [14]. It consists of seven questions about common symptoms of anxiety experienced, such as feeling nervous, worrying too much, and having trouble relaxing. Participants are expected to respond on a Likert scale between 0 and 3 points depending on how often they have experienced anxiety symptoms over the past two weeks (0=not at all, 3=nearly every day). This study classified 5–9 out of 21 points as mild, 10–14 points as moderate, and 15 or more points as severe. The Korean version of the GAD-7 has shown adequate internal consistency (α=0.91) and good convergent validity (moderate to strong correlation with the Depression Anxiety and Stress Scale-21 anxiety subscale, r=0.73) [15].
Statistical analysis
Data analysis was performed using IBM SPSS Statistics for Windows (version 22; IBM Corp., Armonk, NY, USA). Demographic and clinical characteristics were summarized using descriptive statistics such as numbers and percentages for categorical variables or means and standard deviations for continuous variables. We analyzed the clinical characteristics among the age groups by analysis of variance. We evaluated the reliability of each questionnaire through Cronbach’s alpha, an internal consistency indicator that assesses the homogeneity of the item. Cronbach’s alpha was interpreted according to the criteria presented by George and Mallery [16]. We confirmed the validity of the questionnaire through principal component factor analysis. Pearson’s correlation analysis with PSS and WHOQOL-BREF was performed to evaluate the concurrent validity of PHQ-9 and GAD-7.
RESULTS
Demographic and clinical characteristics
When the 1,327 subjects were classified by age group, 429 (32.3%) were in their 20s, and 39 (2.9%) were in their 10s. Demographic characteristics, such as gender, history of psychiatric treatment, and family history of psychiatric illness, were analyzed, excluding missing data (n=1,289). Of the 1,289 subjects, 527 (40.9%) were males, and 762 (59.1%) were females. Of the subjects, 82 (6.4%) had experience in psychiatric diagnosis, and 43 (3.3%) had a family history of mental illness (Table 1).
Demographic and clinical variables of participants
The reliability and validity of the questionnaire
Depression (PHQ-9)
The Cronbach’s alpha value of PHQ-9 was 0.870, which was confirmed to be good. We confirmed the structural validity of PHQ-9 through principal component analysis and found one component with an eigenvalue of 1 or more (Table 2). It means that it is appropriate to explain PHQ-9 as one factor, and it is consistent with previous studies [17,18].
Validity and reliability of the Korean version of the PHQ-9 and GAD-7 scales
Anxiety (GAD-7)
The Cronbach’s alpha value, an index of internal consistency of GAD-7, was found to be very good at 0.919.
We confirmed the structural validity of the questionnaire through principal component analysis. We confirmed that 68.5% of the total variation of GAD-7 was described as one factor, and it was appropriate to explain it as one factor as in previous studies (Table 2) [19,20]. Therefore, we can say that GAD-7 shows structural validity.
Results of mental health indicators
Distribution of depression analyzed through the PHQ-9 questionnaire
Table 3 summarizes the depression scores measured through PHQ-9. The distribution of PHQ-9 questionnaire responses showed the highest frequency at 0 and a strong positive skewness. The depression score evaluated with the PHQ-9 questionnaire was 4.58 (±4.92) on average and the average score was the highest in teenagers (7.56±7.82). As a result of classifying the PHQ-9 score by a cutoff of 9, 231 (17.4%) of the total subjects experienced significant depression. Among them, 25 (1.9%) had a high level of depression to the extent that medical intervention should be considered.
Depression score (PHQ-9) according to the age of subjects
Distribution of anxiety analyzed through the GAD-7 questionnaire
We summarized the anxiety scores measured through GAD-7 in Table 4. The distribution of the GAD-7 questionnaire responses also showed the highest allocation at 0, showing positive skewness. The average total score was 2.85 (±3.95), and the average score was highest among teenagers (4.87±6.11). When the GAD-7 score classified severity, 6.6% of the subjects showed moderate to severe anxiety. The proportion of subjects with moderate to severe anxiety was the highest in their teens (15.4%).
Anxiety score (GAD-7) according to the age of subjects
Concurrent validity of PHQ-9 and GAD-7
Pearson’s correlation between the PHQ-9 and the GAD-7 had a strong positive correlation between the two measurements of depression and anxiety (r=0.785, p<0.01). The PHQ-9 and the WHOQOL-BREF showed a moderate negative correlation (r=-0.471, p<0.01). The PHQ-9 and the PSS had a weak positive linear relationship (r=0.151, p<0.01). The GAD-7 and the WHOQOL-BREF also showed a moderate negative correlation (r=-0.432, p<0.01). Lastly, the GAD-7 and the PSS had a weak positive linear relationship (r=0.199, p<0.01).
DISCUSSION
This study attempted to confirm whether the mental health screening test through a mobile application has the same significance as when the same test was performed on paper (faceto-face). Previous face-to-face studies on the psychometric characteristics of the PHQ-9 and the GAD-7 revealed that the internal consistency of the Korean version of the PHQ-9 was 0.79–0.95, and the test-retest reliability was 0.91. The GAD-7 has shown internal consistency of 0.91 and shown good convergent validity [13,15,17]. As a result of our study, we found that both PHQ-9 and GAD-7 showed good internal consistency (Cronbach’s alpha=0.870, 0.919, respecively). When confirmed through principal component analysis, both questionnaires showed good structural validity. We also revealed that as face-to-face evaluation, PHQ-9 and GAD-7 conducted by mobile applications have good reliability and validity.
A study comparing smartphone-based and non-smartphone-based Beck Depression Inventory (BDI) found overall high comparability (intraclass correlation coefficient=0.921; p<0.001). Although the study included a relatively small sample of 173 participants, it has the strength that they used various evaluation tools for validation (non-smartphone-based BDI, clinical rating scale) [21]. The results of this study show that the measurement of mental health through mobile applications has overall high validity and can be a useful monitoring tool.
According to the 2021 National Health Statistics and the 2022 Statistics Korea Youth Statistics, 11.3% of adults experienced depression, and 26.8% of middle and high school students experienced depression [22,23]. The results of national statistics are consistent with the results of this study in that they show the highest proportion in their teens (depression, 41%; anxiety, 15.4%). Teenagers of all ages are the group that can handle mobile devices the most skillfully, so screening and therapeutic intervention of mental health through mobile devices can be more effective. Mental health monitoring through mobile applications has potential positive effects, primarily on youth and subclinical populations [8]. Studies have shown that self-monitoring mobile applications are suited to first-step interventions. It may help young people to gain more information about their mental health and to utilize more adaptive coping strategies [24,25].
Measurement using mobile devices is not always practical. For example, measuring through mobile applications can be convenient and valuable for those skilled in using mobile devices. But it can be cumbersome for those not used to mobile devices, such as the elderly. In addition, it is sometimes difficult to understand the contents of the survey, but it is difficult to provide additional help if they do not perform a face-to-face examination. However, mental health care in the future will broadly collect data from daily life and move toward evaluation and management. Therefore, this study, which evaluates mental health status using the mobile application, which is the basis for this approach, will be an essential cornerstone for future research.
This study has several limitations. First, it is difficult to closely identify the presence or absence of a disease because subjectivity is greatly reflected due to the nature of self-survey. Second, due to the nature of mobile applications, there is more likely to be an insincere response [26]. Although there are these limitations, it will be more useful than the limitation when it is recalled that it has the purpose of screening rather than diagnosis.
All medical fields, including psychiatry, are facing a new phase with the development of digital technology. Many applications for screening and treating mental illness are being developed, but it is still difficult to find applications confirmed for clinical usefulness and safety [4,27]. Therefore, efforts to confirm the usefulness and safety of a new mental health management tool will be continuously needed. Therefore, efforts to ensure the usefulness and safety of a new mental health management tool will be continuously needed.
Notes
Availability of Data and Material
The datasets generated or analyzed during the study are available from the corresponding author on reasonable request.
Conflicts of Interest
The authors have no potential conflicts of interest to disclose.
Author Contributions
Conceptualization: Hwagyu Suh, Eunsoo Moon. Data curation: Kyungwon Kim, Hyun Ju Lim. Formal analysis: Hwagyu Suh, Jeonghyun Park, Hyun Ju Lim. Funding acquisition: Eunsoo Moon. Investigation: Hwagyu Suh, Eunsoo Moon. Methodology: Eunsoo Moon, Je-Min Park, Byung-Dae Lee, Young-Min Lee. Project administration: Eunsoo Moon. Resources: Eunsoo Moon. Software: Hwagyu Suh, Jeonghyun Park, Hyun Ju Lim. Supervision: Je-Min Park, Byung-Dae Lee, Young-Min Lee, Hee-Jeong Jeong. Validation: Kyungwon Kim, Jeonghyun Park. Visualization: Hwagyu Suh. Writing—original draft: Hwagyu Suh. Writing—review & editing: Je-Min Park, Byung-Dae Lee, Young-Min Lee, Hee-Jeong Jeong, Kyungwon Kim.
Funding Statement
This research was financially supported by the National Information Technology Industry Promotion Agency (NIPA) (Grant Number: G0211-21-1090).