A Validation Study of Mental Health Monitoring Through a Mobile Application

Article information

Psychiatry Investig. 2023;20(6):575-580
Publication date (electronic) : 2023 June 21
doi : https://doi.org/10.30773/pi.2023.0047
1Department of Psychiatry, Biomedical Research Institute, Pusan National University Hospital, Busan, Republic of Korea
2Department of Psychiatry, Pusan National University School of Medicine, Yangsan, Republic of Korea
3Department of Psychology, Gyeongsang National University, Jinju, Republic of Korea
Correspondence: Eunsoo Moon, MD, PhD Department of Psychiatry, Pusan National University Hospital, Pusan National University School of Medicine, 179 Gudeok-ro, Seo-gu, Busan 49241, Republic of Korea Tel: +82-51-240-7303, Fax: +82-51-248-3648, E-mail: esmun@hanmail.net
The results of this study were partly presented at the 35th ECNP (European College of Neuropsychopharmacology) congress, Oct 15-18, 2022, Vienna, Austria.
Received 2023 February 16; Revised 2023 March 31; Accepted 2023 April 3.

Abstract

Objective

Face-to-face evaluation is the most important in psychiatric evaluation, but smart healthcare, including non-face-to-face evaluation, can be beneficial considering the situation in which face-to-face evaluation is limited or the preventive aspect of mental illness. In this paper, we aimed to check whether mental health screening tests have the same significance as paper-based tests even when collected through mobile applications.

Methods

A smart mental healthcare screening test was conducted on the 1,327 community subjects. We measured two indicators of depression (Patient Health Questionnaire 9-item scale, PHQ-9) and anxiety (Generalized Anxiety Disorder 7-item scale, GAD-7) to check mental health conditions.

Results

The average Cronbach’s alpha value of the PHQ-9 questionnaire was good at 0.870. As a result of PHQ-9’s principal component analysis, one component with an eigenvalue of 1 or more was identified, which is suitable to be described as a single factor. The average Cronbach’s alpha value of the GAD-7 was 0.919. The structural validity of the GAD-7 was confirmed through principal component analysis.

Conclusion

Our results show that PHQ-9 and GAD-7 scales performed through mobile applications can have the same meaning as paper-based tests. Surveys using a tablet PC, or smartphone application can monitor residents’ mental health and accumulate data. Based on these data, smart mental health management can check the mental health of residents and treat mental illness in connection with medical services.

INTRODUCTION

Psychiatric treatment has developed and improved the symptoms of many patients. Increasingly, attention is shifting to preventing diseases and early treatment as healthcare services expand. Screening and monitoring are essential processes in the prevention and treatment of diseases [1]. Unlike other conditions, psychiatric symptoms are challenging to quantify or visually reveal. It is not easy to check the mental health of people who do not visit hospitals because many diagnostic parts rely on face-to-face diagnosis by experts. Self-survey is not available for diagnostic purposes because it is less accurate to diagnose diseases. Still, it can be used to check the possibility of illness before visiting the hospital and nonmedical institutions such as state and local mental health welfare centers.

It is still essential for doctors and patients to meet, observe, and talk directly for accurate diagnosis and effective treatment. But repeated pandemics face the limitations of the classical care system. Most recently, pandemics threatening the world, such as coronavirus disease-19, severe acute respiratory syndrome, and middle east respiratory syndrome, have repeatedly emerged. In situations such as pandemics, where contact between doctors and patients can be minimal, mobile application tests can be helpful in that non-face-to-face evaluation is possible [2].

The trend of modern medicine is moving toward preventing diseases and shifting toward individuals knowing and managing their health conditions. Several researchers also point out that mobile devices can be helpful tools for patients to keep their daily health on their own. The growing number of smartphone users worldwide also shows that mobile devices have the potential to be valuable tools for health care [3-5].

A mobile application can be an attractive alternative in terms of cost-effectiveness without visiting the institution in person. The cost reduction effect is even more remarkable because it is easy to perform the test and collect and analyze data. Therefore, in this paper, we aimed to confirm whether mental health screening tests, which can be the basis for smart mental health care, have the same significance as the paper-based test even when collected through mobile applications [6].

The questionnaires used for psychiatric screening in this study are the Patient Health Questionnaire 9-item scale (PHQ-9) and the Generalized Anxiety Disorder 7-item scale (GAD-7). Depressive and anxiety disorders are the most common psychiatric diseases, and we used these two scales to measure depression and anxiety [7

METHODS

Subjects

Smart mental care screening tests were conducted on the local community population in Busan, Korea, and the subjects were recruited by opening posters. After explaining the purpose and method of this study, we proceeded with written consent. A total of 1,327 subjects participated, with a mean age of 41.47 (±19.15) years recruited. The sample size of this study was 1,327, which is relatively much larger than the sample size of previous papers that collected data using mobile devices [8,9].

Procedure

This study was conducted with the approval of the Institutional Review Board at Pusan National University Hospital (H-2110-011-108). We collected demographic and basic medical data such as age, gender, educational background, psychiatric treatment history, and family history from the subjects. In addition, we measured four indicators of depression (PHQ-9), anxiety (GAD-7), stress (Perceived Stress Scale, PSS), and quality of life (abbreviated version of the World Health Organization Quality of Life, WHOQOL-BREF) to check mental health medical status. All information, including questionnaires, was collected on tablet PCs and mobile phones.

Outcome measures

Depression (PHQ-9)

PHQ-9 is a survey developed by Spitzer et al. [10] in 1999 to help detect and diagnose major depressive disorders in primary clinical scenes. PHQ-9 consists of 9 questions, and each item was created using the diagnostic criteria of Diagnostic and Statistical Manual of Mental Disorders, 4th Edition [11] depression illustration. Each item must respond according to the frequency of symptoms experienced over the past two weeks, from “not at all” to “near every day,” on a 4-point Likert scale. After calculating the total score for each item, 0–4 points can be evaluated as minimal, 5–9 points as mild, 10– 14 points as moderate, 15–19 points as moderate-high, and 20 points or more as high depression [12]. In this study, we used the Korean version of the depression screening tool translated by An et al. [13] The internal consistency and test-retest reliability were 0.95, 0.91, and more than 9 points out of 27 were set as the cutting point for depression (sensitivity: 88.5%, specificity: 94.7%).

Anxiety (GAD-7)

GAD-7 is a screening tool developed to identify individuals with Generalized Anxiety Disorder [14]. It consists of seven questions about common symptoms of anxiety experienced, such as feeling nervous, worrying too much, and having trouble relaxing. Participants are expected to respond on a Likert scale between 0 and 3 points depending on how often they have experienced anxiety symptoms over the past two weeks (0=not at all, 3=nearly every day). This study classified 5–9 out of 21 points as mild, 10–14 points as moderate, and 15 or more points as severe. The Korean version of the GAD-7 has shown adequate internal consistency (α=0.91) and good convergent validity (moderate to strong correlation with the Depression Anxiety and Stress Scale-21 anxiety subscale, r=0.73) [15].

Statistical analysis

Data analysis was performed using IBM SPSS Statistics for Windows (version 22; IBM Corp., Armonk, NY, USA). Demographic and clinical characteristics were summarized using descriptive statistics such as numbers and percentages for categorical variables or means and standard deviations for continuous variables. We analyzed the clinical characteristics among the age groups by analysis of variance. We evaluated the reliability of each questionnaire through Cronbach’s alpha, an internal consistency indicator that assesses the homogeneity of the item. Cronbach’s alpha was interpreted according to the criteria presented by George and Mallery [16]. We confirmed the validity of the questionnaire through principal component factor analysis. Pearson’s correlation analysis with PSS and WHOQOL-BREF was performed to evaluate the concurrent validity of PHQ-9 and GAD-7.

RESULTS

Demographic and clinical characteristics

When the 1,327 subjects were classified by age group, 429 (32.3%) were in their 20s, and 39 (2.9%) were in their 10s. Demographic characteristics, such as gender, history of psychiatric treatment, and family history of psychiatric illness, were analyzed, excluding missing data (n=1,289). Of the 1,289 subjects, 527 (40.9%) were males, and 762 (59.1%) were females. Of the subjects, 82 (6.4%) had experience in psychiatric diagnosis, and 43 (3.3%) had a family history of mental illness (Table 1).

Demographic and clinical variables of participants

The reliability and validity of the questionnaire

Depression (PHQ-9)

The Cronbach’s alpha value of PHQ-9 was 0.870, which was confirmed to be good. We confirmed the structural validity of PHQ-9 through principal component analysis and found one component with an eigenvalue of 1 or more (Table 2). It means that it is appropriate to explain PHQ-9 as one factor, and it is consistent with previous studies [17,18].

Validity and reliability of the Korean version of the PHQ-9 and GAD-7 scales

Anxiety (GAD-7)

The Cronbach’s alpha value, an index of internal consistency of GAD-7, was found to be very good at 0.919.

We confirmed the structural validity of the questionnaire through principal component analysis. We confirmed that 68.5% of the total variation of GAD-7 was described as one factor, and it was appropriate to explain it as one factor as in previous studies (Table 2) [19,20]. Therefore, we can say that GAD-7 shows structural validity.

Results of mental health indicators

Distribution of depression analyzed through the PHQ-9 questionnaire

Table 3 summarizes the depression scores measured through PHQ-9. The distribution of PHQ-9 questionnaire responses showed the highest frequency at 0 and a strong positive skewness. The depression score evaluated with the PHQ-9 questionnaire was 4.58 (±4.92) on average and the average score was the highest in teenagers (7.56±7.82). As a result of classifying the PHQ-9 score by a cutoff of 9, 231 (17.4%) of the total subjects experienced significant depression. Among them, 25 (1.9%) had a high level of depression to the extent that medical intervention should be considered.

Depression score (PHQ-9) according to the age of subjects

Distribution of anxiety analyzed through the GAD-7 questionnaire

We summarized the anxiety scores measured through GAD-7 in Table 4. The distribution of the GAD-7 questionnaire responses also showed the highest allocation at 0, showing positive skewness. The average total score was 2.85 (±3.95), and the average score was highest among teenagers (4.87±6.11). When the GAD-7 score classified severity, 6.6% of the subjects showed moderate to severe anxiety. The proportion of subjects with moderate to severe anxiety was the highest in their teens (15.4%).

Anxiety score (GAD-7) according to the age of subjects

Concurrent validity of PHQ-9 and GAD-7

Pearson’s correlation between the PHQ-9 and the GAD-7 had a strong positive correlation between the two measurements of depression and anxiety (r=0.785, p<0.01). The PHQ-9 and the WHOQOL-BREF showed a moderate negative correlation (r=-0.471, p<0.01). The PHQ-9 and the PSS had a weak positive linear relationship (r=0.151, p<0.01). The GAD-7 and the WHOQOL-BREF also showed a moderate negative correlation (r=-0.432, p<0.01). Lastly, the GAD-7 and the PSS had a weak positive linear relationship (r=0.199, p<0.01).

DISCUSSION

This study attempted to confirm whether the mental health screening test through a mobile application has the same significance as when the same test was performed on paper (faceto-face). Previous face-to-face studies on the psychometric characteristics of the PHQ-9 and the GAD-7 revealed that the internal consistency of the Korean version of the PHQ-9 was 0.79–0.95, and the test-retest reliability was 0.91. The GAD-7 has shown internal consistency of 0.91 and shown good convergent validity [13,15,17]. As a result of our study, we found that both PHQ-9 and GAD-7 showed good internal consistency (Cronbach’s alpha=0.870, 0.919, respecively). When confirmed through principal component analysis, both questionnaires showed good structural validity. We also revealed that as face-to-face evaluation, PHQ-9 and GAD-7 conducted by mobile applications have good reliability and validity.

A study comparing smartphone-based and non-smartphone-based Beck Depression Inventory (BDI) found overall high comparability (intraclass correlation coefficient=0.921; p<0.001). Although the study included a relatively small sample of 173 participants, it has the strength that they used various evaluation tools for validation (non-smartphone-based BDI, clinical rating scale) [21]. The results of this study show that the measurement of mental health through mobile applications has overall high validity and can be a useful monitoring tool.

According to the 2021 National Health Statistics and the 2022 Statistics Korea Youth Statistics, 11.3% of adults experienced depression, and 26.8% of middle and high school students experienced depression [22,23]. The results of national statistics are consistent with the results of this study in that they show the highest proportion in their teens (depression, 41%; anxiety, 15.4%). Teenagers of all ages are the group that can handle mobile devices the most skillfully, so screening and therapeutic intervention of mental health through mobile devices can be more effective. Mental health monitoring through mobile applications has potential positive effects, primarily on youth and subclinical populations [8]. Studies have shown that self-monitoring mobile applications are suited to first-step interventions. It may help young people to gain more information about their mental health and to utilize more adaptive coping strategies [24,25].

Measurement using mobile devices is not always practical. For example, measuring through mobile applications can be convenient and valuable for those skilled in using mobile devices. But it can be cumbersome for those not used to mobile devices, such as the elderly. In addition, it is sometimes difficult to understand the contents of the survey, but it is difficult to provide additional help if they do not perform a face-to-face examination. However, mental health care in the future will broadly collect data from daily life and move toward evaluation and management. Therefore, this study, which evaluates mental health status using the mobile application, which is the basis for this approach, will be an essential cornerstone for future research.

This study has several limitations. First, it is difficult to closely identify the presence or absence of a disease because subjectivity is greatly reflected due to the nature of self-survey. Second, due to the nature of mobile applications, there is more likely to be an insincere response [26]. Although there are these limitations, it will be more useful than the limitation when it is recalled that it has the purpose of screening rather than diagnosis.

All medical fields, including psychiatry, are facing a new phase with the development of digital technology. Many applications for screening and treating mental illness are being developed, but it is still difficult to find applications confirmed for clinical usefulness and safety [4,27]. Therefore, efforts to confirm the usefulness and safety of a new mental health management tool will be continuously needed. Therefore, efforts to ensure the usefulness and safety of a new mental health management tool will be continuously needed.

Notes

Availability of Data and Material

The datasets generated or analyzed during the study are available from the corresponding author on reasonable request.

Conflicts of Interest

The authors have no potential conflicts of interest to disclose.

Author Contributions

Conceptualization: Hwagyu Suh, Eunsoo Moon. Data curation: Kyungwon Kim, Hyun Ju Lim. Formal analysis: Hwagyu Suh, Jeonghyun Park, Hyun Ju Lim. Funding acquisition: Eunsoo Moon. Investigation: Hwagyu Suh, Eunsoo Moon. Methodology: Eunsoo Moon, Je-Min Park, Byung-Dae Lee, Young-Min Lee. Project administration: Eunsoo Moon. Resources: Eunsoo Moon. Software: Hwagyu Suh, Jeonghyun Park, Hyun Ju Lim. Supervision: Je-Min Park, Byung-Dae Lee, Young-Min Lee, Hee-Jeong Jeong. Validation: Kyungwon Kim, Jeonghyun Park. Visualization: Hwagyu Suh. Writing—original draft: Hwagyu Suh. Writing—review & editing: Je-Min Park, Byung-Dae Lee, Young-Min Lee, Hee-Jeong Jeong, Kyungwon Kim.

Funding Statement

This research was financially supported by the National Information Technology Industry Promotion Agency (NIPA) (Grant Number: G0211-21-1090).

References

1. World Health Organization. Prevention of mental disorders: effective interventions and policy options: summary report Geneva: World Health Organization; 2004.
2. Sharma S, Kumari B, Ali A, Yadav RK, Sharma AK, Sharma KK, et al. Mobile technology: a tool for healthcare and a boon in pandemic. J Family Med Prim Care 2022;11:37–43.
3. Newzoo. Global mobile market report 2021 San Francisco, CA: Newzoo; 2021.
4. Lagan S, Emerson MR, King D, Matwin S, Chan SR, Proctor S, et al. Mental health app evaluation: updating the American Psychiatric Association’s framework through a stakeholder-engaged workshop. Psychiatr Serv 2021;72:1095–1098.
5. Statista. Number of smartphone users worldwide from 2013 to 2028 [Internet]. Available at: https://www.statista.com/forecasts/1143723/smartphone-users-in-the-world. Accessed February 11, 2023.
6. Wang K, Varma DS, Prosperi M. A systematic review of the effectiveness of mobile apps for monitoring and management of mental health symptoms or disorders. J Psychiatr Res 2018;107:73–78.
7. World Health Organization. Depression and other common mental disorders: global health estimates Geneva: World Health Organization; 2017.
8. Dubad M, Winsper C, Meyer C, Livanou M, Marwaha S. A systematic review of the psychometric properties, usability and clinical impacts of mobile mood-monitoring applications in young people. Psychol Med 2018;48:208–228.
9. Girela-Serrano BM, Spiers ADV, Ruotong L, Gangadia S, Toledano MB, Di Simplicio M. Impact of mobile phones and wireless devices use on children and adolescents’ mental health: a systematic review. Eur Child Adolesc Psychiatry 2022 Jun 16. [Epub] https://doi.org/10.1007/s00787-022-02012-8.
10. Spitzer RL, Kroenke K, Williams JBW, ; the Patient Health Questionnaire Primary Care Study Group. Validation and utility of a self-report version of PRIME-MD: the PHQ primary care study. JAMA 1999;282:1737–1744.
11. American Psychiatric Association. DSM-IV-TR Washington, DC: American Psychiatric Association; 2000.
12. Kroenke K. PHQ-9: global uptake of a depression scale. World Psychiatry 2021;20:135–136.
13. An JY, Seo ER, Lim KH, Shin JH, Kim JB. Standardization of the Korean version of screening tool for depression (Patient Health Questionnaire-9, PHQ-9). J Korean Soc Biol Ther Psychiatry 2013;19:47–56.
14. Spitzer RL, Kroenke K, Williams JB, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med 2006;166:1092–1097.
15. Lee B, Kim YE. The psychometric properties of the Generalized Anxiety Disorder scale (GAD-7) among Korean university students. Psychiatr Clin Psychopharmacol 2019;29:864–871.
16. George D, Mallery P. Using SPSS for windows step by step: a simple guide and reference (4th ed) London: Pearson Education; 2003.
17. Shin C, Ko YH, An H, Yoon HK, Han C. Normative data and psychometric properties of the Patient Health Questionnaire-9 in a nationally representative Korean population. BMC Psychiatry 2020;20:194.
18. Dadfar M, Kalibatseva Z, Lester D. Reliability and validity of the Farsi version of the Patient Health Questionnaire-9 (PHQ-9) with Iranian psychiatric outpatients. Trends Psychiatry Psychother 2018;40:144–151.
19. Byrd-Bredbenner C, Eck K, Quick V. GAD-7, GAD-2, and GAD-mini: psychometric properties and norms of university students in the United States. Gen Hosp Psychiatry 2021;69:61–66.
20. Sousa TV, Viveiros V, Chai MV, Vicente FL, Jesus G, Carnot MJ, et al. Reliability and validity of the Portuguese version of the Generalized Anxiety Disorder (GAD-7) scale. Health Qual Life Outcomes 2015;13:50.
21. Goltermann J, Emden D, Leehr EJ, Dohm K, Redlich R, Dannlowski U, et al. Smartphone-based self-reports of depressive symptoms using the remote monitoring application in psychiatry (ReMAP): interformat validation study. JMIR Ment Health 2021;8:e24333.
22. Korea Disease Control and Prevention Agency. Korea national health statistics 2021 Cheongju: Korea Disease Control and Prevention Agency; 2022.
23. Ministry of Gender Equality and Family. The report of the Korean adolescent in 2022. [Internet]; Available at: http://www.mogef.go.kr/nw/rpd/nw_rpd_s001d.do?mid=news405&bbtSn=708572. Accessed March 30, 2023.
24. Dennis ML, Scott CK, Funk RR, Nicholson L. A pilot study to examine the feasibility and potential effectiveness of using smartphones to provide recovery support for adolescents. Subst Abus 2015;36:486–492.
25. Kauer SD, Reid SC, Crooke AH, Khor A, Hearps SJ, Jorm AF, et al. Selfmonitoring using mobile phones in the early stages of adolescent depression: randomized controlled trial. J Med Internet Res 2012;14:e67.
26. Stone AA, Shiffman S, Schwartz JE, Broderick JE, Hufford MR. Patient compliance with paper and electronic diaries. Control Clin Trials 2003;24:182–199.
27. Torous J, Roberts LW. Needed innovation in digital health and smartphone applications for mental health: transparency and trust. JAMA Psychiatry 2017;74:437–438.

Article information Continued

Table 1.

Demographic and clinical variables of participants

Variable Value (N=1,327)*
Age (yr) 41.47±19.15
Gender (N=1,289)
 Male 527 (40.9)
 Female 762 (59.1)
Hx of psychiatric treatment (N=1,289)
 Yes 82 (6.4)
 No 1,207 (93.6)
FHx of psychiatric illness (N=1,289)
 Yes 43 (3.3)
 No/don’t know 1,246 (96.7)
PHQ-9 4.58±4.92
GAD-7 2.85±3.95
PSS 29.10±4.50
WHOQOL-BREF 93.20±17.93

Values are presented as mean±standard deviation or number (%).

*

sample size might be different in each comparison group due to the missing values in categorical variables.

Hx, history; FHx, family history; PHQ-9, Patient Health Questionnaire 9-item scale; GAD-7, Generalized Anxiety Disorder 7-item scale; PSS, Perceived Stress Scale; WHOQOL-BREF, abbreviated version of the World Health Organization Quality of Life

Table 2.

Validity and reliability of the Korean version of the PHQ-9 and GAD-7 scales

Questionnaires
Validity
Reliability
items Factor loading Communality Eigenvalue Variance (%) Cronbach’s alpha
PHQ-9 4.607 51.186 0.870
Item 1 Anhedonia 0.720 0.519
Item 2 Depressed or hopeless 0.797 0.635
Item 3 Sleep changes 0.639 0.408
Item 4 Fatigue or loss of energy 0.699 0.489
Item 5 Appetite changes 0.664 0.441
Item 6 Worthlessness or guilt 0.738 0.545
Item 7 Trouble concentrating on things 0.743 0.551
Item 8 Psychomotor agitation or retardation 0.734 0.539
Item 9 Suicidal ideation 0.693 0.481
GAD-7 4.797 68.531 0.919
Item 1 Felling nervous, anxious, or on edge 0.853 0.728
Item 2 Not being able to stop or control worrying 0.849 0.720
Item 3 Worrying too much 0.816 0.666
Item 4 Trouble relaxing 0.862 0.744
Item 5 Being so restless 0.848 0.719
Item 6 Becoming easily annoyed or irritable 0.752 0.566
Item 7 Feeling afraid as if something awful might happen 0.808 0.654

PHQ-9, Patient Health Questionnaire 9-item scale; GAD-7, Generalized Anxiety Disorder 7-item scale

Table 3.

Depression score (PHQ-9) according to the age of subjects

Age Number PHQ-9
F p
Depression (%)* Mean±SD
10s (ages 10–19 yr) 39 16 (41.0) 7.56±7.82 6.844 <0.001
20s (ages 20–29 yr) 429 90 (21.0) 5.25±4.97
30s (ages 30–39 yr) 199 29 (14.6) 4.63±4.88
40s (ages 40–49 yr) 202 38 (18.8) 4.69±4.88
50s (ages 50–59 yr) 153 18 (11.8) 3.78±4.07
60s (ages 60–69 yr) 119 13 (10.9) 3.32±4.43
70s (ages 70–79 yr) 138 15 (10.9) 3.18±3.77
80+ (ages 80 yr or more) 48 12 (25.0) 5.29±6.19
Total 1,327 231 (17.4) 4.58±4.92

p-values are from ANOVA.

*

number of subjects with depression: PHQ-9, cutoff 9.

SD, standard deviation; PHQ-9, Patient Health Questionnaire 9-item scale

Table 4.

Anxiety score (GAD-7) according to the age of subjects

Age Number GAD-7
F p
Anxiety (%)* Mean±SD
10s (ages 10–19 yr) 39 6 (15.4) 4.87±6.11 5.339 <0.001
20s (ages 20–29 yr) 429 28 (6.5) 3.16±3.87
30s (ages 30–39 yr) 199 18 (9.0) 3.22±4.39
40s (ages 40–49 yr) 202 15 (7.4) 3.08±3.99
50s (ages 50–59 yr) 153 5 (3.3) 2.16±2.94
60s (ages 60–69 yr) 119 7 (5.9) 2.32±3.94
70s (ages 70–79 yr) 138 3 (2.2) 1.67±2.91
80+ (ages 80 yr or more) 48 5 (10.4) 2.81±4.72
Total 1,327 87 (6.6) 2.85±3.95

p-values are from ANOVA.

*

number of subjects with moderate to severe anxiety: GAD-7, cutoff 10.

SD, standard deviation; GAD-7, Generalized Anxiety Disorder 7-item scale