Validation of the Korean version of the 16-Item Prodromal Questionnaire in a Non-Help-Seeking College Population

Article information

Psychiatry Investig. 2018;15(2):111-117
Publication date (electronic) : 2017 November 29
doi :
1Mindlink, Gwangju Bukgu Community Mental Health Center, Gwangju, Republic of Korea
2Department of Psychiatry, Chonnam National University Medical School, Gwangju, Republic of Korea
3Gwangju Mental Health Commission, Gwangju, Republic of Korea
4Department of Psychiatry, Chonbuk National University Medical School, Jeonju, Republic of Korea
5Department of Psychology, Chonnam National University, Gwangju, Republic of Korea
Correspondence: Jin-Sang Yoon, MD, PhD Department of Psychiatry, Chonnam National University Medical School, 160 Baekseo-ro, Dong-gu, Gwangju 61469, Republic of Korea Tel: +82-62-220-6142, Fax: +82-62-225-2351, E-mail:
Received 2017 March 12; Accepted 2017 April 24.



To examine the reliability and validity of the Korean version of the 16-item Prodromal Questionnaire (KPQ-16) in non-help-seeking university students.


Among 2,246 university students participated in the initial screening, 257 subjects with KPQ-16 scores ≥4 were interviewed. The criteria for ultra-high risk (UHR) of psychosis of the Comprehensive Assessment of At-Risk Mental States (CAARMS) were the gold standard for diagnosis. An exploratory modified version of the questionnaire (mKPQ-16), to which three items from the Eppendorf Schizophrenia Inventory were added, was also used to compensate for items on thought and cognitive problems.


Seventeen subjects met the CAARMS criteria for UHR of psychosis. The area under the receiver operating characteristic curve (AUROC) was highest for the mKPQ-16 total score (AUROC=0.831, p<0.001). The use of cutoff total scores of 7 for the mKPQ-16 and 6 for the KPQ-16 resulted in the best balance of sensitivity (76.5% and 64.7%, respectively) and specificity (75.4% and 71.2%, respectively).


The Korean versions of the PQ-16 are good instruments for screening for psychosis risk in university students. This validation of a questionnaire version with a small number of items may make it feasible to screen large numbers of young adults in the community.


Schizophrenia is a severe mental illness that typically begins in adolescence or early adulthood. Intervention during the early stages of the disorder can reduce its ultimate severity [1]. However, individuals’ first contact with mental health professionals and psychiatric treatment tend to be delayed, even when they develop symptoms of psychosis [2,3]. Treatment delay and a long duration of untreated psychosis are associated strongly with a poor treatment response and outcomes [4,5]. The reported duration of untreated psychosis in Korea is longer than those in Western countries [6,7]. Therefore, a strategy for the early identification of psychosis in Korean young adults and adolescents is urgently needed. Recently, an effective strategy for the prevention of psychosis development from a prodromal state was reported [8,9].

The retrospective concept of “prodrome” was replaced with the terms “ultra-high risk” (UHR) and “clinical high risk” (CHR) of psychosis to emphasize the prospective nature of the evaluation and treatment of patients with early signs of illness [10]. Research findings regarding the pathogenesis and course of UHR of psychosis have also accumulated [11-13]. Three UHR/CHR criteria are in use: 1) attenuated psychotic symptoms (subthreshold, attenuated positive psychotic symptoms), 2) brief limited intermittent psychotic symptoms (short episodes of frank psychotic symptoms that resolve without treatment), and 3) trait vulnerability (schizotypal personality disorder or a first-degree relative with psychosis) in addition to a marked decline in psychosocial functioning or chronic low functioning [10,13]. The Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5), includes “attenuated psychosis syndrome” (APS) as a new diagnosis in Section 3, which describes conditions requiring further research [14]. This new diagnosis has been developed based on evidence that treatment and intervention in these UHR or CHR states can prevent or delay the onset of psychosis [13]. However, many individuals with subthreshold symptoms do not seek mental health services early. Therefore, timely screening for and identification of APS in young individuals in community and school settings are necessary.

Two structured interview tools are used widely to identify individuals at UHR of psychosis: the Comprehensive Assessment of At-Risk Mental States (CAARMS) [15,16] and the Structured Interview for Prodromal Syndromes [17]. However, these structured face-to-face diagnostic interview tools are unsuitable for the screening and identification of UHR individuals in the general population; for those who do not seek specialized help; and for mental health gatekeepers who may not have sufficient time or expertise to complete structured interviews [18,19]. Therefore, brief self-reported screening instruments that are suitable for initial pre-diagnostic filtering of the risk of psychosis before entering into structured in-depth diagnostic evaluation are needed [18,19]. To increase the efficiency of identifying increased psychosis risk, a preferable strategy is a stepwise process, with initial screening via self-report and subsequent detailed assessment using a structured interview. Effective screening instruments should be reliable and valid, as well as brief, simple, and easily applicable [18].

The 16-item version of the Prodromal Questionnaire (PQ-16) is a self-report measure used widely to screen for UHR of psychosis in the general population [20]. The PQ-16 was derived from the original 92-item Prodromal Questionnaire [21], which takes approximately 20 minutes to complete, with the number of items reduced for easier administration in large populations. A European study demonstrated that the PQ-16 had good psychometric properties and validity in comparison with the original PQ [20]. In Korea, the Eppendorf Schizophrenia Inventory (ESI) [22] has been validated and used for the identification of individuals at UHR of psychosis [19]. The ESI was originally developed to assess subjective symptoms that are predominant in psychotic patients. It consists of 40 items, of which 34 are combined into four scales and 6 are used to evaluate frankness and general survey motivation. Although the ESI is a valid instrument for the screening of psychosis risk, a briefer, simpler measure is needed for mass screening. Therefore, this study examined the reliability and validity of the Korean version of the 16-item Prodromal Questionnaire (KPQ-16) in non-help-seeking university students.


Subjects and procedures

This study had two stages: initial screening with the KPQ-16 and the performance of semi-structured interviews to investigate the instrument’s diagnostic validity. Freshman students aged ≥18 years completed the KPQ-16 as part of a packet of anonymous questionnaires at the beginning of the school year. After the study purpose and procedure were explained to first-year students (n=2,921), 2,246 (76.9%) students participated voluntarily in the initial screening. Based on the initial validation study conducted in Europe [20], individuals with scores ≥4 [97% sensitivity, 99% negative predictive value (NPV)] were selected for semi-structured interviews because we believed that the risk of missing true-positive subjects should be very low. In the screening test, 518 (23.1%) subjects had KPQ-16 scores ≥4. Our method was in line with those of previous validation studies, in which people with PQ-positive symptom scores in the top 20% of the distribution were selected for further investigation [23]. Exclusion criteria were age >25 years, a diagnosis of psychotic disorder, current or previous use of antipsychotic medication, and insufficient fluency in the Korean language.

Subjects with KPQ-16 scores ≥4 were assessed with the CAARMS to investigate whether they truly had UHR symptoms. Of the 518 subjects with such scores, 361 (70.0%) voluntarily gave us contact phone numbers. One subject was excluded from this analysis because he was older than 25 years. A trained clinical psychologist, psychiatric nurse, or psychiatric social worker first contacted each subject by telephone for an in-depth interview and interviewed him or her using the CAARMS. Of the 518 subjects, 73 (20.3%) subjects did not respond to more than three contact attempts and 30 (8.3%) subjects refused further evaluation, despite our recommendations. When investigators strongly suspected that subjects did not meet the CAARMS UHR criteria, they made final diagnoses of no UHR of psychosis (n=197, 54.7%). Further face-to-face interviews by psychiatrists who had been trained and had considerable experience with the CAARMS were arranged for 60 (16.7%) subjects. These subjects completed the ESI and KPQ-16 again to examine convergent validity and test-retest reliability. Final diagnoses were reviewed at a research team meeting with experienced psychiatrists.

The Institutional Review Board of Chonnam National University Hospital approved this study (CNUH 2016088), and all participants provided written informed consent. The initial pre-screening was administered in February 2016 and the diagnostic interviews were conducted between March and early April 2016. The interval between the first and second contacts was about 1 month.



The PQ-16 is a self-report screening questionnaire that assesses the presence of attenuated positive and negative psychotic symptoms on a 2-point scale (true/false) [20]. For each endorsed item, distress was rated on a 4-point scale (ranging from no distress to much distress) [21]. The initial total score, used to determine whether a further structured interview would take place, was calculated from the number of items with responses of true, and ranged from 0 to 16. The distress scale ranged from 0 to 96 and was also used for exploratory analyses. The PQ-16 consists of nine items on the perceptual abnormalities/hallucinations subscale, five items related to unusual thought content/delusional ideas/paranoia, and two negative symptoms. Compared with the number of items assessing perceptual abnormalities, relatively few items assess thought content and cognitive symptoms. The authors who validated the PQ-16 also stated that perceptual aberrations seem to be overrepresented [20]. Therefore, for exploratory investigation to increase the sensitivity of the questionnaire, we modified the KPQ-16 by adding three ESI items (4, 5, and 8; ideas of reference, cognitive impairment, and persecutory ideas); we called this instrument the modified Korean version of the 16-item Prodromal Questionnaire (mKPQ-16). The three added items were selected from those that showed high degrees of predictability in a validation study of the Korean version of the ESI [19], and based on our experience in screening with the ESI in community mental health service settings. The authors (SWK, JM, YSK) translated the PQ-16 into Korean after obtaining permission from the original authors of the PQ-16. Bilingual translators majoring in medicine back-translated the KPQ-16 into English. As no discrepancies between the translations and the original instrument, the final version of the KPQ-16 was completed.


The diagnosis of UHR of psychosis was assessed using the semi-structured CAARMS as the gold standard. The CAARMS is used to determine the intensity/severity and frequency/duration of psychotic like symptoms, with scores ranging from 0 to 6. The original CAARMS consists of seven subscales, but symptomatic criteria for psychosis risk are based exclusively on positive symptom items (disordered thought contents, perceptual abnormalities, and disorganized speech). In this study, four items from the positive symptom subscale were administered to subjects for the diagnosis of UHR of psychosis.

Measures for depression

When investigators conducted face-to-face evaluations, depressive disorder was diagnosed according to the DSM-IV criteria using the Mini-International Neuropsychiatric Interview (MINI) [24]. The nine-item Patient Health Questionnaire (PHQ-9) [25,26] was administered to investigate the level of depressive symptoms.

Statistical analyses

Using data from the 60 subjects who underwent face-to-face evaluations, the internal consistency, reliability, and test-retest reliability of the KPQ-16 were estimated using Cronbach’s alpha and the intraclass correlation coefficient (ICC). Scores >0.65 were considered to indicate sufficient internal consistency [20]. Using data from these subjects, the degree of correlation between KPQ-16 and ESI scores was assessed using Spearman’s correlation coefficient to examine convergent validity. We conducted receiver operating characteristic (ROC) analyses using data from the 257 subjects who finished secondary interviews. We determined optimal cutoff values for total and distress KPQ-16 and mKPQ-16 (19 items) scores based on the sensitivity, specificity, positive predictive value (PPV), and NPV. The PPV reflects the probability that an individual has a CAARMS diagnosis in the case of a positive PQ test result. The accuracy of the test depends on how well it separates the group being tested into those with and without the disease in question, and is measured by the area under the curve (AUC). An AUC value of 0.5 indicates the absence of a difference, and a value of 1 indicates perfect separation of the two groups. We compared the prevalence of depressive disorder according to the diagnosis of UHR of psychosis with Fisher’s exact test. For concurrent and discriminant validity, the Mann-Whitney U-test was performed to compare the KPQ-16, mKPQ-16, and PHQ-9 scores according to the diagnosis of UHR of psychosis based on CAARMS and major depressive disorder and depressive disorder [MINI, not otherwise specified (NOS)] criteria. The data were analyzed using SPSS for Windows (ver. 21.0; IBM Corp., Armonk, NY, USA), and p values <0.05 were considered to be statistically significant.


Diagnostic results

The mean age of the subjects who participated in both evaluations (n=257) was 19.2±0.7 (range 18–23) years. The male: female ratio was almost 1:1 [n=129 (50.2%) females]. The mean age of the subjects who underwent face-to-face investigations (n=60) was also 19.2±0.7 years, but males [n=35 (58.3%)] predominated compared with all study subjects. Among the subjects who were interviewed (n=257), 17 (6.6%) met the CAARMS criteria for UHR of psychosis. One subject met subthreshold frequency criteria and 16 subjects met subthreshold intensity criteria. Thought problems were more common than perceptual disturbance in the UHR diagnosis group. All but one subject (with a subthreshold frequency of hallucination) had subthreshold delusional symptoms. Only four subjects had subthreshold intensity hallucinations meeting the CAARMS criteria.

Receiver operating characteristic analysis

To predict CAARMS-based diagnoses of UHR of psychosis, ROC curves were plotted for the KPQ-16 and mKPQ-16 total scores (Figure 1). AUCs were significant for the KPQ-16 (AUC=0.702, p=0.005) and mKPQ-16 (AUC=0.831, p<0.001) total scores, indicating that these tools are useful for the screening identification of those at UHR of psychosis. The AUC of the mKPQ-16 was greater than that of the KPQ-16, with a difference of 0.129, although both were significant. Exploratory analyses of distress scores of the measures yielded AUC values that were satisfactory and similar to those of the total scores (KPQ-16, 0.716; mKPQ-16, 0.796).

Figure 1.

The receiver operating characteristic (ROC) curves of the Korean version of Prodromal Questionnaire-16 item (KPQ-16) and the modified Korean version of Prodromal Questionnaire-16 item (mKPQ-16) with Comprehensive Assessment of At-Risk Mental States (CAARMS) criteria of ultra-high risk of psychosis as a gold standard.

Validity and reliability

Table 1 summarizes the diagnostic validity of the two scales based on the AUCs and the optimal cutoff points for sensitivity and specificity. For the KPQ-16, cutoff total item scores of 6 and 7 resulted in the best balance of sensitivity (64.7% and 52.9%, respectively) and specificity (71.2% and 84.6%, respectively). For the mKPQ-16, a cutoff total score of 6 resulted in the greatest sensitivity (100%), despite a low specificity value (53.3%) and a PPV of 13.2%. A cutoff total mKPQ-16 score of 7 was more balanced, with favorable sensitivity (76.5%) and specificity (75.4%) and a PPV of 18.1%. Cutoff KPQ-16 and mKPQ-16 distress scores of 14 and 16, respectively, resulted in the greatest sensitivity (64.7% and 76.5%, respectively), with specificities of 74.6% and 74.2%, respectively, and PPVs of 15.2% and 17.3%, respectively.

Diagnostic validity of KPQ-16 and mKPQ-16 based on the area under receiver operating characteristics curve (AUROC) and the optimal cutoff points for sensitivity and specificity

Cronbach’s alpha values for KPQ-16 and mKPQ-16 items were 0.680 and 0.685, respectively. Test-retest reliability was evaluated using data from subjects who underwent face-to-face evaluations, and the ICC for the KPQ-16 total item score was 0.478 (p=0.007). For this dataset, Spearman’s coefficient of correlation between the ESI and KPQ-16 item total scores was 0.644 (p<0.001).

Depressive disorder was diagnosed in 15 (25.0%) subjects (2 with major depressive disorder and 13 with depressive disorder, NOS). The prevalence of depressive disorder in subjects with UHR diagnoses was 29.4%, which did not differ significantly from the prevalence in those with no UHR diagnosis (p=0.743). Table 2 compares the KPQ-16, mKPQ-16, and PHQ-9 scores according to diagnoses of UHR of psychosis and depressive disorder. KPQ-16 and mKPQ-16 scores were significantly higher in subjects with UHR diagnoses, but not in those with depressive disorder, whereas the PHQ-9 score was significantly higher in subjects with depressive disorder, but not in those at UHR of psychosis.

Comparisons of scores on the KPQ-16, mKPQ-16, and Patient Health Questionnaire-9 according to diagnosis


This study validated Korean versions of the PQ-16 as a screening measure of psychosis risk in the college setting. Overall, the Korean versions of the PQ-16 showed good internal consistency and diagnostic validity, successfully distinguishing subjects at UHR of psychosis from the general population. Furthermore, they showed favorable convergent validity with the ESI and discriminant validity against depression. We believe that the validation of a short screening test for psychosis risk will promote screening and the early identification of psychosis risk in Korean schools and communities.

In the context of use as a screening instrument, a cutoff mKPQ-16 score of 7 had a satisfactory balance of sensitivity (76.5%) and specificity (75.4%), comparable to those of the Korean version of the ESI (77% sensitivity, 70% specificity), in differentiating subjects at UHR of psychosis from those with no CAARMS diagnosis. Based on mKPQ-16 scores, 8 of every 10 normal individuals were rejected accurately; using a positive test cutoff score of 7, the probability that an individual had a psychosis risk syndrome was approximately 18%. Investigators who do not want to lose false-negative subjects at UHR of psychosis could apply a cutoff mKPQ-16 score of 6 to improve the sensitivity to 100%, despite the relatively low specificity (53.3%) and PPV (13.2%). A cutoff KPQ-16 score of 6 yielded the same results as did the score in the original study, with generally favorable sensitivity (64.7%) and specificity (71.2%), although these were lower than in previous studies18 and with the mKPQ-16. The characteristics of the PQ-16 with a high number of perceptual disturbance items might contribute to the missing of subjects with no perceptual aberration. This speculation was supported by the finding that the addition of three ESI items on thought and cognitive problems to the KPQ-16 increased the instrument’s sensitivity. The AUC values and performance of the KPQ-16 and mKPQ-16 distress scores were generally similar to those of the measures’ respective total item scores. Considering simplicity and validity, the mKPQ-16 total score might be the most useful screening tool.

In this study, PPVs were relatively low for a predictable screening tool [27,28]. This result may be explained by the observation that PPVs were relatively high in studies of selected help-seeking samples visiting specialized prodrome clinics and community mental health clinics [27,28], and low in studies of subjects with no help-seeking behavior, similar to ours [19,29,30]. Therefore, investigators who wish to increase the PPV of the KPQ-16 may use a higher cutoff score, which would lower the false-positive rate and sensitivity. Nevertheless, when a tool is administered to screen for a serious, low-prevalence disease such as psychosis, the chosen cutoff point should provide good sensitivity, even when the PPV is low or the false-positive rate is high [19]. Moreover, because the PQ-16 is not used for final diagnosis, but simply for pre-testing of the general population to identify those needing further clinical interviews, it could be an appropriate primary mass-screening measure. Screening of the general population for psychosis risk using a self-report questionnaire could result in a high false-positive rate unless other factors are considered [30-32]. In particular, functional decline and high subjective distress should be considered when a diagnosis of UHR of psychosis is given to an individual who does not seek treatment.

Given the high false-positive rate of this measure, we should carefully consider the potential stigma of psychosis screening in the non-treatment-seeking population [31,33]. False identification of youth as prodromal to psychosis can lead to unnecessary concern and emotional harm [31], particularly considering the relatively low PPV of the tool. However, the psychotic behavior and poor clinical outcome of schizophrenia may be more damaging than evaluation in terms of stigma [34,35]. Furthermore, the amelioration of distressful symptoms and prevention of a major psychiatric illness would have much benefit compared with the risk of stigma associated with the screening process. If prevention is successful with screening, the potential stigma generated by a psychotic illness is also ultimately resolved.

Several limitations should be considered when interpreting our data. First, the participants in the study were from non-help-seeking general college population. In addition, the study subjects were young adults. Previous studies have shown that children and adolescents are more likely to report UHR symptoms [36-39]. Therefore, we should consider use of the KPQ-16 cutoff score identified in this study carefully because a different score may be more applicable in a clinical setting involving a treatment-seeking or younger population. Second, this study was cross sectional, rather than longitudinal. Consequently, further research investigating the predictive validity of the KPQ-16 for the longitudinal transition to psychosis is warranted. However, recent studies demonstrated that the longitudinal course of UHR of psychosis is pluripotential and dysfunctional, even when it does not lead to the development of psychosis [13,40]. Therefore, the identification of UHR of psychosis, regardless of psychotic conversion, is important. Finally, selection bias may have affected the results because subjects with KPQ-16 scores <4 were not enrolled in the second investigation and many candidates did not participate in face-to-face clinical interviews.

In conclusion, the results of this study suggest that the KPQ-16 and mKPQ-16 are good screening instruments for use in community and school settings. The small numbers of items make it feasible to screen large numbers of students and young adults. The validation of these measures could contribute to the early identification of psychosis and shorten the duration of untreated psychosis by prompt early intervention in the Korean community.


This study was supported by a grant of the Korean Mental Health Technology R&D Project, Ministry of Health & Welfare, Republic of Korea (HM15C1140). This research work was conducted as part of the Gwangju Mental Health Demonstration Project supported by Ministry of Health & Welfare and Gwangju Metropolitan city, Republic of Korea. We wish to thank students for their participation in this project and Ju-Wan Kim, Taeyoung Yoo, Ga-Young Lee, Hye-Young Yu, and Jin-Hee Hong for their clinical interviews of participants.


1. McGorry PD, Killackey E, Yung AR. Early intervention in psychotic disorders: detection and treatment of the first episode and the critical early stages. Med J Aust 2007;187(7 Suppl):S8–S10.
2. Addington J, Heinssen RK, Robinson DG, Schooler NR, Marcy P, Brunette MF, et al. Duration of untreated psychosis in community treatment settings in the United States. Psychiatr Serv 2015;66:753–756.
3. Kim SW, Jang JE, Lee JY, Lee GY, Yu HY, Park C, et al. Effects of group cognitive-behavioral therapy in young patients in the early stage of psychosis. Psychiatry Investig 2017;14:609–617.
4. Schimmelmann BG, Huber CG, Lambert M, Cotton S, McGorry PD, Conus P. Impact of duration of untreated psychosis on pre-treatment, baseline, and outcome characteristics in an epidemiological first-episode psychosis cohort. J Psychiatr Res 2008;42:982–990.
5. Kane JM, Robinson DG, Schooler NR, Mueser KT, Penn DL, Rosenheck RA, et al. Comprehensive versus usual community care for first-episode psychosis: 2-year outcomes from the NIMH RAISE early treatment program. Am J Psychiatry 2016;173:362–372.
6. Kim SW, Lee BJ, Kim JJ, Yu JC, Lee KY, Won SH, et al. Design and Methodology of the Korean Early Psychosis Cohort Study. Psychiatry Investig 2017;14:93–99.
7. Yoo J, Ahn S, Cho Y, Lee M. A study of duration of untreated psychosis (DUP) for first episode psychosis. Ment Health 2011;2:12–16.
8. Hutton P, Taylor PJ. Cognitive behavioural therapy for psychosis prevention: a systematic review and meta-analysis. Psychol Med 2014;44:449–468.
9. Amminger GP, Mechelli A, Rice S, Kim SW, Klier CM, McNamara RK, et al. Predictors of treatment response in young people at ultra-high risk for psychosis who received long-chain omega-3 fatty acids. Transl Psychiatry 2015;5:e495.
10. Yung AR, McGorry PD, McFariane CA, Jackson HJ, Patton GC, Rakkar A. Monitoring and care of young people at incipient risk of psychosis. Schizophr Bull 1996;22:283–303.
11. Kim SW, Jhon M, Kim JM, Smesny S, Rice S, Berk M, et al. Relationship between erythrocyte fatty acid composition and psychopathology in the vienna omega-3 study. PLoS One 2016;11e0151417.
12. Kim SW, Schafer MR, Klier CM, Berk M, Rice S, Allott K, et al. Relationship between membrane fatty acids and cognitive symptoms and information processing in individuals at ultra-high risk for psychosis. Schizophr Res 2014;158:39–44.
13. Nelson B, Yuen HP, Wood SJ, Lin A, Spiliotacopoulos D, Bruxner A, et al. Long-term follow-up of a group at ultra high risk (“prodromal”) for psychosis: the PACE 400 study. JAMA Psychiatry 2013;70:793–802.
14. American Psychiatric-Association. Diagnostic and Statistical Manual of Mental Disorder 5th Edth ed. Arlington, VA: American Psychiatric Publishing; 2013.
15. Yung AR, Pan Yuen H, Mcgorry PD, Phillips LJ, Kelly D, Dell’olio M, et al. Mapping the onset of psychosis: the comprehensive assessment of at-risk mental states. Aust N Z J Psychiatry 2005;39:964–971.
16. Fusar-Poli P, Cappucciati M, Rutigliano G, Lee TY, Beverly Q, Bonoldi I, et al. Towards a standard psychometric diagnostic interview for subjects at ultra high risk of psychosis: CAARMS versus SIPS. Psychiatry J 2016;2016:7146341.
17. Miller TJ, McGlashan TH, Rosen JL, Cadenhead K, Ventura J, McFarlane W, et al. Prodromal assessment with the structured interview for prodromal syndromes and the scale of prodromal symptoms: predictive validity, interrater reliability, and training to reliability. Schizophr Bull 2003;29:703–715.
18. Addington J, Stowkowy J, Weiser M. Screening tools for clinical high risk for psychosis. Early Interv Psychiatry 2015;9:345–356.
19. Chung YC, Kang NI, Im YJ, Kim SW, Cho IH, Lee YM, et al. Validation of the Korean version of the Eppendorf Schizophrenia Inventory as a screening measure to detect adolescents at ultra-high risk for psychosis. Early Interv Psychiatry 2013;7:71–79.
20. Ising HK, Veling W, Loewy RL, Rietveld MW, Rietdijk J, Dragt S, et al. The validity of the 16-item version of the Prodromal Questionnaire (PQ-16) to screen for ultra high risk of developing psychosis in the general help-seeking population. Schizophr Bull 2012;38:1288–1296.
21. Loewy RL, Bearden CE, Johnson JK, Raine A, Cannon TD. The prodromal questionnaire (PQ): preliminary validation of a self-report screening measure for prodromal and psychotic syndromes. Schizophr Res 2005;79:117–125.
22. Mass R, Girndt K, Matouschek AK, Peter PM, Plitzko N, Andresen B, et al. Introducing the Eppendorf Schizophrenia Inventory (ESI) as a psychometric method for schizotypy research. Pers Individ Dif 2007;42:525–534.
23. Loewy RL, Therman S, Manninen M, Huttunen MO, Cannon TD. Prodromal psychosis screening in adolescent psychiatry clinics. Early Interv Psychiatry 2012;6:69–75.
24. Sheehan DV, Lecrubier Y, Sheehan KH, Amorim P, Janavs J, Weiller E, et al. The Mini-International Neuropsychiatric Interview (M.I.N.I.): the development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. J Clin Psychiatry 1998;59(Suppl 20):22–33.
25. Han C, Jo SA, Kwak JH, Pae CU, Steffens D, Jo I, et al. Validation of the Patient Health Questionnaire-9 Korean version in the elderly population: the Ansan Geriatric study. Compr Psychiatry 2008;49:218–223.
26. Spitzer RL, Kroenke K, Williams JB. Validation and utility of a self-report version of PRIME-MD: the PHQ primary care study. Primary Care Evaluation of Mental Disorders. Patient Health Questionnaire. JAMA 1999;282:1737–1744.
27. Kobayashi H, Nemoto T, Koshikawa H, Osono Y, Yamazawa R, Murakami M, et al. A self-reported instrument for prodromal symptoms of psychosis: testing the clinical validity of the PRIME Screen-Revised (PS-R) in a Japanese population. Schizophr Res 2008;106:356–362.
28. Loewy RL, Pearson R, Vinogradov S, Bearden CE, Cannon TD. Psychosis risk screening with the Prodromal Questionnaire--brief version (PQ-B). Schizophr Res 2011;129:42–46.
29. Muller M, Vetter S, Buchli-Kammermann J, Stieglitz RD, Stettbacher A, Riecher-Rossler A. The Self-screen-Prodrome as a short screening tool for pre-psychotic states. Schizophr Res 2010;123:217–224.
30. Owoso A, Ndetei DM, Mbwayo AW, Mutiso VN, Khasakhala LI, Mamah D. Validation of a modified version of the PRIME screen for psychosis-risk symptoms in a non-clinical Kenyan youth sample. Compr Psychiatry 2014;55:380–387.
31. Loewy RL, Johnson JK, Cannon TD. Self-report of attenuated psychotic experiences in a college population. Schizophr Res 2007;93:144–151.
32. O’Toole BI. Screening for low prevalence disorders. Aust N Z J Psychiatry 2000;34(Suppl):S39–S46.
33. Post SG. Preventing schizophrenia and Alzheimer disease: comparative ethics. Schizophr Res 2001;51:103–108.
34. Kim SW, Polari A, Melville F, Moller B, Kim JM, Amminger P, et al. Are current labeling terms suitable for people who are at risk of psychosis? Schizophr Res 2017;188:172–177.
35. Yang LH, Link BG, Ben-David S, Gill KE, Girgis RR, Brucato G, et al. Stigma related to labels and symptoms in individuals at clinical high-risk for psychosis. Schizophr Res 2015;168:9–15.
36. Escher S, Romme M, Buiks A, Delespaul P, Van Os J. Independent course of childhood auditory hallucinations: a sequential 3-year follow-up study. Br J Psychiatry Suppl 2002;43:s10–s18.
37. Schimmelmann BG, Michel C, Martz-Irngartinger A, Linder C, Schultze-Lutter F. Age matters in the prevalence and clinical significance of ultra-high-risk for psychosis symptoms and criteria in the general population: Findings from the BEAR and BEARS-kid studies. World Psychiatry 2015;14:189–197.
38. McGorry PD, McFarlane C, Patton GC, Bell R, Hibbert ME, Jackson HJ, et al. The prevalence of prodromal features of schizophrenia in adolescence: a preliminary survey. Acta Psychiatr Scand 1995;92:241–249.
39. Fonseca-Pedrero E, Gooding DC, Ortuno-Sierra J, Paino M. Assessing self-reported clinical high risk symptoms in community-derived adolescents: a psychometric evaluation of the Prodromal Questionnaire-Brief. Compr Psychiatry 2016;66:201–208.
40. McGorry P, Nelson B. Why We need a transdiagnostic staging approach to emerging psychopathology, early diagnosis, and treatment. JAMA Psychiatry 2016;73:191–192.

Article information Continued

Figure 1.

The receiver operating characteristic (ROC) curves of the Korean version of Prodromal Questionnaire-16 item (KPQ-16) and the modified Korean version of Prodromal Questionnaire-16 item (mKPQ-16) with Comprehensive Assessment of At-Risk Mental States (CAARMS) criteria of ultra-high risk of psychosis as a gold standard.

Table 1.

Diagnostic validity of KPQ-16 and mKPQ-16 based on the area under receiver operating characteristics curve (AUROC) and the optimal cutoff points for sensitivity and specificity

Scale Score AUROC 95% CI p-value Cutoff Sensitivity (%) Specificity (%) PPV (%) NPV (%)
KPQ-16 Item total 0.702 0.564–0.840 0.005 5 82.4 37.5 8.5 96.8
6 64.7 71.2 13.8 96.6
7 52.9 84.6 19.6 96.2
mKPQ-16 Item total 0.831 0.764–0.898 <0.001 6 100 53.3 13.2 100
7 76.5 75.4 18.1 97.8
8 58.8 85.8 22.7 96.7
KPQ-16 Distress 0.716 0.567–0.866 0.003 12 82.4 56.7 11.9 97.8
13 70.6 64.6 12.4 96.9
14 64.7 74.6 15.2 96.8
15 58.8 80.4 17.5 96.5
mKPQ-16 Distress 0.796 0.697–0.894 <0.001 14 88.2 60.4 13.6 98.6
15 76.5 67.5 14.3 97.6
16 76.5 74.2 17.3 97.8
17 52.9 81.2 16.7 96.1

KPQ-16: Korean version of Prodromal Questionnaire-16 item, mKPQ-16: modified Korean version of Prodromal Questionnaire-16 item, 95% CI: 95% confidence interval, PPV: positive predictive value, NPV: negative predictive value

Table 2.

Comparisons of scores on the KPQ-16, mKPQ-16, and Patient Health Questionnaire-9 according to diagnosis

Ultra-high risk of psychosis
Depressive disorder
Yes (N=17) No (N=240) Z p-value Yes (N=15) No (N=45) Z p-value
KPQ-16, total 7 (5–7.5) 5 (4–6) 2.912 0.004 6 (4–8) 7 (5–8) 0.486 0.627
mKPQ-16, total 8 (6.5–8) 5 (5–6) 4.664 <0.001 7 (6–8) 8 (6–8) 0.096 0.924
KPQ-16, distress 16 (12–19.5) 11 (9–14) 2.994 0.003 14 (10–18) 16 (12–19) 1.010 0.313
mKPQ-16, distress 19 (15–22) 12 (10–16) 4.098 <0.001 16 (12–23) 17 (15–20.5) 0.660 0.509
Patient Health Questionnaire-9 3 (2–8) 5 (3–7)* 0.438 0.662 7 (5–11) 4 (2–6) 3.282 0.001

number of subjects were 43.

KPQ-16: Korean version of Prodromal Questionnaire-16 item, mKPQ-16: modified Korean version of Prodromal Questionnaire-16 item