Validation of the Korean Version of the Delirium Diagnostic Tool-Provisional (K-DDT-Pro)

Kyeong Mee Kim; Man-shik Shim; Dahyun Yi; So Yeon Jeon; Jeong Lan Kim

doi:10.30773/pi.2022.0151

Psychiatry Investig > Volume 19(9); 2022 > Article

Kim, Shim, Yi, Jeon, and Kim: Validation of the Korean Version of the Delirium Diagnostic Tool-Provisional (K-DDT-Pro)

Original Article

Psychiatry Investigation 2022;19(9):748-753.

Published online: September 22, 2022

DOI: https://doi.org/10.30773/pi.2022.0151

Validation of the Korean Version of the Delirium Diagnostic Tool-Provisional (K-DDT-Pro)

Kyeong Mee Kim¹

, Man-shik Shim²

, Dahyun Yi³

, So Yeon Jeon^1,⁴

, Jeong Lan Kim^1,⁴

¹Department of Psychiatry, Chungnam National University Hospital, Daejeon, Republic of Korea

²Department of Cardiovascular & Thoracic Surgery, Chungnam National University Hospital, Daejeon, Republic of Korea

³Institute of Human Behavioral Medicine, Seoul National University Medical Research Center, Seoul, Republic of Korea

⁴Department of Psychiatry, Chungnam National University College of Medicine, Daejeon, Republic of Korea

Correspondence: So Yeon Jeon, MD, PhD Department of Psychiatry, Chungnam National University Hospital, Chungnam National University College of Medicine, 282 Munhwa-ro, Jung-gu, Daejeon 35015, Republic of Korea Tel: +82-42-280-7281, Fax: +82-42-280-8423, E-mail: iris870422@gmail.com

Correspondence: Jeong Lan Kim, MD, PhD Department of Psychiatry, Chungnam National University Hospital, Chungnam National University College of Medicine, 282 Munhwa-ro, Jung-gu, Daejeon 35015, Republic of Korea Tel: +82-42-280-7281, Fax: +82-42-280-8423, E-mail: kimjl@cnu.ac.kr

Received June 5, 2022 Revised July 14, 2022 Accepted July 19, 2022

This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Objective

The Delirium Diagnostic Tool-Provisional (DDT-Pro) was designed to detect the presence and severity of delirium briefly and objectively regardless of psychiatric expertise. We translated the DDT-Pro into Korean and validated it in elderly Korean patients.

Methods

To validate the translation and evaluate inter-rater reliability, a psychiatric trainee and a research nurse independently assessed the same patients referred to the Department of Psychiatry. The results were compared with the reference evaluations performed by a psychiatrist using the Diagnostic and Statistical Manual of Mental Disorders, 5th Edition (DSM-5).

Results

We enrolled 42 elderly patients. The Cronbach’s alpha coefficient were 0.839 (the trainee), 0.822 (the nurse). The Cohen’s weighted κ between the trainee and nurse, ranged from 0.555±0.102 to 0.776±0.062. The Pearson correlation coefficients (Korean version of the DDT-Pro [K-DDT-Pro] and Korean version of the Delirium Rating Scale-Revised-98 [DRS-R98-K] total score) were γ=-0.850 (the trainee), and γ=-0.821 (the nurse). The areas under the ROC curves (AUCs) were 0.974 (the trainee) and 0.893 (the nurse).

Conclusion

The K-DDT-Pro exhibited high internal consistency and relatively substantial inter-rater reliability. The correlation with the DRS-R98-K was strongly negative. The accuracy of the K-DDT-Pro was excellent, regardless of expertise. In conclusion, the K-DDT-Pro is a brief and simple tool that usefully screens for delirium in elderly patients.

Keywords: K-DDT-Pro, Delirium, Reliability, Validity

INTRODUCTION

Delirium is a disturbance of consciousness, attention, and baseline cognitive function according to the Diagnostic and Statistical Manual of Mental Disorders, 5th Edition (DSM-5) [1]. Delirium increases morbidity, the hospitalization time, and mortality [2-5]. Delirium is common (7%-47%) in elderly inpatients [6-8]. As populations age, delirium will become more prominent. It is important to prevent delirium or at least to detect delirium early; this allows rapid medical intervention. However, in clinical studies, about two-thirds of delirium cases were not diagnosed; in emergency settings, 57%-83% of delirium cases were missed, because it is difficult for medical staff lacking expertise with psychiatric interviews and differential diagnosis to identify and evaluate the characteristic neuropsychiatric symptoms of delirium and to distinguish them from those of other diseases [9-11]. There is a need for a simple screening tool that can be used by medical staff who are not specialists in psychiatry or neurology. However, the validity of a delirium evaluation must be verified, as must the capacity of the tool to distinguish delirium from dementia. Also, the various aspects of delirium must be revealed, and, finally, the tool must be practical [12].

The Confusion Assessment Method algorithm (CAM-A) is one of the most commonly used tools that enables nonpsychiatric clinicians to detect delirium quickly and easily. The CAM-A evaluates the presence of four important symptoms of delirium: acute onset and fluctuating course, inattention, disorganized thinking, and altered level of consciousness. The CAM-A can be completed in less than 5 minutes. Because of its simplicity and convenience, CAM-A is widely used by nonexpert clinicians for delirium detection. However, some training is required for optimal use of the CAM. The CAM allows stupor or coma to be diagnosed as delirium and is a dichotomous tool not allowing it to be a continuous measure even with adaptations [13-16]. The 4A’s Test (4AT), another brief assessment tools for delirium and cognitive impairment, developed in the UK. 4AT screens patients with four items that assess the alertness, orientation, attention, and history of a patient’s acute change or fluctuation course in cognition or other mental function. It is short (<2 minutes), simple screening tool for delirium and shows good diagnostic test accuracy for delirium. In the recent meta-analysis study, the pooled sensitivity was 88% and the pooled specificity was 88%. It does not require special training and has the advantage of being easy to implement. However, the test results are affected by whether dementia is accompanied [17]. In addition to CAM-A and 4AT, many tools to detect delirium are being developed and verified.

The Delirium Diagnostic Tool-Provisional (DDT-Pro) developed by Kean et al. [18] is an excellent screen allowing a provisional diagnosis of delirium; the tool is delirium-specific, sensitive, and reliable. The DDT-Pro evaluates the three core symptom domains of delirium: cognition (vigilance in DDTPro), higher-level thinking (comprehension in DDT-Pro), and circadian rhythm (sleep-wake cycle in DDT-Pro). The presence of these core domain symptoms could accurately reflect delirium while other brief tools do not evaluate all the three core symptom domains. In comparison to the CAM-A, because of its highly structured, quantitative items, the DDT-Pro would be more useful to provisionally diagnose delirium. The previous validation study of the DDT-Pro in general hospital patients shows a significant advance over the commonly used CAM. Although the English and Spanish versions of DDT-Pro have been validated with excellent results, validation in other languages employed by specific patient populations is also required [16,19,20]. A Korean version would be useful as Korean hospitals lack such tools. We translated the DDT-Pro from English to Korean (Korean version of the DDT-Pro, K-DDT-Pro). We compared the performance of the K-DDT-Pro to that of the Korean version of the Delirium Rating Scale-Revised-98 (DRS-R98-K), which is validated and is commonly used in Korea [21,22]. We explored the inter-rater reliability of the K-DDT-Pro between an expert and a non-expert; the reference diagnosis was that of a psychiatrist using DSM-5 criteria.

METHODS

Patients

From April 2021 to March 2022, we enrolled inpatients aged 60 years or older of the Department of Psychiatry referred because of suspected delirium or a risk of delirium from various departments at Chungnam National University Hospital. The exclusion criteria included a refusal to participate, any severe communication problem (e.g., hearing loss), stupor, and coma. The Chungnam National University Hospital Institutional Review Board (IRB) approved this study (IRB no. 2021-03-058), and written informed consent was obtained from patients or if necessary, their surrogates.

The Delirium Diagnostic Tool-Provisional description

The three items of DDT-Pro (comprehension, vigilance, and the sleep-wake cycle) are derived from two well-validated delirium diagnostic tools, the Cognitive Test for Delirium (CTD) (items 1 and 2), and the Delirium Rating Scale-Revised-98 (DRS-R98) (item 3). These items represent the core symptoms of delirium. Item 1 (comprehension) and item 2 (vigilance) are expressed in two equivalent forms (A and B), which can be used alternatively used during repeated assessments. Both forms directly reflect the patients’ performance. Item 3 (sleep-wake cycle) is evaluated by the patient interview or by another source for the previous 12-24 hours. Item 3 taken from the DRS-R98 has its scoring reversed to align with that of the CTD items. Each item on the DDT-Pro is scored on a 4-point scale (0-3). Thus, the total score ranges from 0 to 9 points. The lower the score, the higher the severity of delirium, where a score of 8-9 is normal.

Translation and back-translation

We contacted Dr. Trzepacz, the original author of the DDT-Pro, via e-mail and got permission to produce a K-DDT-Pro. The DDT-Pro was translated using the guidelines of the Translation and Cultural Adaptation Group. The initial translation was prepared by a psychiatric trainee and then revised and supplemented by two psychiatrists working independently. Then, all researchers met to discuss progress. When translating item 2, we took care to ensure that patients could easily distinguish the words. The final Korean version was given to a professional translator who speaks both Korean and English for back-translation; we did not provide any information on the original version. The back-translated version of K-DDT-Pro was sent to Dr. Trzepacz by e-mail and was approved. Next, in April 2021, the psychiatric trainee conducted a preliminary study on five delirious patients.

Inter-rater reliability and validation of delirium assessment

Prior to the full-scale evaluation, all raters met to discuss the assessment method. One experienced psychiatrist evaluated all patients using DSM-5 delirium criteria. Then, one psychiatric trainee; a psychiatry resident who had graduated from medical school (who used form A) and one research nurse (who used form B) independently performed delirium assessments on the same subjects using the K-DDT-Pro. The trainee was the psychiatric expert, and the nurse was the non-expert. During the evaluation, the trainee also evaluated patients using the DRS-R98-K with information from their family members and caregivers. The patient assessments of the three raters were completely independent (Figure 1). All evaluations were conducted between 14:00 and 18:00 to minimalize any possible bias arising from fluctuations in each patient’s condition.

Statistical analysis

We classified the patients into the delirium group or non-delirium group based on their DSM-5 diagnoses. The two groups were compared. As the sample sizes were small (both n<30), we subjected all demographic and clinical variables to normality analysis. We then compared the means between the two groups. Age, which satisfied the normality requirement, was compared using the independent t test. Years of education, which was not normally distributed, was compared employing the Mann-Whitney U test. We used the chi-squared test to compare all other nominal variables. To assess the internal reliability of each K-DDT-Pro item, we compared the trainee and nurse ratings by deriving the Cronbach’s α coefficient. The Cohen’s weighted kappa (κ) coefficient was used to assess the inter-rater reliability of the K-DDT-Pro forms administered by the psychiatric trainee (form A) and the nurse (form B). We evaluated all item scores and the overall scores of the two independent raters.

The Criterion validity of the K-DDT-Pro (versus the DRS-R98-K total and severity scores) was evaluated via Pearson correlation analysis. We drew receiver operating characteristic (ROC) curves to assess the accuracy, sensitivity, and specificity of the K-DDT-Pro scores. The references were the psychiatrist DSM-5-based diagnoses. We then determined a K-DDT-Pro cutoff that balanced sensitivity and specificity. All analyses were performed using the SPSS ver. 26.0 software (IBM Co., Armonk, NY, USA).

RESULTS

Baseline characteristics

The patient flow diagram is shown in Figure 2. We interviewed 95 patients who met the inclusion criteria, but 53 also met the exclusion criteria. The ‘communication problem’ of the exclusion criteria is when the patient has severe hearing loss or can neither talk nor motion a response. Therefore, we enrolled 42 patients. The sociodemographic characteristics of the subjects are summarized in Table 1. Using the DSM-5 criteria, 25 were diagnosed with delirium by the psychiatrist. Eight had previously been diagnosed with dementia by psychiatrists or neurologists. The independent t test showed that patients with delirium were older than others. However, no other parameter differed significantly between the two groups.

Internal and inter-rater reliability of the Korean version of the Delirium Diagnostic Tool-Provisional

The Cronbach’s α coefficients for the psychiatric trainee and the nurse were 0.839 and 0.822, respectively. The internal reliability of the K-DDT-Pro was very good. The Cohen weighted κ coefficients for the K-DDT-Pro items were (item 1) 0.555±0.102 (p<0.001), (item 2) 0.776±0.062 (p<0.001), (item 3) 0.647±0.081 (p<0.001); and that for the total score 0.718±0.048 (p<0.001) (Table 2). The inter-rater agreement was moderate to substantial.

Validity of the Korean version of the Delirium Diagnostic Tool-Provisional and selection of the cutoff score for delirium

The K-DDT-Pro and the DRS-R98-K total and severity scale scores were inversely related as revealed by the Pearson rho values (Table 3). The K-DDT-Pro assessments of the psychiatric trainee correlated with the DRS-R98-K severity score (-0.850 [p<0.001]) and with the DRS-R98-K total score (-0.857 [p<0.001]). The figures for the nurse were -0.821 (p<0.001) and -0.815 (p<0.001), respectively. Thus, the correlations were very high regardless of the rater’s psychiatric expertise.

The areas under the ROC curves (AUCs) for K-DDT-Pro diagnostic accuracy using the DSM-5 criteria as the reference were 97.4% (95% confidence interval [CI]=93.3%-100%) for the psychiatric trainee and 89.4% (95% CI=80%-98.6%) for the nurse (Figure 3). The sensitivity and specificity of the K-DDT-Pro diagnoses compared to the of DSM-5 diagnoses as revealed by ROC analysis are listed in Table 4. The ascending sensitivity and descending specificity gradients were determined for various cutoff values. K-DDT-Pro scores ≤6 and ≤7 both seemed to balance sensitivity and specificity. Scores ≤7 balanced sensitivity and specificity only in the assessments of the psychiatric trainee. Therefore, the optimal cutoff may be ≤6 or ≤7.

DISCUSSION

We prepared a K-DDT-Pro and evaluated its reliability and validity in terms of diagnosing delirium. In terms of reliability, all three items evidenced high internal consistency, with Cronbach’s α coefficient of 0.839 (the trainee) and 0.822 (the nurse). In terms of the Cohen weighted κ coefficients for each item, the tool evidenced moderate (0.555±0.102, item 1) to substantial (0.776±0.062, item 2) inter-rater reliability between the psychiatric expert and the non-expert. Note that the inter-rater reliability of item 1 was lower than that of others. For comparison, when the item 1 scores of the two raters were averaged, that of the psychiatric trainee (2.4286) was higher than that of the nurse (2.2143). One possible reason is that form A (used by the trainee) could be more intuitively understood by the elderly patients than form B (used by the nurse). Also, the possibility that the evaluation methods differed (even slightly) between the raters cannot be excluded.

The nurse ratings in this study were a little less strong than the trainee. In our opinions, because the research nurse can’t observe the patient’s conditions bedside unlike clinical nurses (representing non-experts in previous studies), it must have been more difficult for research nurse to acquire information about patients than clinical nurses or the trainee. Of course, it can be explained with absolute results of the present study, but further research on clinical nurses (instead of the research nurse) may be needed to explain that the DDT-Pro can be used as useful in non-experts (e.g., nurses, rehabilitation staff) as physicians.

A significant inverse Pearson correlation was apparent between the K-DDT-Pro and DRS-R98-K scores. As the DRS-R98-K is well-validated, this constitutes indirect evidence that the K-DDT-Pro will also usefully screen for delirium. We then confirmed the validity of the K-DDT-Pro.

To ensure that the tool would be clinically useful, the optimal cutoff for delirium diagnosis was obtained via ROC analyses; this was ≤6 or ≤7, as for the English and Spanish versions [16,18,19]. Considering subsyndromal delirium (SSD), the scores of 6 and 7 indicate SSD and full delirium is denoted by the lower score range.19 Compared to the original validation studies of DDT-Pro (sensitivity 100% and specificity 94% in Kean et al. [18], 2010; sensitivity 90% and specificity 87% in Franco et al.,16 2020), the K-DDT-Pro exhibited similar sensitivity and specificity values.

Our work had several limitations. First, the sample size was too small (n=42) to compare the outcomes of patients with and without dementia. Given the small sample size, we sought to render the samples homogeneous, but this was difficult. In the present study, a research nurse was the non-psychiatric expert. Clinical nurses who observe the patient’s conditions bedside should be evaluated as raters.

In conclusion, the K-DDT-Pro exhibited high validity and reliability in terms of delirium screening regardless of the rater’s psychiatric expertise. Thus, the K-DDT-Pro will find clinical applications in Korean hospitals. However, additional studies with larger sample sizes are required. Also, clinical nurses who observe patients’ conditions bedside should be evaluated as raters.

We hope that all medical staff, regardless of their psychiatric expertise, will use the brief K-DDT-Pro to improve the prognosis of elderly inpatients with delirium. Early detection of delirium will trigger rapid medical intervention.

Notes

Availability of Data and Material

The datasets generated or analyzed during the study are available from the corresponding author on reasonable request.

Conflicts of Interest

The authors have no potential conflicts of interest to disclose.

Author Contributions

Conceptualization: Kyeong Mee Kim, So Yeon Jeon, Jeong Lan Kim. Data curation: Kyeong Mee Kim. Formal analysis: Kyeong Mee Kim. Investigation: Kyeong Mee Kim, So Yeon Jeon. Methodology: Kyeong Mee Kim, So Yeon Jeon, Jeong Lan Kim. Project administration: Kyeong Mee Kim, So Yeon Jeon, Jeong Lan Kim. Resources: all authors. Writing—original draft: Kyeong Mee Kim, So Yeon Jeon, Jeong Lan Kim. Writing—review & editing: Kyeong Mee Kim, So Yeon Jeon, Jeong Lan Kim.

Funding Statement

This work was supported by research fund of Chungnam National University.

Figure 1.

The process of the assessment. DSM-5, Diagnostic and Statistical Manual of Mental Disorders, 5th Edition; K-DDTPro, Korean version of the Delirium Diagnostic Tool-Provisional; DRS-R98-K, Korean version of the Delirium Rating Scale-Revised-98.

Figure 2.

Flow diagram of the study participants. ^*when the patient has severe hearing loss or can neither talk nor motion a response. DSM-5, Diagnostic and Statistical Manual of Mental Disorders, 5th Edition.

Figure 3.

AUC for the K-DDT-Pro for delirium diagnosis (DSM-5 independent diagnosis). K-DDT-Pro, Korean version of the Delirium Diagnostic Tool-Provisional; AUC, area under the receiver operating characteristic curve; CI, confidence interval.

Table 1.

Baseline characteristics of participants

	Total (N=42)	Delirium group (N=25)	No delirium group (N=17)	p value (t, U, χ²)
Age (yr)	73.69±7.41	76.08±7.36	70.18±6.13	0.01^*
Sex, female	20 (47.6)	11 (44.0)	9 (52.9)	0.569^†
Education level (yr)	7.45±4.16	7.04±3.81	8.06±4.67	0.264^‡
Marital status, married	31 (73.8)	19 (76.0)	12 (70.6)	0.366^†

Values are presented as mean±standard deviation for age, education level, and as N (%) for sex (female) and marital status (married). Delirium group, participants who diagnosed to delirium with Diagnostic and Statistical Manual of Mental Disorders, 5th Edition (DSM-5) by a psychiatrist.

^* significant at p<0.05 for independent t test;

^† p value by chi-square tests;

^‡ p value by Mann-Whitney U test

Table 2.

Inter-rater reliability of K-DDT-Pro (psychiatric trainee and nurse)

	Kappa (κ)±standard error (95% CI)
	Cohen’s simple kappa	Cohen’s weighted kappa
K-DDT-Pro total^†	0.318±0.085^*	0.718±0.048^*
K-DDT-Pro item 1	0.421±0.115^*	0.555±0.102^*
K-DDT-Pro item 2	0.601±0.094^*	0.776±0.062^*
K-DDT-Pro item 3	0.512±0.101^*	0.647±0.081^*

^* p<0.001;

^† total score of all items in K-DDT-Pro.

K-DDT-Pro, Korean version of the Delirium Diagnostic Tool-Provisional; CI, confidence interval

Table 3.

Correlations among K-DDT-Pro and DRS-R98-K

K-DDT-Pro total^‡	Pearson’s r
K-DDT-Pro total^‡	DRS-R98-K severity^*	DRS-R98-K total^†
Trainee	-0.850 (p<0.001)	-0.857 (p<0.001)
Nurse	-0.821 (p<0.001)	-0.815 (p<0.001)

^* severity item score for DRS-R98-K;

^† total score of all items in DRS-R98-K;

^‡ total score of all items in K-DDT-Pro.

K-DDT-Pro, Korean version of the Delirium Diagnostic Tool-Provisional; DRS-R98-K, Korean version of the Delirium Rating Scale-Revised-98

Table 4.

Sensitivity and specificity analysis using patient operating characteristic for the K-DDT-Pro for delirium diagnosis in accordance with DSM-5 (total N=42)

Cutoff	K-DDT-Pro by a psychiatric trainee		K-DDT-Pro by a nurse
Cutoff	Sensitivity (%)	Specificity (%)	Sensitivity (%)	Specificity (%)
1	8	100	4	100
2	12	100	32	100
3	36	100	44	100
4	48	100	52	100
5	60	100	56	100
6	84	94.1	80	82.4
7	100	82.4	92	52.9
8	100	41.2	100	29.4

K-DDT-Pro, Korean version of the Delirium Diagnostic Tool-Provisional; DSM-5, Diagnostic and Statistical Manual of Mental Disorders, 5th Edition

REFERENCES

1. American Psychiatric Association. Diagnostic and statistical manual of mental disorders. 5th ed. Arlington, VA: American Psychiatric Publishing; 2013.

2. Curyto KJ, Johnson J, TenHave T, Mossey J, Knott K, Katz IR. Survival of hospitalized elderly patients with delirium: a prospective study. Am J Geriatr Psychiatry 2001;9:141-147.

3. Inouye SK. Delirium in older persons. N Engl J Med 2006;354:1157-1165.

4. Inouye SK. Delirium in hospitalized older patients. Clin Geriatr Med 1998;14:745-764.

5. Lima DP, Ochiai ME, Lima AB, Curiati JA, Farfel JM, Filho WJ. Delirium in hospitalized elderly patients and post-discharge mortality. Clinics (Sao Paulo) 2010;65:251-255.

6. Jang HJ, Kim JH, Shin SJ, Hwang EH. Adaptation of a translated guideline on screening for delirium, dementia, and depression in older adults. J Korean Gerontol Nurs 2015;17:76-88.

7. Shin JY, Moon YS, Lee JM, Uhm KE, Jeong MY, Choi JK, et al. Clinical practice guideline for screening of delirium in hospitalized older adults. Korean J Fam Pract 2018;8:645-653.

8. Lee DK. Clinical approach to confusion and delirium in the elderly. Korean J Clin Geri 2014;15:1-8.

9. Han JH, Wilson A, Ely EW. Delirium in the older emergency department patient: a quiet epidemic. Emerg Med Clin North Am 2010;28:611-631.

10. Fong TG, Tulebaev SR, Inouye SK. Delirium in elderly adults: diagnosis, prevention and treatment. Nat Rev Neurol 2009;5:210-220.

11. Hshieh TT, Yue J, Oh E, Puelle M, Dowal S, Travison T, et al. Effectiveness of multicomponent nonpharmacological delirium interventions: a meta-analysis. JAMA Intern Med 2015;175:512-520.

12. Korean Association for Geriatric Psychiatry. Textbook of geriatric psychiatry. Seoul: Jungang Moonhwa Co.; 2004, p. 177-187.

13. Inouye SK, van Dyck CH, Alessi CA, Balkin S, Siegal AP, Horwitz RI. Clarifying confusion: the confusion assessment method. A new method for detection of delirium. Ann Intern Med 1990;113:941-948.

14. Wei LA, Fearing MA, Sternberg EJ, Inouye SK. The confusion assessment method: a systematic review of current usage. J Am Geriatr Soc 2008;56:823-830.

15. Rolfson DB, McElhaney JE, Jhangri GS, Rockwood K. Validity of the confusion assessment method in detecting postoperative delirium in the elderly. Int Psychogeriatr 1999;11:431-438.

16. Franco JG, Ocampo MV, Velásquez-Tirado JD, Zaraza DR, Giraldo AM, Serna PA, et al. Validation of the delirium diagnostic tool-provisional (DDT-Pro) with medical inpatients and comparison with the confusion assessment method algorithm. J Neuropsychiatry Clin Neurosci 2020;32:213-226.

17. Tieges Z, Maclullich AMJ, Anand A, Brookes C, Cassarino M, O’connor M, et al. Diagnostic accuracy of the 4AT for delirium detection in older adults: systematic review and meta-analysis. Age Ageing 2021;50:733-743.

18. Kean J, Trzepacz PT, Murray LL, Abell M, Trexler L. Initial validation of a brief provisional diagnostic scale for delirium. Brain Inj 2010;24:1222-1230.

19. Franco JG, Trzepacz PT, Sepúlveda E, Ocampo MV, Velásquez-Tirado JD, Zaraza DR, et al. Delirium diagnostic tool-provisional (DDT-Pro) scores in delirium, subsyndromal delirium and no delirium. Gen Hosp Psychiatry 2020;67:107-114.

20. Sepúlveda E, Bermúdez E, González D, Cotino P, Viñuelas E, Palma J, et al. Validation of the delirium diagnostic tool-provisional (DDT-Pro) in a skilled nursing facility and comparison to the 4 ‘A’s test (4AT). Gen Hosp Psychiatry 2021;70:116-123.

21. Lim KO, Kim SY, Lee YH, Lee SW, Kim JL. A validation study for the Korean version of delirium rating scale-revised-98 (K-DRS-98). J Korean Neuropsychiatr Assoc 2006;45:518-526.

22. Lee Y, Ryu J, Lee J, Kim HJ, Shin IH, Kim JL, et al. Korean version of the delirium rating scale-revised-98: reliability and validity. Psychiatry Investig 2011;8:30-38.