Since the inflammatory process has been implicated in the pathophysiology of psychiatric disorder, an important issue emerging is to assess the test-retest reliability of cytokine measurement in healthy individuals and patients with schizophrenia. The objective of the present study was to investigate the test-retest reliability of bead-based multiplex immunoassay technology (BMIT) for cytokine measurement by using a Bland-Altman plot (BAP).
Twenty healthy individuals and twenty patients with schizophrenia were enrolled, and a 17-plex cytokine assay was used to measure inflammatory biomarkers at baseline and two weeks later. The test-retest reliability was examined by BAP, 95% limits of agreement (LOA), intraclass correlation coefficient (ICC), and coefficient of repeatability (CoR).
In the healthy controls, only interleukin (IL)-2, IL-13, IL-10, IL-17, and macrophage inflammatory protein-1β showed excellent ICC. The BAP with 95% LOA determined that 13 cytokines showed acceptable 95% LOA for a 2-week test-retest reliability, and only IL-1β, IL-12 and tumor necrosis factor (TNF)-α had significant test-retest bias. The CoR of cytokines varied significantly, ranging from 1.72 to 218.1. Compared with healthy controls, patients with schizophrenia showed significantly higher levels of IL-5, IL-13, and TNF-α and significantly lower levels of IL-4, IL-12, and interferon-gamma (IFN-γ). Of these six cytokines, IL-12 and TNF-α were considered suboptimal reliability.
The findings from ICC and CoR implied that the test-retest reliability of BMIT for cytokine measurement were suboptimal. However, the BAP with 95% LOA confirmed that BMIT can reliably distinguish schizophrenia from healthy individuals in cytokine measurement, while significant within-subject variation and between-group overlapping were evident in cytokine expression.
A growing body of evidence from ecological, animal, and clinical studies has suggested that neuroinflammation contributes to the pathogenesis of schizophrenia [
As in first episode schizophrenia, inflammatory abnormalities are noted and suggested to be independent of medication. Evidence therefore suggests an ongoing immune process in patients with schizophrenia, and this population may have a unique inflammatory profile compared to healthy controls [
Some authors have recommended using a Bland-Altman plot (BAP) and 95% limits of agreement (LOA) to assess the testretest reliability [
To date, little data exists on the reliability and agreement of BMIT in measuring cytokine levels. The principle aim of this study is to examine the reliability and agreement of BMIT (17-plex panel) on repeated measurements by using BAPs, 95% LOA, intraclass correlation coefficient (ICC), and coefficient of repeatability (CoR). The secondary aim is to compare the cytokines showing optimal test-retest reliability in patients with schizophrenia with those of healthy controls.
A total of 40 subjects were recruited in Taipei city, northern Taiwan, including 20 patients with schizophrenia and 20 healthy controls. All of the patients were hospitalized in a chronic ward with stable psychiatric condition for more than six months. The control group were enrolled using advertisement, such as leaflets and internet. Patients were interviewed by two certified psychiatrists to confirm the diagnosis of schizophrenia based on the criteria of the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision [
The exclusion criteria were as follows: 1) a history of major medical or neurological disorder; 2) previous head trauma with loss of consciousness; 3) a diagnosis of substance-related disorders; 4) a history of heavy tobacco use or heavy alcohol consumption; and 5) cigarette smoking or alcohol drinking in the most recent three months. Tobacco use and alcohol consumption were not allowed during the study period.
The study procedures were approved by the Institutional Review Board for the Protection of Human Subjects at the Tri-Service General Hospital, a medical teaching hospital within the National Defense Medical Center in Taiwan (TSGH IRB No. 2-103-05-096). Written informed consent was obtained in accordance with the National Health and Medical Research Council guidelines. All participants fully understood the study protocol and all details involved. They were free to withdraw their consent at any time.
Venous blood samples were obtained by venipuncture in the morning between 8:00 and 9:00 AM in fasting status. For the preparation of plasma samples, 10 mL of peripheral blood was drawn into an EDTA-containing Vacutainer Tube and immediately centrifuged to separate the plasma (1,000×g at 4℃ for 10 min). The samples were then placed into NUNC-cryovial tubes and stored at -84℃. For BMIT assays, thawed aliquots were gently vortexed and then centrifuged at 13,200 rpm for 10 min at 4℃ immediately prior to testing. Repeated sampling from healthy controls was performed 14 days later.
The multiplex magnetic bead-based antibody 17-pex panel (Bio-Plex® by Bio-Plex ProTM, Catalog #M5000031YV) was used to analyze samples according to the manufacturer’s instructions on BMIT analyzers (Bio-Plex 200, Genmall Biotechnology Co., LTD., Taipei, Taiwan). The surveyed cytokines were: IL-1β, IL-2, IL-4, IL-5, IL-6, IL-7, IL-8, IL-10, IL-12, IL-13, IL-17, Granulocyte Colony Stimulating Factor (G-CSF), Granulocyte/Macrophage Colony Stimulating Factor (GMCSF), IFN-γ, monocyte chemoattractant protein 1 (MCP-1), macrophage inflammatory protein 1-beta (MIP-1β), and TNF-α. Several cytokines are classified as chemokines, including CC chemokines [CCL2 (MCP-1), CCL4 (MIP-1β), and CXCL8 (IL-8)]. Standard curves were created from duplicate values and all samples were analyzed as single determinations. The coefficient of variation was below 10% for intra-assay and below 15% for inter-assay.
BAP was drawn, which can quantify agreement in test-retest design. Bias was calculated as the mean of test-retest change values, which is the average test-retest difference. Significant bias was considered to exist if the 95% confidence interval (CI) of bias did not include the value of zero or if the Wilcoxon signed-rank test reached significance. 95% LOA for each test-retest difference was calculated as bias ±1.96×the standard deviation (SD) of the paired difference. Agreement between repeated measurements made on the same subject depends only on the within-subject SD. The range of the upper and lower LOA reflected how far apart the repeated measurements differences were likely to be for most individuals. 95% CIs of upper and lower LOA were calculated by taking each limit plus or minus the product of 1.96 and standard error [
Group differences in demographic characteristics were analyzed by Mann-Whitney U tests. Data on cytokines were transformed using the natural logarithms for general linear regression analysis. BMI and age were analyzed as covariates for the general linear regression model. Differences between test-retest measurements in cytokines were examined by the Wilcoxon signed-rank test.
All statistical tests were considered significant if the p value<0.05 (two-tailed). All analyses were performed using the Statistical Package for the Social Sciences software for Windows Ver. 19.0 (IBM Corp., Armonk, NY, USA) and Med-Calc Statistical Software version 18 (MedCalc Software bvba, Ostend, Belgium;
The data of 95% LOA, CoR, and ICC are shown in
This study showed that by using BMIT for cytokine measurement, the within-subject cytokine variation was large, making the test-retest reliability suboptimal by testing ICC. When using BAPs with 95% LOA, most cytokines had acceptable test-retest reliability; only 4.37% of cytokine data were considered outliers, and only three cytokines showed significant bias. Although most cytokine expression in patients with schizophrenia was distinct from that of healthy controls (
A previous study used ICC to examine the test-retest reliability of BMIT in measuring cytokine levels in healthy individuals and patients with major depression. However, the authors found that the levels of cytokines varied differently in health participants during a 30-week interval [
Our findings support optimal test-retest reliability for several cytokines assayed by BMIT, including IL-2, IL-4, IL-5, IL-6, IL-7, IL-8, IL-10, IL-13, IL-17, G-CSF, GM-CSF, IFN-γ, MCP-1, and MIP-1β. For these cytokines, the lines of equality (zero) were within 95% CI of bias, implying reliable agreement over a 14-day test-retest interval. However, the difference of retest on IL-7 needs to be emphasized, as the CoR was relatively large (31.58, 95% CI: 24.16 to 45.61 pg/mL), and the Wilcoxon signed-rank test reached significance. On the other hand, IL-1β and IL-12 appeared to have larger intraindividual variation than those of other cytokines, as both cytokines showed significant cytokine changes on retest and significant bias on BAPs. This means that the results from cross-sectional studies assessing the levels of IL-1β, IL-7, and IL-12 may be unreliable. Previous studies compared the immunoassay results between an enzyme-linked immunosorbent assays (ELISA) and BMIT for cytokine measurements, revealing robust correlations for IL-1β, IL-7, and IL-12 [
The ICC for IL-4 (0.35), IL-6 (0.36), and IL-7 (0.03) in our study showed discrepancy from those of a previous study in which ICC was larger than 0.90 over a 30-day interval [
In the current study, patients with schizophrenia had significantly higher levels of IL-5 and IL-13, and significantly lower levels of IL-4 and IFN-γ, which appeared not to support the schizophrenia theory of T helper (Th)2 shifting (IL-4, IL-5, and IL-13,) and blunted Th1 (IL-2, IL-12, INF-γ, and TNF-α). A meta-analysis reported significant decrease in IFN-γ levels in a chronic schizophrenia subgroup with a large effect size (Hedge’s=-1.07), which was consistent with our findings [
A strength in our study is that we have confirmed the test-retest agreement for numerous important cytokines on BMIT using plasma samples, which can assist further research in exploring these cytokine biomarkers or performing meta-analytical work. However, our findings should be interpreted carefully with consideration of the following limitations. First, our sample size was small, so the 95% LOA and 95% CI of bias were wide. For example, in the levels of IFN-γ, the Wilcoxon signedrank test did not show significant difference on the repeated measurement and the BAP did not determine a significant bias; however, the CoR of IFN-γ was 218.06 pg/mL, with 95% CI ranging from 166.83 to 314.90 pg/mL. Another limitation is that we did not stratify the patient groups by clinical features or treatments due to the small sample size. A previous metaanalysis has analyzed the cytokine differences by subgrouping into first-episode psychosis and medicated patients to eliminate the effects of psychotropics [
In conclusion, this study provides evidence that BMIT has good reproducibility and test-retest reliability on cytokine measurements. Patients with schizophrenia demonstrated aberrant cytokine expression compared to healthy individuals, and their inflammatory profiles can be distinguished by selecting reliable cytokines. Currently, the use of cytokines as potential disease markers is still limited. Knowledge of the stability of plasma cytokines levels based on BMIT will help researchers to better delineate inflammatory profiles between different populations.
The online-only Data Supplement is available with this article at
Funding for this study was provided by Civilian Administration Division of Beitou Branch, Tri-Service General Hospital (TSGH-BT-104-04; TSGH-BT-106-005). The Civilian Administration Division had no further role in study design; in the collection, analysis and interpretation of data; in the writing of the report; and in the decision to submit the paper for publication.
The authors have no potential conflicts of interest to disclose.
Conceptualization: Ta-Chuan Yeh, Chih-Sung Liang. Data curation: Hsuan-Te Chu. Formal analysis: Fu-Chi Yang. Funding acquisition: ChihSung Liang. Investigation: Chia-Kuang Tsai. Methodology: Hsin-An Chang. Project administration: San-Yuan Huang. Resources: San-Yuan Huang. Sofware: Fu-Chi Yang. Supervision: Chih-Sung Liang. Validation: San-Yuan Huang. Visualization: Ta-Chuan Yeh. Writing—original draf: Ta-Chuan Yeh. Writing—review & editing: Chih-Sung Liang.
Bland-Altman plots for repeat measurements of multiplex cytokine assay in 20 healthy controls. The Bland-Altman plots represent the difference of repeated measurements of cytokine levels against the average value for each pair of measurements. The solid line is the mean difference; a mean difference of zero means no change in cytokine levels after a 14-day interval. The two dotted lines represent the 95% limits of agreement, and the four dashed lines are the 95% confidence intervals for the upper and lower limits of agreement. IL: interleukin, G-CSF: granulocyte colony-stimulating factor, IFN-γ: interferon gamma, MCP-1: monocyte chemotactic protein 1, MIP: macrophage inflammatory protein, TNF-α: tumor necrosis factor alpha.
Comparisons of baseline cytokines between patients with schizophrenia and healthy controls. Forest plot shows the cross-sectional cytokine levels (mean with standard deviation). The Y-axis represents different cytokines. The X-axis represents concentration of cytokines (pg/mL). Solid dots represent data from schizophrenia; hollow dots represent data from healthy controls. HC: healthy controls, IL: interleukin, IFN-γ: interferon gamma, SCH: schizophrenia.
Bias, 95% limits of agreement, repeatability coefficient, and intraclass correlation coefficient for 14-day repeated cytokine level (pg/mL) measurements in 20 healthy controls
Cytokines | Baseline | Re-test | p |
Bias | 95% CI of bias | Upper LOA | Lower LOA | CoR with 95% CI | ICC | Outlier |
---|---|---|---|---|---|---|---|---|---|---|
IL-1β | 2.08±1.08 | 1.35±0.95 | <0.01 | -0.73 | -1.18–-0.27 | 1.19 | -2.64 | 2.35 (1.79–3.39) | 0.54 | 2 |
IL-2 | 6.72±9.30 | 5.70±6.92 | 0.45 | -1.02 | -3.27–1.24 | 8.43 | -10.46 | 9.42 (7.21–13.61) | 0.83 | 2 |
IL-4 | 3.60±1.64 | 3.08±1.38 | 0.31 | -0.52 | -1.32–0.29 | 2.87 | -3.9 | 3.45 (2.64–4.98) | 0.35 | 1 |
IL-5 | 4.31±2.09 | 3.44±1.40 | 0.06 | -0.87 | -1.77–0.03 | 2.9 | -4.64 | 4.05 (3.10–5.85) | 0.41 | 1 |
IL-6 | 10.28±5.15 | 8.45±4.58 | 0.28 | -1.83 | -4.42–0.77 | 9.03 | -12.68 | 11.17 (8.54–16.13) | 0.36 | 0 |
IL-7 | 11.28±15.77 | 6.13±2.18 | 0.02 | -5.15 | -12.48–2.19 | 25.56 | -35.85 | 31.58 (24.16–45.61) | 0.03 | 1 |
IL-8 | 15.17±5.63 | 12.88±5.78 | 0.13 | -2.30 | -5.09–0.5 | 9.41 | -14 | 12.26 (9.38–17.71) | 0.45 | 0 |
IL-10 | 22.32±13.27 | 17.88±10.67 | 0.12 | -4.43 | -9.75–0.88 | 17.84 | -26.7 | 23.38 (17.89–33.76) | 0.55 | 1 |
IL-12 | 30.82±17.53 | 22.62±13.87 | 0.02 | -8.2 | -15.52–-0.87 | 22.49 | -38.88 | 33.95 (25.97–49.02) | 0.19 | 0 |
IL-13 | 2.31±1.37 | 2.09±1.39 | 0.11 | -0.21 | -0.62–0.2 | 1.5 | -1.93 | 1.72 (1.32–2.49) | 0.8 | 1 |
IL-17 | 16.44±12.75 | 13.73±13.33 | 0.25 | -2.71 | -7.54–2.13 | 17.55 | -22.96 | 20.44 (15.64–29.52) | 0.69 | 1 |
G-CSF | 25.92±9.87 | 23.12±10.77 | 0.25 | -2.8 | -7.64–2.03 | 17.47 | -23.07 | 20.51 (15.69–29.61) | 0.5 | 0 |
GM-CSF | NA | NA | NA | NA | NA | NA | NA | NA | NA | NA |
IFN-γ | 242.42±112.24 | 227.65±113.89 | 0.60 | -14.77 | -67.72–38.18 | 206.97 | -236.52 | 218.06 (166.83–314.9) | 0.5 | 0 |
MCP-1 | 13.29±3.93 | 12.94±2.98 | 0.84 | -0.36 | -2.15–1.44 | 7.16 | -7.88 | 7.36 (5.63–10.63) | 0.4 | 1 |
MIP-1β | 40.49±9.37 | 40.76±12.66 | 0.81 | 0.26 | -3.76–4.29 | 17.13 | -16.6 | 16.44 (12.58–23.75) | 0.7 | 1 |
TNF-α | 13.24±7.75 | 9.7±5.61 | 0.05 | -3.54 | -6.79–-0.29 | 10.08 | -17.16 | 14.98 (11.46–21.62) | 0.47 | 2 |
test-retest differences were examined by Wilcoxon signed-rank test; outlier was defined as paired samples value exceeding range of CoR and 95% LOA; and cytokines with bold text indicate significant changes in re-test measurement or 95% CI of bias across zero.
CoR: coefficient of repeatability, ICC: intraclass correlation coefficient, IL: interleukin, G-CSF: granulocyte colony-stimulating factor, GM-CSF: granulocytemacrophage colony-stimulating factor, IFN-γ: interferon gamma, LOA: limit of agreement, MCP-1: monocyte chemotactic protein 1, MIP: macrophage inflammatory protein, TNF-α: tumor necrosis factor alpha
Demographic data and baseline levels of cytokines and chemokines in patients with schizophrenia and healthy controls
Schizophrenia (N=20) | Healthy controls (N=20) | p |
|
---|---|---|---|
Age | 46.60±8.88 | 28.05±3.59 | <0.01 |
BMI | 25.49±4.33 | 23.11±1.76 | 0.01 |
IL-1β | 3.02±3.18 | 2.08±1.08 | 0.15 |
IL-2 | 3.77±3.87 | 6.72±9.3 | 0.67 |
IL-4 | 1.69±0.63 | 3.60±1.64 | <0.03 |
IL-5 | 9.96±3.59 | 4.31±2.09 | <0.01 |
IL-6 | 8.61±5.48 | 10.28±5.15 | 0.69 |
IL-7 | 7.17±3.12 | 11.28±15.77 | 0.87 |
IL-8 | 8.58±3.97 | 15.17±5.63 | 0.05 |
IL-10 | 10.27±7.91 | 22.32±13.27 | 0.10 |
IL-12 | 8.95±5.43 | 30.82±17.53 | <0.01 |
IL-13 | 13.10±11.26 | 2.31±1.37 | <0.01 |
IL-17 | 18.83±14.01 | 16.44±12.75 | 0.43 |
G-CSF | 23.88±13.31 | 25.92±9.87 | 0.54 |
IFN-γ | 75.30±52.66 | 242.42±112.24 | <0.01 |
MCP-1 | 17.21±10.85 | 13.29±3.93 | 0.62 |
MIP-1β | 31.46±15.5 | 40.49±9.37 | 0.19 |
TNF-α | 50.16±101 | 13.24±7.75 | 0.01 |
age and BMI were analyzed by Mann-Whitney U tests; cytokines (pg/mL) were transformed using the natural logarithms and analyzed by a general linear model with covariates of age and BMI.
BMI: body mass index, IL: interleukin, G-CSF: granulocyte colony-stimulating factor, GM-CSF: granulocyte-macrophage colonystimulating factor, IFN-γ: interferon gamma, MCP-1: monocyte chemotactic protein 1, MIP: macrophage inflammatory protein, TNF-α: tumor necrosis factor alpha