Although stress can have some positive effects, such as providing energy and motivation and sharpening our focus, chronic stress can be detrimental. Psychological stress has been linked to numerous health issues, including mental disorders [1], cardiovascular diseases [2], cancers [3], and autoimmune diseases [4]. Additionally, work-related stress can lead to employee burnout [5] and negatively impact an organization’s productivity [6]. Therefore, managing stress is essential for enhancing long-term quality of life. Regular stress monitoring can help identify when active self-care or professional assistance is necessary. However, stress diagnosis is subjective, which poses challenges in developing a universally applicable diagnostic model [7]. Clinically, stress levels are typically measured through self-reported, expert-verified questionnaires, such as the Perceived Stress Scale [8], or visual analog scales, such as a stress thermometer [9]. Nonetheless, these methods are subjective and may not provide a consistent measurement scale for stress levels.

Extensive efforts have been made to develop biological markers that objectively and consistently measure stress, including metrics such as blood pressure, heart rate variability, electroencephalogram signals, and finger temperature [7]. Since the early 1980s, speech and voice have also been recognized as valuable biomarkers, providing probabilistic information about mental health [10]. Stress impacts both linguistic and non-linguistic speech characteristics [11]. Linguistic features—such as vocabulary diversity, lexical and grammatical complexity, syntactic structures, and semantic skills—can indicate cognitive impairment [11]. However, these features can be influenced by an individual’s educational and social background [12]. Non-linguistic features pertain to how speech sounds are produced through the modulation of muscle tension and respiration [13]. Increased stress can raise both muscle tension and respiration rates, thereby altering vocal sounds [14]. Recent developments in speech-based vocal stress models include the work of Lu et al. [15], who developed StressSense. This system uses pitch features, spectral centroids, high-frequency ratios, speaking rates, mel-frequency cepstral coefficients (MFCCs), and nonlinear features derived from the Teager Energy Operator [16]. They trained classification models using Gaussian mixture models (GMMs) for predetermined acoustic feature sets. Similarly, Sondhi et al. [14] analyzed stress using seven acoustic features: pitch, shimmer, four formants, and mean pitch. Traditional methods for building prediction models for voice data have relied on a variety of statistical approaches. For example, GMMs model the distribution of the voice feature vectors including MFCCs, to capture the characteristics of speech sounds. Support vector machines (SVM) are utilizing hyperplanes to separate different classes in high-dimensional space of numerous acoustic features. Since the traditional approaches are limited in handling the complexity of voice data, they often struggle with the variability and complexity of real-world speech. On the other hand, deep learning models can identify complex patterns and features in voice data that traditional models often miss, leading to higher accuracy and performance in voice recognition tasks [17]. Deep learning models incorporate various architectures like Convolutional Neural Networks (CNNs) [18] and Recurrent Neural Networks (RNNs) [19] to handle diverse and complex voice signals, improving robustness and adaptability to various conditions. Additionally, these models can better capture temporal and contextual information in voice data, that is difficult to account for with traditional models. Recent deep learning models including Emphasized Channel Attention, Propagation and Aggregation in Time delay neural network (ECAPA-TDNN) [20] use sophisticated feature extraction techniques and attention mechanisms such as Squeeze and Excitation blocks [21] to focus on the most relevant parts of the audio signal. This targeted approach improves the model’s precision and discriminative power. Despite significant advances in deep learning modeling of speech signals, its application for stress modeling has been relatively rare. Additionally, most models have been developed for Western populations, and few vocal stress models have been tailored for Koreans.

In this study, we aimed to develop stress prediction models for vocal biomarkers using an in-house dataset generated from a Korean clinical study. Recently introduced high-performance models based on deep learning structures, capable of speaker recognition from speech recordings, have been explored [20,22-25]. By leveraging these speaker-verification model architectures, we aim to enhance the accuracy of detecting individual stress fluctuations by accurately capturing unique voice characteristics.

In total, 130 participants were recruited for this study. We selected data from those who exhibited sustained stress, as indicated by self-reported stress thermometer readings or cortisol levels. Data from 15 individuals who showed no signs of stress after induction were excluded from further analysis. A significant increase in either the stress thermometer score or cortisol level was considered evidence of successful stress induction. The demographic characteristics of the 115 participants are presented in Table 1. Of these, 33.91% were male. The mean age was 33.56 years (standard deviation [SD], 7.90) at SNUBH and 37.71 years (SD, 10.18) at BMC.

The distribution of stress thermometer scores and cortisol levels is illustrated in Figure 3 at two time points: before (T0) and after (T1) stress induction. The mean stress thermometer scores at T0 and T1 were 1.304 (SD=1.482) and 2.896 (SD=2.253), respectively. The mean cortisol levels were 0.165 (SD=0.083) at T0 and 0.322 (SD=0.21) at T1. Both differences were statistically significant, with a p-value <0.0001 in the Mann-Whitney test.

After completing the stress-induction experiments and isolating participants’ voices from the interviewer’s voices and background noises, the average length of the voice recordings was 74.86 seconds (SD=8.89) for script reading and 125.89 seconds (SD=65.17) for interviews. The preprocessed voices were used to train the stress prediction models and test their performance.

In preliminary studies, the distribution of prediction scores for each acoustic feature across three different model architectures was investigated: CNN [18], conformer [24], and ECAPA-TDNN. To assess the stress level of the participants, we computed stress prediction scores for each individual segment, made a simple binary decision for each segment, and finally determined the stress level based on the ratio of predicted states from T0 to T1. The ECAPA-TDNN model exhibited the best prediction performance at 77.5%, followed by the CNN at 60%, and the conformer at 62.5%. Consequently, we selected the ECAPA-TDNN model to compute the stress status prediction scores. These prediction scores, associated with stress status, are hereafter referred to as the stress score.

To determine whether trimming the beginning or end of a voice recording could affect the accuracy of stress status detection, we examined the trend of stress model scores throughout the speech. Scatter plots of the scores against time, supplemented by smoothing splines for 20 samples from the test dataset, are presented in Figure 4 to assess the overall trend along the time axis. Although some participants exhibited a linear trend in their stress scores, we did not observe a consistent pattern that would justify the removal or selection of specific parts of the voice recordings across all participants. These varied patterns of stress score changes likely highlight interindividual variability. Therefore, full-length speech recordings were used in the subsequent stress detection tests.

Additionally, we compared the characteristics of stress scores derived from script reading and free speech in interview responses. We estimated the impact of stress induction on the scores for each type of voice recording and compared the magnitude of these impacts to determine which recording type provided the most distinctive information. When comparing T0 and T1, we used mixed-effects models that accounted for unique score variances among individuals by integrating an individual-specific random-effects term to address individual variability. After adjusting for individual effects by including the individual as a random effects covariate, the mean difference in the distribution of stress scores between the two time points was estimated to be greater for free speech (0.148 for recording type 2) compared to script reading (0.132 for recording type 1). Although both results showed significant p-values (p<0.001), the effect size and z-score of the test statistics were greater for free speech. These results indicate that the differences in stress scores between the two time points within each individual were more pronounced for free speech than for script reading, suggesting that voice recordings of free speech are more effective for distinguishing between relaxed (T0) and stressed (T1) states.

The distribution of scores for free-speech voices at T0 and T1 for the 20 test samples was further explored using box plots (Figure 5). Although the stressed state tended to have higher median scores, there was no complete separation between the relaxed and stressed states in the segment-wise stress scores, with an overlapping range across the two states. Among the 20 participants in the test dataset, BMC032, SNUBH042, and SNUBH048 showed significant differences in the median scores between T0 and T1. However, 6 participants, including BMC052, SNUBH043, and SNUBH062, showed little difference. The statistical significance of the differences between T0 and T1 for each individual was tested using the one-sided Wilcoxon rank test, a nonparametric method for group comparisons. Of these, 14 participants showed increased stress levels. In summary, through a two-step procedure—computing stress scores based on a deep learning model for free speech recordings and then conducting a statistical test for two-group comparisons—we correctly detected the stress status in 70% of the test participants. The test statistics and p-values are listed in Supplementary Table 1.

We collected data from 130 participants and obtained 115 datasets. We developed a deep learning model to compute stress-associated scores using sound frequency features, MFCCs, and the speaker identification model architecture, ECAPA-TDNN [20]. The participants’ voices were captured using two methods: reading a script and answering one or two randomly selected neutral questions. In our statistical analyses, the stress scores computed using voice signals from free speech (answering questions) showed a stronger association with an individual’s stress status. As we observed that the distribution of individuals’ stress prediction scores differed in both shape and location, we implemented a comparative test procedure that analyzed two sets of stress scores from the same individual to determine their stress status.

There are many studies on the computational modeling of stress, most of which use physiological and behavioral information requiring special sensor devices, or psychological information that could be subjective [7]. Our model uses only voices, which can be collected using personal devices such as mobile phones, tablets, smartwatches, or computers. In a prior study, Han et al. [40] developed a vocal stress classification model using long short-term memory (LSTM). They trained the model using 25 subsampled voice segments, each 4 sec long, from the same 25 participants and achieved an accuracy of 66.4% for the Korean language [40]. We adopted the latest advanced deep-learning architecture for speech models using speaker identification, which is sensitive to subtle differences in voice signals. Consequently, our algorithm for stress detection achieved an accuracy of 70% when evaluated using a test dataset that had been set aside prior to model development. As performance was evaluated using independent samples, the accuracy results provide a more rigorous measure of external validity or generalizability.

The coronavirus disease-2019 (COVID-19) pandemic has significantly stressed populations worldwide. Historically, the detrimental effects of pandemic infectious diseases on mental health are expected outcomes [41]. Concurrently, COVID-19-related mandates such as quarantine and social distancing have increased the demand for digital healthcare technologies to enhance mental health [42]. Monitoring various personal health conditions using digital devices enables personalized and precise management through self- or remote digital healthcare services. Measurements related to behavior, physiological processes, and other health aspects are referred to as digital biomarkers [43]. Our voice-based stress model is a digital biomarker compatible with various mobile devices, including smartphones and tablets. Using our vocal stress test procedure, periodic monitoring of mental stress levels can be conducted anywhere. Longitudinal records of the stress test results can serve as an objective history to provide a relative quantification of stress levels. Unlike traditional clinical questionnaires, which require introspective answers, our method potentially increases compliance by reducing inconvenience.

Our stress score model can be integrated into healthcare applications to assess mental stress. The procedure for testing mental stress levels is illustrated in Figure 6. Upon initial use, the application prompts the user to record their voice in a stress-free and relaxed state to establish a baseline reference. To capture the users’ free-speech voices, they are presented with one or more randomly selected questions from a predetermined list of neutral questions. The recording lasts for at least 1 min. The stress score is computed using the model stored in the application, and these values are set as user references. Subsequently, users can gauge their stress levels by recording their voices in response to questions. Although the score itself correlates with mental stress, the values derived from speech recordings can vary among individuals based on their speaking style. To adjust for this variability, we introduced a statistical test based on user-reference values, which enhances the detection accuracy of elevated stress levels.

The development of stress biomarkers and mobile applications offers several advantages: 1) they enable employees to monitor individualized stress patterns encountered in daily life, 2) they enhance the potential to prevent mental disorders related to stress, such as depression, and physiological conditions like cardiovascular disorders at a preclinical stage, and 3) they provide immediate stress-monitoring tools for clinical populations in need.

One concern regarding mobile health solutions is safeguarding personal information and ensuring user privacy. However, as the stress score model can be downloaded directly from the service provider, stress assessment can be performed without transmitting personal data. Using smartphone applications for stress management facilitates cost-effectiveness and consistent tracking.

Our study’s sample size was small. Future studies with larger sample sizes are required to further enhance the proposed model’s performance. Recent research has demonstrated that transfer learning, particularly using self-supervised models trained on public datasets, can improve performance across various tasks, even with a limited number of analysis samples. Thus, such advanced modeling approaches will be adopted in our future studies.

In conclusion, our study highlighted voice biomarkers as potentially effective digital tools for measuring stress levels. Using a deep learning model, we demonstrated that our method can detect stress with considerable accuracy. Moreover, this study introduces linguistically tailored digital health solutions, emphasizing the utility and novelty of using the Korean language as reliable stress biomarkers. The practical implications of this research are substantial, especially during times of challenge, such as the COVID-19 pandemic, when the demand for remote mental health monitoring surges.

Integrating this voice-based system into mobile health platforms allows individuals to track their mental stress conveniently and privately. Although our findings are promising, the sample size was limited. Future studies with larger sample sizes are crucial to refine the effectiveness of the system. We also plan to apply advanced modeling techniques, such as transfer learning, to further improve model performance.