The Economic Impact of Providing Evidence-Based Pediatric Mental Health Literacy Training to Primary Care Physicians

Article information

Psychiatry Investig. 2021;18(7):695-700
Publication date (electronic) : 2021 July 22
doi :
1Alberta Health Services, Child and Adolescent Addiction and Mental Health and Psychiatry Program, Calgary, Canada
2Department of Psychiatry, The University of Calgary, Cumming School of Medicine, Calgary, Canada
3Department of Paediatrics, The University of Calgary, Cumming School of Medicine, Calgary, Canada
4Department of Community Health Sciences, The University of Calgary, Cumming School of Medicine, Calgary, Canada
Correspondence: David Cawthorpe, BSc, MSc, PhD Department of Psychiatry, The University of Calgary, Cumming School of Medicine, 2nd Floor, Richmond Road Diagnostic and Treatment Centre, 1820 Richmond Rd SW, Calgary, AB T2T 5C7, Canada Tel: +1-403-955-8760, E-mail:
Received 2021 January 26; Revised 2021 March 25; Accepted 2021 May 9.



This paper presents a review of the current state of child and adolescent mental health literacy and provides current evidence of the economic impact of a pediatric mental health literacy (MHL) training program.


Employing a case-series-comparison design, physician referrals to urgent and specialized mental health services were linked with patient-specific information comparing referrals from MHL participants and non-participating physicians. The economic impact analysis was based on changes in the admitted referral frequency and lengths of stay for the MHL group, compared to themselves pretraining, and over the same time period compared to non-participating physicians.


Average scheduled ambulatory admission rates per physician remained constant for trained and untrained pre-post groups. Average scheduled ambulatory admission wait time and length of stay reduced significantly post-training for MHL-trained physicians compared to pre-training and untrained physicians. In addition to reductions in length of stay, the total bed costs saving for emergency/inpatients admission deferrals was $2,932,112 or about $20,000 per MHL-trained physician.


The estimated economic impact of the MHL training shows a substantial return on investment and supports wider implementation. The MHL training program should be a key feature of mental health reform strategies, as well as continuing and undergraduate medical education.


In this paper, we present a review of the current state of child and adolescent mental health literacy (MHL) targeting primary-care physicians and an economic analysis of implementing an evidence-based mental health literacy program developed by the REACH Institute (, implemented locally under the name CanREACH: Canadian Research and Education for the Advancement of Child Health. Launched in May 2014, CanREACH delivered the MHL training with the goal of building skills and confidence in primary care physicians in southern, Alberta (Calgary and surrounding area) in respect to screening, assessment and diagnosis, in-practice treatment, and when necessary, making referrals to tertiary mental health care (e.g., inpatient and ambulatory regional child mental health services). The CanREACH continuing medical education (CME) initiative has expanded dissemination throughout Alberta with uptake in several other Canadian provinces.

Mental health disorders are costly to all systems of care. Existing evidence points to the greatest return on mental health investment occurring with services aimed at children and youth [1], which is not surprising given that seventy percent of mental health problems have onset during childhood [2]. Access to services is imperative with investments to build capacity by delivering mental health services where children naturally come into contact with community professionals, such as family physicians. Primary care physicians are optimally positioned to monitor children’s mental health status across development; however, primary care providers report a lack of training and confidence in relation to pediatric mental illness [3]. Inadequate identification and intervention in primary care clinics results in deterioration of youths’ mental health to the point of crisis, often requiring hospital emergency care [4]. Continuing medical education (CME) is a nexus for health care innovation, yet delivery barriers often prevent the spread and sustainable CME uptake. Despite awareness that utilizing CME strategies targeting provider behaviors is optimal, there remains a lack of theoretical frameworks and a reliance on minimally effective evaluation techniques, such as pre- and post-training surveys of self-reported knowledge and attitudes [5].

In respect to scoping practice knowledge of pediatric mental health, primary care practitioners often report little or no undergraduate or graduate training, yet self-report competence [6]. The effectiveness of the CanREACH training has been measured in the practice field [7], finding that the MHL training program measurably improves the quality of referrals to regional child mental health services [7]. Objective standard screening measurement of participating physicians’ referrals by blind third-party intake workers at regional mental health services indicated improved referral quality post-MHL training compared to pre-MHL training and in comparison to non-participating physician referral [7].

The main objective of this current paper is to examine the economic impact of the CanREACH initiative under the following hypothesis: compared to pre-MHL training and to un-exposed physicians, exposure to CanREACH would lead to cost reductions in terms of the post-training participant referrals to regional child mental health services.


Program description

The CanREACH program based in Calgary Alberta, Canada implemented an evidence-based CME, “Patient-Centered Mental Health in Pediatric Primary Care (PPP)” as developed by the REACH Institute, into the Canadian context in efforts to build capacity in primary care providers with regards to pediatric mental health [7]. The CanREACH Mental Health Literacy Training Program is an interactive 6-month mini-fellowship that is delivered by two parts: 1) A three-day face-to-face training workshop employing a combination of intensive teaching and hands-on practice; participants receive custom-designed toolkits with guides, assessment instruments, and other resources and complete evaluations of the training units and conference overall, and 2) a six-month clinical care distance learning teleconference calls: ongoing, case-based coaching using distance learning methods (twice-monthly one-hour peer-learning conference calls, and supporting web resources); completed for a six-month period following the conference. The content focuses on mental health concerns relevant to the primary care setting, such as ADHD, Anxiety, Depression (mood disorders) and Aggression, and details the screening, comprehensive assessment and multi-modal treatment approaches as informed by evidence-based best practices for each. For instance, general screening is recommended and practiced using a tool such as the Pediatric Symptom Checklist (PSC) to facilitate the recognition of cognitive, emotional, and behavioral problems; following which a streamlined approach guides the primary care provider in further assessment and intervention areas (i.e., for high externalizing scores further exploration into ADHD may be warranted, how to do this, and the application of Canadian guidelines via CADDRA). The learning in each of these diagnostic areas is enhanced by using practical application both within the conference, and again longitudinally with the application of the learnings to clinical setting.

CanREACH received continuing medical education accreditation approval from both the Royal College of Physicians and Surgeons of Canada and the College of Family Physicians Canada. From 2014 to November 2019, 276 trained physicians were included in this study.

Ethics approval

The research proposal underpinning the evaluation of the CanREACH impact was granted ethics approval by the Calgary Conjoint Health Research Ethics Board (REB14-0566). CanREACH participants provided informed consent to have their referrals tracked. Data from non-participating physician referrals was collected under waiver of consent. All data was anonymized and encrypted.

Data collection

The CanREACH PPP 6-month mini-fellowship CME was delivered in its entirety, including both the face-to-face training workshop and the teleconference calls, to ten cohorts of physician participants (n=276) from May 2013 to May 2019, with data collection up to November 2019. Participants provided informed consent permitting tracking the referral made within the Alberta Health Services Calgary Health Zone to the Child and Adolescent Addictions and Mental Health and Psychiatric Program (CAAMPPH). CAAMHPP has a regional access and intake system (RAIS) for tracking all referrals and admissions to emergency, inpatient, and ambulatory services, including wait time (days) and length of stay (days). The RAIS system process, utility, and impact on CAAMHPP capacity and community program evaluation has been described in detail elsewhere [8]. For the purpose of this analysis, data extracted from the RAIS system formed the basis of comparison of referral frequency and disposition within CAAMHPP served as the basis of evaluating the economic impact of the CanREACH program. Comparison groups formed naturally based on pre-MHL and post-MHL literacy exposure referral rates per CanREACH participant, as well as the referrals based on grouping the unexposed non-participating physicians over the same time period. Both groups had similar lead-in periods pre-CanREACH, so that before and after comparisons could be made for both participating and non-participating physicians groups. Participating and non-participating physician referral data were extracted from the regional access and intake system. Of the trained physicians making referrals to the CAAMHPP over the period of study, 62 made referrals before training and 66 made referrals after training, permitting the pre-post basis of comparison.

Measurement of post-training outcomes was represented by measures of physician referral frequency, wait times (days) and lengths of stay (days) extracted from the RAIS [8]. This database tracks the referral sources of registrations and tracks admissions to the Child and Adolescent Addictions Mental Health and Psychiatry Program (CAAMHPP) in Calgary, Alberta. Extracted data permitted grouping uniquely physician referrals in the basis of CanREACH training participation and non-participation. The admitted referrals from participating physicians were compared to those of physicians not participating in the CanREACH training program, who nevertheless made referrals to child and adolescent mental health services in Calgary during the study period.

Data collection of all physician referrals was constrained for participant groupings to 1 year before the first training in May 2014 (program commencement) for the first and similarly for each subsequent cohort and up to the time that concluded at 6-months post-training interval (November 2019) for the last cohort trained in May 2019 (Table 1). Pre-training cutoff date was the date that CanREACH training commenced. The untrained group was constrained from 5 years before the first training in May 2014 up to November 2019, six months after the May 2019 training. For ease of comparison the non-participating, untrained physician group was broken into two temporal groups with midpoint between May 2009 and November 2019, so that the comparison of long-term referral patterns of non-participants could be examined and the second 2014–2019 group of non-participants was of the same duration as the CanREACH program.

Description of the trained cohort


The RAIS data extracted for this study included admitted referrals, level of care (emergent/inpatient and scheduled ambulatory), length of stay, and wait times linked to referring physician identifiers and constrained based on the time periods described above.

Data summary and analysis

Comparison pre-post referrals rates within the trained group of physicians, as well as in comparison to the untrained group of physicians also grouped into two time frames formed the basis of analysis. The emergency/inpatient and ambulatory costs noted in the results were based on standard estimates obtained from the Alberta Health Services Finance Department.

The physician specializations were summarized (Table 1). Descriptive statistics by physician group with comparison of 95% CIs to show the admission frequency, wait times, and length of stay of admitted referrals made to CAAMHPP by participating physicians in comparison to these same physicians before training and to non-participating physicians over the same time period (Table 2). Between group statistical significance was based on comparison of 95% confidence intervals between groups, with significance (p<0.05) based on non-overlapping 95% confidence intervals (CI). Non-significant differences are noted in the text where between group comparisons of averages and ranges (e.g., referral/admission rates) are presented.

Sample size, means, upper and lower 95% CIs of by group comparing wait times and lengths of stay

Economic impact analysis

The economic impact assessment of the CanREACH program was based on the actual cost of admissions via emergency to inpatient services based on the summed bed-days of patient lengths of stay (~$870CDN per diem.) and the estimated normative total cost of the typical 8 ambulatory visits (~$2000CDN) per admission. The values were used to estimate the differences (increase or decrease per physician) in ambulatory referral rates pre/post-training. Overall, physicians made relatively few referrals to emergency. Emergency referrals are often subsequently admitted to inpatient services. The cost of these admissions was substantial. The admissions to inpatient services via emergency were based on the actual inpatient lengths of stay made by untrained physicians and CanREACH physicians pre-training for equal periods before and after the commencement of the CanREACH program in 2014. The post-training CanREACH physician inpatient admissions dropped to zero. This comparison made the assumption that, if the untrained physicians had been trained, the post-training effect would be the same for them.


Description of the trained cohort

Ninety percent of the sample were physicians, primarily from the Calgary region of southern Alberta, although some physicians came from other regions in Alberta and other provinces in Canada. The remainder of the trainees were allied professionals. Table 1 shows the distribution of physician types. Note that the trained total of physicians attending the CanREACH CME was considerably higher than the numbers actively making referrals to CAAMHPP services both pre- and post-training and much lower in total than the referrals made by nonparticipating physicians, even though CanREACH physicians’ admitted referrals were greater on average (Table 2).

Admission rates by groups

Scheduled ambulatory admissions

There were 3,081 scheduled ambulatory admissions from 958 untrained physicians between 2009 and 2013 with an average of 3.2 per physician and a range of 1 to 94. There were 4,104 scheduled ambulatory admissions from 1,257 untrained physicians between 2014 and November 2018 with an average of 3.3 per physician and a range of 1 to 70. That the rate of scheduled ambulatory admissions from untrained non-participating physicians remained constant on average indicates that the referral behavior of this group was stable over the study period.

There were 416 scheduled ambulatory admissions from 62 CanREACH physicians pre-training between 2009 and 2013 with an average of 6.7 per physician and a range of 1 to 150. There were 396 scheduled ambulatory admissions from 66 CanREACH physicians post-training between 2014 and 2018 with an average of 6.0 per physician and a range of 1 to 123. CanREACH physicians made twice as many admitted referrals compared to untrained physicians. Pre- and post-training CanREACH physicians had the same rate of admitted referrals on average, although there was a slight downward shift in the number, average, and range of referrals post-training. The total cost savings the estimated for scheduled ambulatory referrals, a difference of 20 fewer admitted referrals was a marginal $40,000CDN or $145CDN per trained physician.

Inpatient admissions via emergency

There were 64 inpatient admissions via emergency from 60 untrained physicians between 2009 and 2013 with an average of 1.1 per physician and a range of 1 to 2. There were 22 inpatient admissions via emergency from 21 untrained physicians between 2014 and 2018 with an average of 1.0 per physician and a range of 1 to 2. There were 7 inpatient admissions via emergency from 5 CanREACH physicians between 2009 and 2013 with an average of 1.4 per physician and a range of 1 to 2. There were 0 inpatient admissions via emergency from CanREACH physicians between 2014 and 2018.

Table 2 presents admitted referrals comparing the pre-training CanREACH group to the post-training CanREACH and untrained physicians. The post-training CanREACH group admissions had lower wait times and lengths of stay on average.

Economic impact

The scheduled ambulatory admission rate for participating CanREACH physicians was greater than that of untrained physicians, but their pre/post-training scheduled ambulatory service admission rate did not change significantly per trained physician, however the average lengths of stay of patients referred from CanREACH physicians post-training were significantly reduced compared to pre-training, indicating an overall economic saving in addition to the value calculated based on the estimated typical ambulatory admission cost ($2,000CDN) associated with average eight visits per patient. Inpatient admissions were no different comparing CanREACH physicians pretraining and untrained physicians. These two groups were no different than the untrained physicians’ inpatient admission rate during the period of time after CanREACH implementation. Inpatient admissions via emergency dropped to zero in the CanREACH physician group post-training. The economic impact can be assessed based on the assumption that were the 22 inpatient admissions via emergency from 21 untrained physicians between 2014 and 2018, and if trained, then these admissions may not have happened. The actual 3,656 cumulative bed-days of these admitted patients at a very conservative cost estimate of $802CDN per diem, which totaled to $2,932,112CDN. A value that may have been saved were these physicians trained and able to better manage care in the community. This total value was close to the total five-year CanREACH program implementation cost. There were 276 physicians trained within the catchment at the time of this valuation, giving a savings of approximately $20,000CDN per trained physician, even though only a minority of the trainees made referrals to CAAMHPP.


Trained physicians had made pre-training and/or post-training referrals to CAAMHPP services at rates double that of the untrained group. This may reflect pre-existing interest in child and adolescent mental health that influenced self-selection into the CanREACH program. Even though the admitted referral rate was higher, the wait times and lengths of stay reduced significantly, perhaps reflecting unmeasured CanREACH training benefits worthy of follow-up in subsequent study, such as better liaison with the system of care leading to improved follow-up continuity.

The reduced lengths of stay are measured impact on child and adolescent mental health services. The reasons may be related to the quality of the referrals made post-training. For example, the CanREACH program has been shown previously to measurably improve the quality of referrals to CAAMHPP [7], being that the referrals were found as measured blind to be significantly more appropriate. Measurement in clinical practice informs education. The measurement system employed in the CanREACH evaluation has been shown to integrate well into clinical practice and, in and of itself, to be valid, reliable, and economical [8-10]. This method of clinical practice measurement has been shown to inform guidelines in respect to clinical screening, assessment, and illustrated the need for a practice paradigm shift in respect to trauma-informed care assessment, diagnosis and treatment planning [11,12]. In addition to mapping onto the recent provincial SMART goals (Specific, Measurable, Attainable, Rewarding, and Timely),a the method underpinning the clinical measurement in this paper has been generalized into a model applicable to any division of medical practice and education [13], a model recently reviewed in Family Medicine [14].

A critical piece of any strategy building capacity to improve mental health care services delivery to children and youth in the community needs to focus on the settings that young people naturally visit, such as during regular annual visits to primary care physicians. It is reasoned that improved coordination and communication between primary health care and other health care providers (including hospitals and community service agencies) could reduce avoidable hospitalizations in pediatric mental health thereby improving outcomes for children and youth.

CanREACH, recipient of an Alberta Health Services President’s Excellence Award for Research and Innovation (2019), has emerged as an evidence-based, effective CME capable of building capacity in primary care through educational training. This conclusion is supported by the measured results that point to more appropriate service utilization for patients including but not limited to reduced referrals to emergency, lower wait times, and lower lengths of stay presented in this report, together with improved referral quality 7 as measured employing the system noted above [10].

CME remains an elective process. Until the presently delivered form of child mental health primary care education becomes standard content in medical training, the situation underpinning the observed results in the untrained group may persist. Furthermore, many more physicians enrolled in and completed the CanREACH program than referred to CAAMHPP services. For instance, many physicians attend CME for the purpose of earning their annual credits and retaining licensure. Only about 24% of the participating physicians were evaluable as they had made referrals to child and adolescent mental health services over the last 18 years. This ratio is perhaps a primary index of the state of the art in child and adolescent mental health education in primary care, especially given the essential centrality of youth mental disorder and lifespan health [15-17].


Many of the participating physicians did not make referrals to CAAMHPP, hence the effect of training on their practice behavior was unknown, even though all participants reported similar knowledge uptake and commitment to employ in practice the assessment and treatment skills learned during the training program together with the resources provided to them. With provincial expansion of the CanREACH program, even though the RAIS system does not extend beyond the Calgary Health Zone, there is the possibility that a more comprehensive evaluation may be accomplished that takes into account all children and adolescents served by participants and matched controls. This population-based evaluation scheme could, with comparable sample grouping of participating and matched non-participating physicians, together with their served children and adolescents, provide evaluable information, if not more extensive and detailed results. Such information, including all health visits, prescription, clinical testing and lab results, etc. is available and may employed to examine the long term outcomes of the sampled children and adolescents, based on the status the trained physicians. In support of this scheme is a population health index, based on the presence or absence of mental disorder [16,17].

An additional limitation is that the reduction in referrals as an outcome was based on the assumption that were the non-participating physicians already making referrals to CAAMHPP to have received CanREACH training, the outcome would have been the same. In essence this is a comparison of an absence, even though the design was similar to a quasi-experimental case-comparison intervention design examining symptom reduction. However, this study examined referral admissions from physicians exposed to a training program representing the intervention. Three observations support the findings based on this central assumption. First, the improvement in referral quality [7] and second, the reduction of length of stay reported in this study, suggesting improved follow-up and continuity of care, and finally, the convergence of these results with the self-reported intentions of participants to enact what they had learned [7]. Future research along the lines of the population-based evaluative scheme presented above is required.

In summary, the body of evidence taken together indicates that the CanREACH program is effective and supports the wider dissemination of the original Reach Institute’s PPP program with fidelity [18]. The health information is readily available to support the more extensive population-based evaluation plan outlined above in support of the provincial implementation of the CanREACH program.




The authors have no potential conflicts of interest to disclose.

Authors’ contribution

Conceptualization: Eden McCaffrey, Samuel Chang, Geraldine Farrelly, Abdul Rahman, David Cawthorpe. Data curation: Eden McCaffrey, David Cawthorpe. Formal analysis: David Cawthorpe. Methodology: Eden McCaffrey, David Cawthorpe. Supervision: Eden McCaffrey. Validation: Eden McCaffrey, David Cawthorpe. Project administration: Eden McCaffrey, Samuel Chang, Geraldine Farrelly, Abdul Rahman, Blair Ritchie, Roxanne Goldade. Writing—original draft: Eden McCaffrey, David Cawthorpe. Writing—review & editing: all authors.


1. Friedi M, Parsonage L. Promoting Mental Health and Preventing Mental Illness: the Economic Case for Investment in Wales Cardiff: Public Health; 2009.
2. Messonnier ML, Corso PS, Teutsch SM, Haddix AC, Harris J. R. An ounce of prevention ... what are the returns? Second edition, 1999. Am J Prev Med 1999;16:248–263.
3. Tong G, McLeod-Macey J. Children and Youth Mental Health Survey: Getting Help in Ontario: Children’s Mental Health Ontario (CMHO) Toronto: Ipsos; 2017.
4. Ontario Children’s Mental Health. Backlog at Community Based Child and Youth Mental Health Centers Sends Kids to Hospital at Alarming Rates: Capacity in Community-Based Centers Eroded by 50% over the Last Two Decades Toronto: Evidence Exchange Network for Mental Health and Addictions; 2017.
5. Henny KD, Duke CC, Sutton MY. Uptake of online HIV-related continuing medical education training among primary care providers in Southeast United States, 2017-2018. AIDS Care 2020;:1–10.
6. Cawthorpe D. Primary care physician ability to identify pediatric mental health issues. Can Child Adolesc Psychiatr Rev 2005;14:99–102.
7. McCaffrey ESN, Chang S, Farrelly G, Rahman A, Cawthorpe D. Mental health literacy in primary care: Canadian Research and Education for the Advancement of Child Health (CanREACH). Evid Based Med 2017;22:123–131.
8. Melathopolous K, Cawthorpe D. Central intake development and system change impact on per capita child and adolescent mental health discharges from 2002 to 2017: implications for optimizing system design based on shaping demand. Perm J 2019;23:18.215.
9. Novick J, Cawthorpe D, McLuckie A. The validation of the Western Canada Waiting List Children’s Mental Health-Priority Criteria Score Instrument: 2002-2015 results. J Hosp Adm 2016;5:1–8.
10. Novick J, Cawthorpe D, McLuckie A. A measurable treatment plan: Using the Children’s Global Assessment and the Problem Severity scales as outcomes of clinical treatment. J Hosp Adm 2016;6:9–15.
11. Rahman R, Kuntz J, Deegan A, Perri A, Cawthorpe D. On becoming trauma-informed: validation and utility of the Adverse Childhood Experience Survey in tertiary child and adolescent mental health services. Perm J 2018;22:17–054.
12. Cawthorpe D, Marriott B, Paget J, Moulai I, Cheung S. Relationship between adverse childhood experience survey items and psychiatric disorders. Perm J 2018;22:18–001.
13. Cawthorpe D. Objective Skills Clinical Exam: Every Day in the Field of Practice Seattle: Amazon; 2018.
14. Clark M, et al. Objective skills clinical examination: every day in the field of practice. Fam Med 2019;51:364–365.
15. Heinze G, Sartorius N, Guizar Sanchez DP, Bernard-Fuentes N, Cawthorpe D, Cimino L, et al. Integration of mental health comorbidity in medical specialty programs in 20 countries. Int J Psychiatry Med 2021;56:278–293.
16. Wilkes TC, Guyn L, Li B, Lu M, Cawthorpe D. Association of child and adolescent psychiatric disorders with somatic or biomedical diagnoses: do population-based utilization study results support the adverse childhood experiences study? Perm J 2012;16:23–26.
17. Cawthorpe D. A novel population-based health index for mental disorder. Perm J 2013;17:50–54.
18. Satterlee WG, Eggers RG, Grimes DA. Effective medical education: insights from the Cochrane Library. Obstet Gynecol Surv 2008;63:329–333.

Article information Continued

Table 1.

Description of the trained cohort

Trainees Count
Family physicians 135
Pediatricians 78
Pediatrician specialists 8
Residents 10
Psychiatrists 13
Ped specialists 7
Allied professionals 25
Total 276

Table 2.

Sample size, means, upper and lower 95% CIs of by group comparing wait times and lengths of stay

Variable N Unexposed
N Pre-training
N Post-training
Mean (95% CI) Mean (95% CI) Mean (95% CI)
Wait time (days) 17,016 48 (47, 49)* 1,466 55 (51, 58)* 602 33 (31, 36)*
Length of stay (days) 16,287 187 (182, 193) 1,446 198 (187, 209)* 479 144 (130, 158)*

p<0.05 based on non-overlapping 95% confidence intervals (CI)