If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
In recent years, the importance of training healthcare professionals in nontechnical skills using effective methodologies has been increasingly recognised as a means of preventing clinical errors in the practice of health care. The aim of this study was to evaluate the effectiveness of educational interventions on nontechnical skills in the emergency medical services and/or critical care unit settings.
Methods
A systematic search was carried out in the PubMed, SCOPUS, CINAHL, and Web of Science databases according to predetermined inclusion and exclusion criteria. After the initial search, 7952 records were selected after duplicates removed. Finally, a selection of 38 studies was included for quantitative analysis. Separate meta-analyses of standardised mean changes were carried out for each outcome measure assuming a random-effects model. Cochran's Q-statistic and I2 index were applied to verify study heterogeneity. Weighted analyses of variance and meta-regressions were conducted to test the influence of potential moderators and funnel plots using Duval and Tweedie's trim-and-fill method, and Egger's regression test were used to examine publication bias.
Results
All the variables analysed had a significant effect size, with the exception of situational awareness (d+ = −0.448; 95% confidence interval [CI] = −1.034, 0.139). The highest mean effect size was found for knowledge (d+ = −0.925; 95% CI = −1.177, −0.673), followed by the mean effect sizes for global nontechnical skills (d+ = −0.642; 95% CI = −0.849, −0.434), team nontechnical skills (d+ = −0.606; 95% CI = −0.949, −0.262), and leadership nontechnical skills (d+ = −0.571; 95% CI = −0.877, −0.264). Similar mean effect sizes were found for attitude (d+ = −0.406; 95% CI = −0.769, −0.044), self-efficacy (d+ = −0.469; 95% CI = −0.874, −0.064), and communication nontechnical skills (d+ = −0.458; 95% CI = −0.818, −0.099). Large heterogeneity among the standardised mean changes was found in the meta-analyses (I2 > 75% and p < .001), except for self-efficacy where I2 = 58.17%, and there was a nonstatistical result for Cochran's Q. This great variability is also reflected in the forest plots.
Discussion
The use of simulation interventions to train emergency and critical care healthcare professionals in nontechnical skills significantly improves levels of knowledge, attitude, self-efficacy, and nontechnical skills performance.
Clinical errors arising from the practice of health care significantly affect patient safety, increasing mortality, morbidity, and the financial costs of healthcare systems.
Report of the Spanish Bioethics Committee on the ethical aspects of patient safety and, specifically, on the implementation of an effective system for reporting safety incidents and adverse events.
One of the first reports published by the U.S. Institute of Medicine (1999) estimated that 1 million deaths per year were attributable to preventable clinical errors.
noted that more than one in 10 patients are harmed by clinical errors during their care, causing more than 3 million deaths per year at an economic cost of between $1 and 2 trillion annually.
from 2004 to 2015 identified that the three leading causes of these clinical errors were mainly human factors in general, followed by lack of communication and leadership. Both are embraced in the broad universe of human factors and linked to the professionals' ability to perform nontechnical skills. These nontechnical skills are defined as “cognitive, social, and personal resource skills that complement technical skills and contribute to safe and efficient task performance”
skills related to situational awareness, decision-making, communication, teamwork, leadership, stress management, and coping with fatigue. Therefore, preventing clinical errors and delivering high-quality care outcomes requires not only high levels of technical knowledge, clinical competence, and good adaptability but also the acquisition and development of nontechnical skills.
The CRM methodology effectively describes and leverages communication and other components of teamwork to improve performance and the safety of care and, ultimately, the quality of care.
Crew resource management and its applications in medicine.
in: Dhojanmia K.G. Duncan B.W. McDonald K.M. Wachter R.M. Making health care safer: a critical analysis of patient safety practices. Evidence report/techonology assessment No 43, AHRQ. vol. 44. 2001: 511-519
A comparative study on the frequency of simulation-based training and assessment of non-technical skills in the Norwegian ground ambulance services and helicopter emergency medical services.
included 74 studies from different contexts where CRM had been applied, such as the commercial and military aviation industry, the nuclear industry, offshore oil production, commercial shipping, and health care. The authors concluded that the results were insufficient to demonstrate its effectiveness due to a lack of data and recommended a more rigorous evaluation of the methodology and greater access to data and resources. Meanwhile, O'Dea et al.
focused on the evaluation of CRM interventions and the training of critical care professionals in particular, finding them to be effective in the short term. Recent systematic reviews have been published which collect and report in depth many important aspects of CRM methodology
). Such reviews discuss in depth the design and structure of the interventions, content, coaches, evaluation methods, and the effectiveness of the interventions, among other issues. However, since 2014, no further evidence has been found in the secondary literature on the effectiveness of educational interventions using the CRM methodology (or covering concepts related to those addressed in the CRM of nontechnical skills) at the meta-analytic level. Although the CRM methodology is widely used, there are also interventions in the literature from other theoretical frameworks, such as TeamSTEPPS (Team Strategies and Tools to Enhance Performance and Patient Safety) or SBAR (Situation-Background-Assessment-Recommendation), as well as high-fidelity simulation interventions designed exclusively and without following a specific theoretical framework with the aim to train nontechnical skills.
The present meta-analysis was therefore conducted to evaluate the effectiveness of educational interventions involving nontechnical skills (communication, leadership, teamwork, situational awareness, decision-making, fatigue, and/or stress management) and targeting healthcare professionals (nursing, medical, and/or emergency health technicians) in critical care units and emergency medical services, both in-hospital and out-of-hospital.
2. Methods
This systematic review and meta-analysis was conducted according to the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA).
The work performed was registered on Open Science Framework (OSF) on March 3rd, 2022 (Registration DOI: 10.17605/OSF.IO/5QPGH).
2.1 Study selection
Studies were selected according to the following inclusion criteria: (i) An educational intervention aimed at healthcare professionals had been carried out; (ii) the intervention was aimed at medical professionals, nurses, emergency health technicians, and/or paramedics; (iii) the purpose of the intervention was the acquisition of nontechnical skills; (iv) the setting for the intervention was the emergency medical services (in-hospital/out-of-hospital) and/or critical care unit settings; (v) the articles presented the results in terms of a pre–post or postintervention comparison between groups (randomised controlled trial “[RCT]”, controlled clinical trial “[CCT]”, control trial, quasi-experimental).
In addition, the exclusion criteria were as follows: (i) Studies that were not written in English or Spanish, (ii) the intervention was aimed at undergraduate students, and (iii) those that did not provide sufficient statistical data to calculate the effect size.
2.2 Search strategies
The search for studies was conducted in January 2021, using a variety of strategies in order to collect as many articles as possible. The search strategies were developed in conjunction with experts in bibliographic searches in the field of health sciences. In the first instance, a search was carried out in the MEDLINE/PubMed, CINAHL, Web of Science, and Scopus electronic databases. As part of the search strategy, the PubMed thesaurus was consulted, using the following Medical Subject Headings (MeSH) terms: “Crew Resource Management”, “simulation training”, “high fidelity simulation training”, “emergency medicine”, “emergency responders”, “emergency nursing”, “emergency medical technicians”, “emergencies”, “critical care”, and “emergency medical services”. The natural language search terms included in the title and/or abstract fields were: “Crew Resource Management”, “Crisis Resource Management”, “CRM”, “non-technical skills”, “simulation training”, “high fidelity simulation training”, “critical care”, “emergency”, “critical care nursing”, and “emergency medical technicians”. The search strategy used for each database is outlined in supplementary file 1.
A manual search was also performed in journals aimed at the training of healthcare professionals with a specific focus on simulation methodology: “Emergencies”, “Simulation in Healthcare”, “International Emergency Nursing”, “Medical Education”, “Medical Teacher”, “Advances in Simulation”, “BMC Medicine”, and “Nurse Education Today”. No time restrictions were applied in any of these cases, with the aim of being as exhaustive as possible, compiling all existing interventions that meet the established criteria and analysing their effectiveness. Given that simulation methodology has a track record going back several decades and its effectiveness is unlikely to have been impacted by technological advances, no time restrictions were applied to the search. Lastly, the bibliographic references of the articles included were reviewed to find other studies that could be relevant to the meta-analysis but that had not been found using the aforementioned strategies. The process of evaluating the eligibility of the studies was carried out independently by two investigators (M.S.M. and S.E.). Discrepancies were resolved by a third investigator (M.J.C.M.) in order to reach consensus.
2.3 Data extraction
A coding protocol was developed to extract the study characteristics, classifying the extracted variables into three categories: substantive, methodological, and extrinsic.
The following substantive variables were coded: (i) level of knowledge; (ii) attitude score; (iii) self-efficacy score; (iv) overall nontechnical skills score; (v) score for teamwork-related skills; (vi) leadership skills score; (vii) communication skills score; (viii) score for situational awareness skills; (ix) medical specialty (accident and emergency, helicopter emergency medical service, or critical care unit); (x) type of methodology (passive, interactive, mixed, or unspecified); (xi) theoretical model (CRM, other, unspecified); (xii) type of simulation (high fidelity, low fidelity, none), and (xiii) scenario objectives (CRM [design to train elements of the CRM or specifically mentioned], nontechnical skills, others [design to train skills other than nontechnical skills], unspecified).
The following methodological variables were extracted: (i) study design; (ii) intervention comparison group; (iii) assessment period; (iv) sample size of the pretest experimental group; (v) sample size of the pretest control group; (vi) sample size of the posttest experimental group; (vii) sample size of the posttest control group; (viii) sample size of the follow-up experimental group; (ix) sample size of the follow-up control group; (x) randomisation of the groups; (xi) dropout rate; and (xii) the domains of the quality scale Medical Education Research Study Quality Instrument (MERSQI)
Predictive validity evidence for medical education research study quality instrument scores: quality of submissions to JGIM's Medical Education Special Issue.
Appraising the quality of medical education research methods: the medical education research study quality instrument and the newcastle-ottawa scale-education.
(see Assessment of Risk Bias of Selected Studies section). Finally, the extrinsic variables coded were as follows: (i) authors; (ii) year; (iii) educational background of the lead author of the study; and (iv) publication status (published or unpublished and whether it has an ISBN or ISSN).
The coding and extraction of these variables was carried out independently by two researchers (M.S.M. and S.E.). Disagreements between the two researchers who coded these characteristics were resolved with a third researcher (M.J.C.M.).
2.4 Assessment of risk of bias of selected studies
The quality of the studies was measured using the MERSQI,
Predictive validity evidence for medical education research study quality instrument scores: quality of submissions to JGIM's Medical Education Special Issue.
which was designed to measure the quality of experimental, quasi-experimental, and observational studies, specifically in medical education research. The tool includes 10 items clustered in six domains (“study design”; “sampling”; “data type”; “validity”; “analysis”; “outcome”) and a final overall score. These criteria form six domains, each with a maximum score of 3 and a minimum of 0 or 1, that sum to produce a total score that ranges from 5 to 18. It has generally higher interrater reliability (0.68–0.89) and good relationship with the scale Newcastle–Ottawa Scale-Education (rho = 0.60) for studies of simulation-based trained.
Appraising the quality of medical education research methods: the medical education research study quality instrument and the newcastle-ottawa scale-education.
The evaluation was performed in pairs, with each researcher independently evaluating the studies, and then discrepancies were resolved by consensus. All participating researchers (M.S.M., S.E., M.R.A., M.J.C.M., and R.J.S.) were previously trained in extracting and evaluating the methodological quality of studies and in the use of the evaluation instrument used.
2.5 Computation of effect sizes
For maximum comprehensiveness, this meta-analysis aimed to include studies with and without a comparison group. The unit of analysis was therefore the group, not the study, and the effect size was the standardised mean rate of change. This index was calculated as the pretest-posttest mean and this difference was divided by the pretest standard deviation: , with c(m) being a correction factor for small sample sizes.
Negative d values indicated an improvement in the group from the pretest to the posttest. For standardisation, absolute values of d of around 0.2, 0.5, and 0.8 can be interpreted as small, moderate, and large magnitudes, respectively.
In order to calculate the standardised mean changes when the necessary information (means and standard deviations) was not reported in the study, the authors of the corresponding studies were contacted to request the necessary missing data. In the absence of a reply, effect sizes were calculated using different procedures depending on the information available in the study. For example, conversion equations from significance tests (e.g., t-test and Wilcoxon test) and sample size were used.
When results were reported by means of sample size, median, range, and/or interquartile range, the means and standard deviations were estimated by different approximation methods.
Finally, effect sizes were calculated separately for each construct evaluated: knowledge, attitude, self-efficacy, global nontechnical skills, teamwork, communication, leadership, and situational awareness.
2.6 Statistical analyses
Separate meta-analyses were carried out for the effect sizes of each outcome measure in at least three studies, by assuming a random-effects model. This model involves weighting each standardised mean change by its inverse variance, defined as the sum of the within-study variance and between-study variance, the latter being estimated by restricted maximum likelihood.
For each outcome analysed, a forest plot was constructed showing the individual effect sizes, and the mean effect size calculated with a 95% confidence interval (CI) using the method proposed by Hartung.
To assess heterogeneity among the individual standardised mean changes, Cochran's heterogeneity Q statistic and the I2 index were calculated. If heterogeneity was found and the number of studies for the outcome was at least 10, an analysis of potential moderator variables was performed.
For this purpose, mixed-effects models were assumed applying the F statistic described by Knapp-Hartung for testing the significance of the moderator variable.
The QW and QE statistics were calculated to assess the model misspecification of the weighted analyses of variance (ANOVAs) and meta-regressions, respectively, together with an estimate of the percentage of variance accounted for by the moderator variable, R2.
Finally, publication bias was tested by constructing funnel plots using Duval and Tweedie's trim-and-fill method
A statistically significant result for Egger's test (p < .10) was evidence of publication bias. We used p < .10 instead of the usual p < .05 because of the lower statistical power of Egger's test with such a small number of studies.
The screening and coding of studies was performed manually. During the screening process, two reviewers independently excluded 7118 studies based on the title and abstract (Fig. 1). After this first screening, 834 full-text studies were reviewed, with 794 studies that did not meet the inclusion criteria being excluded. We contacted the authors of 11 studies that failed to provide sufficient statistical data to calculate the effect size; only one response was obtained (k = 10 studies were discarded for this reason). Finally, two independent coders extracted data from 38 studies (supplementary file 2) in accordance with the study coding manual.
Fig. 1PRISMA 2020 flow diagram elaborated by Haddaway et al.
PRISMA2020: an R package and Shiny app for producing PRISMA 2020-compliant flow diagrams, with interactivity for optimised digital transparency and Open Synthesis.
Table 1 contains the main characteristics of the 38 studies included in the review. The total number of participants was 3144 (min. = 7; max. = 684), with an average of 82.74 participants per sample (SD = 124.83). The mean ages of the participants ranged from 21 to 43.43 years (M = 33.59; SD = 6.73), the standard deviations of their ages ranged from 5.70 to 10.75 years (M = 8.69; SD = 2.13), and the percentage of females ranged from 11.5 to 94.73 (M = 70.79; SD = 21.85).
Report of the Spanish Bioethics Committee on the ethical aspects of patient safety and, specifically, on the implementation of an effective system for reporting safety incidents and adverse events.
Crew resource management and its applications in medicine.
in: Dhojanmia K.G. Duncan B.W. McDonald K.M. Wachter R.M. Making health care safer: a critical analysis of patient safety practices. Evidence report/techonology assessment No 43, AHRQ. vol. 44. 2001: 511-519
A comparative study on the frequency of simulation-based training and assessment of non-technical skills in the Norwegian ground ambulance services and helicopter emergency medical services.
Predictive validity evidence for medical education research study quality instrument scores: quality of submissions to JGIM's Medical Education Special Issue.
Appraising the quality of medical education research methods: the medical education research study quality instrument and the newcastle-ottawa scale-education.
1. Questionnaire CRM basics (no validated) 2. Questionnaire (a self-administered, no validated)
2016
Germany
N = 26
3/26 (11.50%)
M = 38.7 (SD = 5.70)
2 days
A&E, Catastrophe or HEMS
Medicine
CRM
Mixed: Lectures and simulation
High-fidelity simulation
CRM
Note: The articles included are cited in supplementary material Appendix 2).
A&E, emergencies services; BARS, Behaviorally Anchored Rating Scale; CG, control group; CRM, critical resources management; EART, Emergency airway response team; ECMO Sim, Immersive simulation-based training program designed to teach skills to successfully manage life-threatening crises on extracorporeal membrane oxygenation; EG, experimental group; ENNTS, Emergency Nurses' Non-technical Skills; GRS, Global Rating Scale; HEMS, helicopter emergency medical service; HIRAD, History, Identify Red Flags, Assessment, Interventions, Diagnostics, reassessment and communication; M, average; MPTHS, Mayo High Performance Teamwork Scale; NTS, nontechnical skills; Ottawa GRS, Ottawa Crisis Resource Management Global Rating Scale; SD, standard deviation; SBAR, Situation, Background, Assessment, and Recommendation techniques to transfer the patient to a higher level of care; T_NOTECHS, Trauma Non-technical Skills teamwork scale; TeamSTEPPS, Team Strategies and Tools to Enhance Performance and Patient Safety; T-TPQ, Team- STEPPS® teamwork and perceptions questionnaire; TTPOT, Trauma Team Performance Observation Tool; TTAQ, TeamSTEPPS Teamwork Attitudes Questionnaire.
We analysed 21 articles (55.3%) aimed at emergency health professionals (including disaster or helicopter emergency medical service), 14 articles (39.5%) aimed at critical care professionals, and two (5.3%) articles aimed at both professionals. The predominant teaching methodology was the mixed methodology (passive and interactive) (n = 27; 71.1%). The predominant theoretical model was CRM (n = 18; 47.4%). The most used type of simulation was high fidelity (n = 32; 84.2%), and the objectives of the simulation scenarios were mainly geared towards nontechnical skills training (n = 18; 47.4%), followed by CRM (n = 16; 42.1%), unspecified (n = 3; 7.9%), and other (n = 1; 2.6%). Most studies (n = 32; 84.21%) performed immediate posttest measurements, except for five articles (13.16%) which evaluated the results with variability between 2 and 8 weeks after the intervention, considered as posttest. One article performed the first evaluation measure at 4 months after implementation and, finally, was not included in the statistical analysis. The year of publication of the studies found ranged from 2002 to 2021. In terms of the locations of the studies, four continents were identified: North America, 24 (63.2%); Asia, 1 (2.6%); Europe, 9 (23.7%); Oceania, 3 (7.9%), and International, 1 (2.6%). All studies were written in English.
3.2 Assessment of risk bias of selected studies
With respect to the methodological quality of the studies, the average MERSQI
Predictive validity evidence for medical education research study quality instrument scores: quality of submissions to JGIM's Medical Education Special Issue.
score was 12.88 (SD = 2.21), with scores ranging from 8.5 to 16.5 (median = 14) (Table 2). Study designs included randomised experimental studies (n = 4; 10.52%) and two-group quasi-experimental designs with pretest and posttest (n = 3; 7.89%) and one-group quasi-experimental designs with pretest and posttest (31; 81.58%). These studies predominantly employed a single institution (n = 27, 71.1%), using objective data by external observer ratings (n = 26, 68.4%) and assessing knowledge or skills outcomes (n = 27, 71.1%). Most of the tools (n = 28, 73.7%) used to assess the outcomes had psychometric tests of internal consistency. All articles (100%) reported statistical inference and were appropriate for the design and types of data collected.
Predictive validity evidence for medical education research study quality instrument scores: quality of submissions to JGIM's Medical Education Special Issue.
Report of the Spanish Bioethics Committee on the ethical aspects of patient safety and, specifically, on the implementation of an effective system for reporting safety incidents and adverse events.
Crew resource management and its applications in medicine.
in: Dhojanmia K.G. Duncan B.W. McDonald K.M. Wachter R.M. Making health care safer: a critical analysis of patient safety practices. Evidence report/techonology assessment No 43, AHRQ. vol. 44. 2001: 511-519
A comparative study on the frequency of simulation-based training and assessment of non-technical skills in the Norwegian ground ambulance services and helicopter emergency medical services.
Predictive validity evidence for medical education research study quality instrument scores: quality of submissions to JGIM's Medical Education Special Issue.
Appraising the quality of medical education research methods: the medical education research study quality instrument and the newcastle-ottawa scale-education.
Table 3 shows the results of the eight meta-analyses performed for each outcome measure, and the corresponding forest plots are shown in supplementary files 3 and 4. The mean effect sizes calculated for all eight outcomes were negative, indicating an improvement from pretest to posttest. All of them were statistically significant, with the exception of situational awareness (d+ = −0.448; 95% CI = −1.034, 0.139), most likely due to the low statistical power of such a small number of studies (n = 6).
Table 3Mean effect sizes, 95% confidence, and heterogeneity statistics for the outcome variables.
n
d+
95% CI
Q
I2
dl
du
Knowledge
13
−0.925
−1.177
−0.673
120.515∗∗∗
87.67%
Attitude
6
−0.406
−0.769
−0.044
24.880∗∗∗
81.20%
Self-efficacy
3
−0.469
−0.874
−0.064
4.965
58.17%
Global NTS
19
−0.642
−0.849
−0.434
69.956∗∗∗
78.00%
Team NTS
13
−0.606
−0.949
−0.262
158.125∗∗∗
92.48%
Communication NTS
14
−0.458
−0.818
−0.099
104.610∗∗∗
87.94%
Leadership NTS
10
−0.571
−0.877
−0.264
40.468∗∗∗
83.15%
Awareness NTS
6
−0.448
−1.034
0.139
24.225∗∗∗
82.46%
Note: NTS, nontechnical skills; n, number of studies; d+, Mean effect size; 95% CI, 95% confidence interval for d+; dl and du, lower and upper confidence limits; Q, Cochran's heterogeneity Q statistic; with k – 1 degrees of freedom; I2, Heterogeneity index; ∗∗∗p < .001.
The highest mean effect size was found for knowledge (d+ = −0.925; 95% CI = −1.177, −0.673), followed by the mean effect sizes for global nontechnical skills (d+ = −0.642; 95% CI = −0.849, −0.434), team nontechnical skills (d+ = −0.606; 95% CI = −0.949, −0.262), and leadership nontechnical skills (d+ = −0.571; 95% CI = −0.877, −0.264). Similar mean effect sizes were found for attitude (d+ = −0.406; 95% CI = −0.769, −0.044), self-efficacy (d+ = −0.469; 95% CI = −0.874, −0.064), and communication nontechnical skills (d+ = −0.458; 95% CI = −0.818, −0.099). In all cases, the mean effect sizes were large to medium in magnitude.
Large heterogeneity among the standardised mean changes was found in the meta-analyses (I2 > 75% and p < .001), with the exception of self-efficacy where I2 = 58.17%, and there was a nonstatistical result for Cochran's Q (see Table 3). This great variability is also reflected in the forest plots (see supplementary files 3 and 4).
3.4 Analysis of publication bias
With respect to the funnel plots using the trim-and-fill method, for most of the analysed outcomes, the effect sizes were imputed on the right-hand side of the funnel plots in order to achieve symmetry. For the knowledge measures specifically, three effect sizes were imputed (see supplementary file 5A), leading to a corrected effect size of dc = −0.801 (95% CI = −1.039, −0.564). For the self-efficacy and leadership nontechnical skills measures, two effect sizes were imputed (see supplementary files 5C and 6C, respectively), leading to corrected effect sizes of dc = −0.330 (95% CI = −0.548, −0.112) and dc = −0.475 (95% CI = −0.757, −0.193), respectively. For global nontechnical skills measures, four effect sizes were imputed (see supplementary file 5D), leading to a corrected effect size of dc = −0.556 (95% CI = −0.748, −0.364). For communication skills, six effect sizes were imputed (supplementary file 6B), leading to a corrected effect size of dc = −0.154 (95% CI = −0.482, 0.174). Finally, for situational awareness measures, one effect size was imputed (see supplementary file 6D), leading to a corrected effect size of dc = −0.326 (95% CI = −0.798, 0.147). However, Egger's test showed statistically nonsignificant results (p > .05) for all eight outcome measures, allowing us to rule out publication bias as a threat to the validity of the results of the meta-analyses.
3.5 Analysis of moderator variables
The great heterogeneity found among the standardised mean changes led to analysis of the moderator variables. This analysis was applied to those meta-analyses with at least 10 standardised mean changes, i.e., measures of knowledge, global, team, communication, and leadership skills. Supplementary files 7 to 12 show the results of the weighted ANOVAs and meta-regressions for the results analysed previously.
Of the various potential moderator variables analysed, only simulation showed a statistically significant relationship with the effect sizes on communication nontechnical skills (p < .05). To be precise, the mean effect size obtained for high fidelity (d+ = −0.647) was statistically larger than that of low fidelity (d+ = 0.396), with 38.33% of variance accounted for (see supplementary file 10). Finally, the quality total score did not show a significant effect with any of the outcome measures analysed, i.e., knowledge, global, team, communication, and leadership skills (see supplementary file 12).
4. Discussion
4.1 Summary findings
The present meta-analysis was conducted with a view to evaluating the effectiveness of educational interventions that include certain nontechnical skills and are targeted at healthcare professionals in critical care units and emergency departments, both in-hospital and out-of-hospital. Effects were observed on participants' knowledge, teamwork, global nontechnical skills, leadership, attitude, self-efficacy, and communication, assessed in the short term. There were not enough evaluations to perform a quantitative analysis of the long-term evaluations. O'Dea et al.
reflect how most interventions are evaluated through posttest measures and within the first 2 months, concluding that there is no evidence of the long-term impacts of the training.
4.2 Comparison and contrast with previous studies
Unlike previous meta-analyses, our study has focused on specific clinical units where CRM is particularly relevant, such as emergency and critical care units. In the past, the focus was on industrial and/or military contexts, such as the commercial and military aviation industry, air traffic control, the nuclear sector, or commercial shipping.
such as the operating room, trauma, anaesthesiology, obstetrics, and neonatal, as well as emergency and critical care units. Furthermore, while the present meta-analysis sample is of healthcare professionals, including nurses, physicians, and emergency healthcare technicians, others include students and professionals from other disciplines
with varying degrees of experience, expertise, and training. It is worth noting that this current paper addresses all the dimensions of nontechnical skills,
most were carried out in the United States. Some of the studies included do not provide theoretical training models to support their interventions, which was already evidenced in the research of Walshe et al.
i.e., whereby in addition to being trained by an instructor (passive methodology), healthcare professionals actively participate (interactive methodology) as part of their learning (e.g., training in nontechnical skills using high-fidelity simulation). As the literature indicates, this methodology is more effective than traditional methodologies where training focuses on the transmission of information by an expert in an organised and systematic way, combined with the repetitive practice of clinical procedures and skills.
when the sample consists of expert professionals, high-fidelity simulations are more effective. However, if the sample is composed of nonexperts, e.g., nursing or medical students, then medium-fidelity simulations would be more effective because of their level of knowledge and clinical experience.
With regard to the results of the different interventions, significant effect size statistics were obtained for the knowledge, attitude, self-efficacy, global nontechnical skills, teamwork, communication, and leadership variables. In relation to the outcome variable “knowledge”, the effect size observed in our study was larger than that obtained by O'Connor et al.,
It was not possible to compare the variable “self-efficacy” with previous meta-analyses. However, it is considered a very important variable for the evaluation of training interventions, including nontechnical skills.
A significant effect size was also observed for “global nontechnical skills” as well as for the specific nontechnical skills measured: teamwork, communication, and leadership. Consistent with these findings, the meta-analysis conducted by McEwan et al.
evidenced a moderately large effect size for teamwork. Finally, it should be noted that, as found in previous meta-analyses with healthcare professionals,
situational awareness did not show a statistically significant effect with this type of intervention.
However, the high diversity in the tools used for assessing each of the variables should be considered, which could be influencing the heterogeneity found in this meta-analysis, although we have not reported. O'Dea et at.
have shown that this diversity is one of the main limitations in this field of research since a wide variety of instruments are used, sometimes not validated, and include different types of measures, such as self-administered and direct behavioural observations. Standardisation of assessment is recommended
and, on the other hand, to analyse whether the assessment instruments are moderating the results.
In relation to the analysis of moderator variables, only the fidelity level of simulation showed effects on communication nontechnical skills between the team. The results show that interventions are more effective when trained with high-fidelity simulation than those that performed with low fidelity. In keeping with these results, the meta-analysis conducted by Kim et al.,
whose aim was to determine the effect size of simulation-based educational interventions in nursing and compare the results according to the fidelity level, identify high-fidelity simulation offers benefits over low-fidelity simulation in cognitive and affective learning outcomes. While the characterised results are of psychomotor type, it evidenced the medium-fidelity simulation (full-body manikins and can be controlled by an external) as more effective. Therefore, adapting the most effective simulation method to meet the proposed results and objectives could be key.
In addition, high-fidelity simulation has a high cost. So, it is convenient to assess and consider whether other levels of less expensive simulation can achieve similar results.
In any case, it would be advisable to continue research to detect whether there are differences in learning between the different levels.
4.3 Strengths and limitations
This study has some limitations. Firstly, the search strategy may have excluded potential articles, even though various search methods were employed to minimise this possibility. Secondly, most of the studies found were quasi-experimental studies with uncontrolled pretest and posttest designs, thus limiting their internal validity. Randomised controlled trials are the most appropriate option, but they are more complicated to carry out, especially in emergency health contexts. Thirdly, the results regarding the heterogeneity should be interpreted cautiously since the I2 index presents some limitations. On the one hand, if the number of studies included in the meta-analysis integrates large sample sizes, the I2 will be larger in the absence the heterogeneity. On the other hand, with the I2 being a proportion, the amount of heterogeneity quantified not is absolute.
Finally, moderate to high levels of heterogeneity were found in the meta-analytic synthesis of all the variables, which influences the final effect of the interventions. Particularly noteworthy is the lack of standardisation in the evaluation of outcomes. A wide variety of instruments, at times not validated, have been used to measure the outcome variables.
Despite these limitations, there are sufficient studies demonstrating that educational interventions, such as CRM, are teaching methodologies that improve nontechnical skills training in the context of emergency departments and critical care units. The use of training programs based on these methodologies to improve the performance of healthcare professionals could improve safety, team performance, and, ultimately, the quality of patient care.
Crew resource management and its applications in medicine.
in: Dhojanmia K.G. Duncan B.W. McDonald K.M. Wachter R.M. Making health care safer: a critical analysis of patient safety practices. Evidence report/techonology assessment No 43, AHRQ. vol. 44. 2001: 511-519
4.4 Implications for educators and clinical practice
Specialised services such as emergency and intensive care require highly trained teams to minimise clinical errors and achieve high-quality care performance, and for this, the acquisition of nontechnical skills such as communication, leadership, and teamwork is critical. The results of this meta-analysis evidence the need for professionals to continue to update their skills throughout their careers, as well as the inclusion of high-fidelity simulation in the curricula of healthcare professionals, especially for the acquisition of the communication skills.
4.5 Areas for future research
A feasible option for further research therefore appears to be the exploration of effective methodologies for situational awareness training, as evidenced in this study. It would also be advisable to conduct more longitudinal, multicentre, and multispecialty studies with the aim of establishing, with adequate methodological quality, the long-term effects of training in these methodologies and the impact of their transfer to clinical practice. It is essential to know if the results obtained are maintained over time
and, if not, to know the need to implement reminder interventions and their temporality. All of this will lead to more excellent training for dealing with complex scenarios and, therefore, to improve positive health outcomes, quality of care, and patient safety.
On the other hand, further studies are recommended to examine the efficacy of specific CRM programs versus other simulation programs in other conceptual frameworks aimed at training in nontechnical skills, as well as other factors related to the characteristics of the interventions that could determine their efficacy. The results of this meta-analysis have not been conclusive in this regard for any of the variables studied.
5. Conclusions
Learning and teaching nontechnical skills to healthcare professionals in emergency crisis settings using simulation-based methodologies leads to improvements in their knowledge levels, attitudes, and self-efficacy, as well as in the performance of nontechnical skills, both at a global level and in specific skills such as communication, leadership, and work. The situational awareness of the professionals, however, needs to improve.
Funding
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
CRediT authorship contribution statement
María Sánchez-Marco: conceptualisation, methodology, formal analysis, writing -original draft, writing -reviewing & editing and funding acquisition. Silvia Escribano: conceptualisation, methodology, formal analysis, writing -original draft, writing -reviewing & editing and funding acquisition. María José Cabañero-Martínez: conceptualisation, methodology, resources, reviewing & editing, supervision and funding acquisition. María Rubio-Aparicio: reviewing & editing, methodology and formal analysis. Rocio Juliá-Sanchís: conceptualisation, methodology, writing -original draft and writing -reviewing & editing and funding acquisition.
Conflict of interest
The authors state that they have no conflicts of interest.
Appendix A. Supplementary data
The following are the Supplementary data to this article:
Report of the Spanish Bioethics Committee on the ethical aspects of patient safety and, specifically, on the implementation of an effective system for reporting safety incidents and adverse events.
Crew resource management and its applications in medicine.
in: Dhojanmia K.G. Duncan B.W. McDonald K.M. Wachter R.M. Making health care safer: a critical analysis of patient safety practices. Evidence report/techonology assessment No 43, AHRQ. vol. 44. 2001: 511-519
A comparative study on the frequency of simulation-based training and assessment of non-technical skills in the Norwegian ground ambulance services and helicopter emergency medical services.
Predictive validity evidence for medical education research study quality instrument scores: quality of submissions to JGIM's Medical Education Special Issue.
Appraising the quality of medical education research methods: the medical education research study quality instrument and the newcastle-ottawa scale-education.
PRISMA2020: an R package and Shiny app for producing PRISMA 2020-compliant flow diagrams, with interactivity for optimised digital transparency and Open Synthesis.
00011-5&id=gr1.jpg){kind=link}