Advertisement

A practical approach to establishing a critical care outreach service: An expert panel research design

Open AccessPublished:March 24, 2022DOI:https://doi.org/10.1016/j.aucc.2022.01.008

      Abstract

      Background

      For over two decades, nurse-led critical care outreach services have improved the recognition, response, and management of deteriorating patients in general hospital wards, yet variation in terms, design, implementation, and evaluation of such services continue. For those establishing a critical care outreach service, these factors make the literature difficult to interpret and translate to the real-world setting.

      Aim

      The aim of this study was to provide a practical approach to establishing a critical care outreach service in the hospital setting.

      Method

      An international expert panel of clinicians, managers, and academics with experience in implementing, developing, operationalising, educating, and evaluating critical care outreach services collaborated to synthesise evidence, experience, and clinical judgment to develop a practical approach for those establishing a critical care outreach service. A rapid review of the literature identified publications relevant to the study. A modified Delphi technique was used to achieve expert panel consensus particularly in areas where insufficient published literature or ambiguities existed.

      Findings

      There were 502 publications sourced from the rapid review, of which 104 were relevant and reviewed. Using the modified Delphi technique, the expert panel identified five key components needed to establish a critical care outreach service: (i) approaches to service delivery, (ii) education and training, (iii) organisational engagement, (iv) clinical governance, and (v) monitoring and evaluation.

      Conclusion

      An expert panel research design successfully synthesised evidence, experience, and clinical judgement to provide a practical approach for those establishing a critical care outreach service. This method of research will likely be valuable in other areas of practice where terms are used interchangeably, and the literature is diverse and lacking a single approach to practice.

      Keywords

      1. Introduction

      Rapid response systems (RRSs) were developed in the early 2000s with the aim of reducing major adverse events
      • Cretikos M.A.
      • Chen J.
      • Hillman K.M.
      • Bellomo R.
      • Finfer S.R.
      • Flabouris A.
      The effectiveness of implementation of the medical emergency team (MET) system and factors associated with use during the MERIT study.
      ,
      • Bellomo R.
      • Goldsmith D.
      • Uchino S.
      • Buckmaster J.
      • Hart G.
      • Opdam H.
      • et al.
      Prospective controlled trial of effect of medical emergency team on postoperative morbidity and mortality rates.
      and improving patient outcomes. Major adverse events include in-hospital cardiac arrest, unplanned admission to the intensive care unit (ICU), and unexpected death.
      • Massey D.
      • Aitken L.M.
      • Chaboyer W.
      The impact of a nurse led rapid response system on adverse, major adverse events and activation of the medical emergency team.
      The term RRS is used to describe the whole system responsible for detecting and responding to deteriorating patients regardless of location.
      • Devita M.A.
      • Bellomo R.
      • Hillman K.
      • Kellum J.
      • Rotondi A.
      • Teres D.
      • et al.
      Findings of the first consensus conference on medical emergency teams.
      There are two limbs to the RRS: an afferent (detection) limb, which normally has a track and trigger component to help clinicians identify patient deterioration, and an efferent (response) limb, which provides an escalation response to the deteriorating patient.
      • Devita M.A.
      • Bellomo R.
      • Hillman K.
      • Kellum J.
      • Rotondi A.
      • Teres D.
      • et al.
      Findings of the first consensus conference on medical emergency teams.
      ,
      • Lyons P.G.
      • Edelson D.P.
      • Churpek M.M.
      Rapid response systems.
      Within the efferent limb of the RRS, the terms rapid response team (RRT), medical emergency team (MET), and critical care outreach are often used interchangeably, yet formal definitions exist. Lyon et al.
      • Lyons P.G.
      • Edelson D.P.
      • Churpek M.M.
      Rapid response systems.
      describe an MET as commonly led by a physician, who can “prescribe critical care interventions, obtain central access and facilitate airway management” (p 3). Devita et al.
      • Devita M.A.
      • Bellomo R.
      • Hillman K.
      • Kellum J.
      • Rotondi A.
      • Teres D.
      • et al.
      Findings of the first consensus conference on medical emergency teams.
      describe an RRT as a team that provides an intermediate or ‘ramp up’ approach and a critical care outreach service (CCOS) as a system that includes an RRS component and a focus on prevention. More recently, Lyon et al.
      • Lyons P.G.
      • Edelson D.P.
      • Churpek M.M.
      Rapid response systems.
      describe an RRT as usually being a nurse-led team, acknowledging that whether a team is physician led or nurse led may not affect mortality. This study uses the term CCOS to describe a nurse-led team.
      The literature uses multiple terms to describe nurses working within a CCOS including critical care outreach nurses (CCONs),
      • Garry L.
      • Rohan N.
      • O'Connor T.
      • Patton D.
      • Moore Z.
      Do nurse-led critical care outreach services impact inpatient mortality rates?.
      intensive care outreach nurses, intensive care liaison nurses,
      • Lynch J.
      • Cope V.
      • Murray M.
      The intensive care unit liaison nurse and their value in averting clinical deterioration: a qualitative descriptive study.
      • McIntyre T.
      • Taylor C.
      • Bailey M.
      • Jones D.
      Differences in the characteristics, treatment, and outcomes of patient groups reviewed by intensive care liaison nurses in Australia: a multicentre prospective study.
      • Alberto L.
      • Zotárez H.
      • Cañete Á.
      • Niklas J.E.
      • Enriquez J.M.
      • Gerónimo M.R.
      • et al.
      A description of the ICU liaison nurse role in Argentina.
      patient-at-risk team nurses,
      • Pirret A.M.
      • Takerei S.F.
      • Kazula L.M.
      The effectiveness of a patient at risk team comprised of predominantly ward experienced nurses: a before and after study.
      or in some hospitals, the after-hours clinical team co-ordinator.
      • Massey D.
      • Aitken L.M.
      • Chaboyer W.
      The impact of a nurse led rapid response system on adverse, major adverse events and activation of the medical emergency team.
      The composition of CCOSs also vary, ranging from typically critical care registered nurse responders through to advanced practice providers (APPs), such as nurse practitioners and nurse consultants.
      • Lyons P.G.
      • Edelson D.P.
      • Churpek M.M.
      Rapid response systems.
      ,
      • Garry L.
      • Rohan N.
      • O'Connor T.
      • Patton D.
      • Moore Z.
      Do nurse-led critical care outreach services impact inpatient mortality rates?.
      ,
      • Olsen S.L.
      • Søreide E.
      • Hillman K.
      • Hansen B.S.
      Succeeding with rapid response systems – a never-ending process: a systematic review of how health-care professionals perceive facilitators and barriers within the limbs of the RRS.
      ,
      • Pirret A.M.
      The role and effectiveness of a nurse practitioner led critical care outreach service.
      Increasingly APPs are working within CCOSs and add value to the team by providing diagnostic and treatment expertise, facilitating transfer to the ICU, and improving team communication and education.
      • Kapu A.N.
      Addition of acute care nurse practitioners to medical and surgical rapid response teams: a pilot project.
      Internationally, established CCOSs improve patient outcomes. Whilst methodological flaws exist in many studies,
      • So H.M.
      • Yan W.W.
      • Chair S.Y.
      A nurse-led critical care outreach program to reduce readmission to the intensive care unit: a quasi-experimental study with a historical control group.
      • Kovacs C.
      Outreach and early warning systems for the prevention of intensive care admission and death of critically ill adult patients on general hospital wards.
      • Aitken L.M.
      • Chaboyer W.
      • Vaux A.
      • Crouch S.
      • Burmeister E.
      • Daly M.
      • et al.
      Effect of a 2-tier rapid response system on patient outcome and staff satisfaction.
      research suggests CCOSs reduce admission to the ICU, ward cardiac arrests, and hospital mortality.
      • Massey D.
      • Aitken L.M.
      • Chaboyer W.
      The impact of a nurse led rapid response system on adverse, major adverse events and activation of the medical emergency team.
      ,
      • Garry L.
      • Rohan N.
      • O'Connor T.
      • Patton D.
      • Moore Z.
      Do nurse-led critical care outreach services impact inpatient mortality rates?.
      ,
      • McIntyre T.
      • Taylor C.
      • Bailey M.
      • Jones D.
      Differences in the characteristics, treatment, and outcomes of patient groups reviewed by intensive care liaison nurses in Australia: a multicentre prospective study.
      ,
      • Pirret A.M.
      • Takerei S.F.
      • Kazula L.M.
      The effectiveness of a patient at risk team comprised of predominantly ward experienced nurses: a before and after study.
      Delays in activation of the CCOS is associated with increased mortality
      • Tillmann B.W.
      • Klingel M.L.
      • McLeod S.L.
      • Anderson S.
      • Haddara W.
      • Parry N.G.
      The impact of delayed critical care outreach team activation on in-hospital mortality and other patient outcomes: a historical cohort study.
      ,
      • Pattison N.
      • Eastham E.
      Critical care outreach referrals: a mixed-method investigative study of outcomes and experiences.
      and an increased likelihood of ICU admission.
      • Tillmann B.W.
      • Klingel M.L.
      • McLeod S.L.
      • Anderson S.
      • Haddara W.
      • Parry N.G.
      The impact of delayed critical care outreach team activation on in-hospital mortality and other patient outcomes: a historical cohort study.
      Allen et al.
      • Allen E.
      • Elliott D.
      • Jackson D.
      Recognising and responding to in-hospital clinical deterioration: an integrative review of interprofessional practice issues.
      highlight CCON's knowledge and clinical expertise prevent unnecessary delays through effective escalation and accelerated decision-making.
      Along with the multiple terms used to describe CCOSs, there is limited uniformity and standardisation of how CCOSs are developed, implemented, or operationalised. For hospitals wishing to introduce a CCOS, the literature is diverse, is difficult to interpret, and lacks a clear and well-defined model to follow.
      • Garry L.
      • Rohan N.
      • O'Connor T.
      • Patton D.
      • Moore Z.
      Do nurse-led critical care outreach services impact inpatient mortality rates?.
      Furthermore, this diversity creates challenges in relation to design, education, research and evaluation, and difficulties in translating concepts to the real-world setting. By using a three-step process, this study aims to provide a practical approach for those establishing a CCOS, thereby creating a theory to practice bridge that supports and facilitates knowledge translation.

      2. Method

      2.1 Development steps

      In this study, we outline a practical approach to developing a CCOS using a three-step process: (i) an expert panel, (ii) a rapid review of the literature, and (iii) a modified Delphi technique. Researchers have used similar processes to develop important position and consensus statements.
      • Haugland H.
      • Rehn M.
      • Klepstad P.
      • Krüger A.
      • Albrektsen G.E.
      • Berlac P.A.
      • et al.
      Developing quality indicators for physician-staffed emergency medical services: a consensus process.
      • Khangura S.
      • Konnyu K.
      • Cushman R.
      • Grimshaw J.
      • Moher D.
      Evidence summaries: the evolution of a rapid review approach.
      • Keeney S.
      • Hasson F.
      • McKenna H.P.
      A critical review of the Delphi technique as a research methodology for nursing.
      This process enabled relevant evidence to be presented in a structured but clinically useful method to guide development of CCOSs.
      The expert panel was initiated by the lead author (GW), and this internationally recognised panel included clinicians, managers, and academics from Australia, New Zealand, United Kingdom, and the United States, with experience in implementing, developing, operationalising, educating, and evaluating CCOSs. The panel provided international diversity
      • Khodyakov D.
      • Hempel S.
      • Rubenstein L.
      • Shekelle P.
      • Foy R.
      • Salem-Schatz S.
      • et al.
      Conducting online expert panels: a feasibility and experimental replicability study.
      with a global perspective. Each panel member developed a key area within the study, enabling evidence, experience, and clinical judgement
      • Coulter I.
      • Elfenbaum P.
      • Jain S.
      • Jonas W.
      SEaRCH™ expert panel process: streamlining the link between evidence and practice.
      to be applied to all aspects of establishing a CCOS. The purpose of this expert panel was to provide a balanced and objective practical approach for establishing a CCOS. The recommendations provided are a consensus opinion of the expert panel informed by evidence, experience, and clinical judgement.
      A rapid review of the literature was undertaken by the second author (AP) to ensure appropriate evidence, if available, was considered and supported by the panel, an important process in undertaking a rapid review.
      • O'Leary D.F.
      • Casey M.
      • O'Connor L.
      • Stokes D.
      • Fealy G.M.
      • O'Brien D.
      • et al.
      Using rapid reviews: an example from a study conducted to inform policy-making.
      Whilst systematic reviews are regarded as the gold standard,
      • O'Leary D.F.
      • Casey M.
      • O'Connor L.
      • Stokes D.
      • Fealy G.M.
      • O'Brien D.
      • et al.
      Using rapid reviews: an example from a study conducted to inform policy-making.
      • Haby M.M.
      • Chapman E.
      • Clark R.
      • Barreto J.
      • Reveiz L.
      • Lavis J.N.
      What are the best methodologies for rapid reviews of the research evidence for evidence-informed decision making in health policy and practice: a rapid review.
      • Aronson J.K.
      • Heneghan C.
      • Mahtani K.R.
      • Plüddemann A.
      A word about evidence: ‘rapid reviews’ or ‘restricted reviews’?.
      rapid reviews are a pragmatic and manageable way to synthesise research findings within a short timeframe, unlike systematic reviews that take a lot longer.
      • O'Leary D.F.
      • Casey M.
      • O'Connor L.
      • Stokes D.
      • Fealy G.M.
      • O'Brien D.
      • et al.
      Using rapid reviews: an example from a study conducted to inform policy-making.
      ,
      • Aronson J.K.
      • Heneghan C.
      • Mahtani K.R.
      • Plüddemann A.
      A word about evidence: ‘rapid reviews’ or ‘restricted reviews’?.
      ,
      • Campbell F.
      • Booth A.
      • Weeks L.
      • Kaunelis D.
      • Smith A.
      A scoping review found increasing examples of rapid qualitative evidence syntheses and no methodological guidance.
      Whilst a single reviewer performing the rapid review introduces bias, an expert panel ensures the evidence is appropriate to the topic and relevant literature is not missed during the rapid review process.
      • O'Leary D.F.
      • Casey M.
      • O'Connor L.
      • Stokes D.
      • Fealy G.M.
      • O'Brien D.
      • et al.
      Using rapid reviews: an example from a study conducted to inform policy-making.
      As rapid reviews are not as broad as systematic reviews,
      • O'Leary D.F.
      • Casey M.
      • O'Connor L.
      • Stokes D.
      • Fealy G.M.
      • O'Brien D.
      • et al.
      Using rapid reviews: an example from a study conducted to inform policy-making.
      only two data bases were searched (Scopus and Web of Science) using the key words “critical care outreach”, “intensive care outreach”, intensive care liaison nurse”, and “patient at risk team”. Qualitative and quantitative articles, mixed-methods research, and discussion articles written in English and published between 2012 and 2021 (inclusive) were reviewed; no grey publications were included. References lists were reviewed to provide links to earlier studies that were relevant to the topic. Quality tools were not used to review the studies, which is not uncommon for rapid reviews owing to time restraints.
      • O'Leary D.F.
      • Casey M.
      • O'Connor L.
      • Stokes D.
      • Fealy G.M.
      • O'Brien D.
      • et al.
      Using rapid reviews: an example from a study conducted to inform policy-making.
      The Delphi methodology is commonly used to create formal consensus statements and has also been used to describe numerous important nursing practices.
      • Gill F.J.
      • Kendrick T.
      • Davies H.
      • Greenwood M.
      A two phase study to revise the Australian practice standards for specialist critical care nurses.
      ,
      • Bloomer M.J.
      • Ranse K.
      • Butler A.
      • Brooks L.
      A national position statement on adult end-of-life care in critical care.
      The Delphi methodology uses a structured process and is a scientific method for achieving expert consensus.
      • Hohmann E.
      • Brand J.C.
      • Rossi M.J.
      • Lubowitz J.H.
      Expert opinion Is necessary: Delphi panel methodology facilitates a scientific approach to consensus.
      Common to all Delphi variations is the recruitment of a panel of informed experts. We used modified online Delphi technique to obtain expert panel consensus. Online methods reduce expense related to travel and possible biases related to panel member status or personality and enabled members to participate at a convenient time to them.
      • Khodyakov D.
      • Hempel S.
      • Rubenstein L.
      • Shekelle P.
      • Foy R.
      • Salem-Schatz S.
      • et al.
      Conducting online expert panels: a feasibility and experimental replicability study.
      All panel members reviewed the final manuscipt to agree on the important elements necessary to provide a practice approach to establishing a CCOS. This process enabled knowledge translation, which aims to reduce the gap between evidence generated and decisions being made in the clinical practice setting.
      • Haby M.M.
      • Chapman E.
      • Clark R.
      • Barreto J.
      • Reveiz L.
      • Lavis J.N.
      What are the best methodologies for rapid reviews of the research evidence for evidence-informed decision making in health policy and practice: a rapid review.
      ,
      • Gainforth H.L.
      • Hoekstra F.
      • McKay R.
      • McBride C.B.
      • Sweet S.N.
      • Martin Ginis K.A.
      • et al.
      Integrated knowledge translation: guiding principles for conducting and disseminating spinal cord injury research in partnership.

      3. Findings

      The literature search identified 502 publications; 104 publications were relevant to the study and reviewed. The expert panel identified five components needed to establish a CCOS. These included the following: (i) approaches to service delivery, (ii) education and training, (iii) organisational engagement, (iv) clinical governance, and (v) monitoring and evaluation.

      3.1 Approaches to service delivery

      The key objectives of a CCOS are to avert ICU admission, enable timely ICU admission, facilitate ICU discharge, and share ICU skills with the ward interdisciplinary team.
      • Pirret A.M.
      The role and effectiveness of a nurse practitioner led critical care outreach service.
      ,
      • Ball C.
      Critical care outreach services--do they make a difference?.
      ,
      • Story D.A.
      • Shelton A.C.
      • Poustie S.J.
      • Colin-Thome N.J.
      • McIntyre R.E.
      • McNichol P.L.
      Effect of an anaesthesia department led critical care outreach and acute pain service on postoperative serious adverse events.
      Roles of nurses within a CCOS may be proactive, reactive, or a combination of both.
      • Lyons P.G.
      • Edelson D.P.
      • Churpek M.M.
      Rapid response systems.
      Proactive teams are often stand-alone teams that may use a variety of surveillance techniques to identify and prevent clinical deterioration, such as continuous vital sign monitoring and electronic risk stratification, or through other markers, such as reviewing patients after ICU discharge and proactive rounding.
      • Lyons P.G.
      • Edelson D.P.
      • Churpek M.M.
      Rapid response systems.
      ,
      • Jones D.A.
      • Dunbar N.J.
      • Bellomo R.
      Clinical deterioration in hospital inpatients: the need for another paradigm shift.
      ,
      • Williams G.
      • Rotering L.
      • Samuel A.
      • Du Plessis J.
      • Abdel Khaleq M.H.A.
      • Crilly J.
      Staff's perception of the intensive care outreach nurse role: a multisite cross-sectional study.
      Reactive teams, such as the MET, requires the patient to deteriorate before the team is activated.
      • Jones D.A.
      • Dunbar N.J.
      • Bellomo R.
      Clinical deterioration in hospital inpatients: the need for another paradigm shift.
      In both models, CCONs need to have the ability to flex rapidly from one stressful situation to another throughout a workday as well as being broadly skilled and experienced to respond appropriately to the wide variety of cases, ages, comorbidities, presenting symptoms, and ward staff's experience levels on each occasion. In addition to clinical skill and experience, CCONs require good communication, problem-solving, and bedside teaching skills.
      • McIntyre T.
      • Taylor C.
      • Bailey M.
      • Jones D.
      Differences in the characteristics, treatment, and outcomes of patient groups reviewed by intensive care liaison nurses in Australia: a multicentre prospective study.
      ,
      • Hession C.A.
      • Meaney T.
      Ward nurses’ experiences and perceptions of the critical care outreach service: a qualitative study undertaken in a large teaching hospital in the West of Ireland.
      Our experience suggests CCOSs have incorporated different approaches to achieving these objectives dependent on the current needs within each organisation and the maturity of the CCOS. Approaches include implementing an early warning scoring system (EWSS), a nurse concern trigger, ICU discharge follow-up, patient and family activated call for concern, and proactive rounding, each of which will be briefly described.

      3.1.1 Early warning scoring system

      Commonly, an afferent limb (detection limb) uses an early warning score (EWS) to identify patients requiring a CCOS review or MET.
      • Pirret A.M.
      • Takerei S.F.
      • Kazula L.M.
      The effectiveness of a patient at risk team comprised of predominantly ward experienced nurses: a before and after study.
      ,
      • Odell M.
      • Victor C.
      • Oliver D.
      Nurses' role in detecting deterioration in ward patients: systematic literature review.
      Although various EWSs exist internationally, the United Kingdom National Early Warning Score (NEWS) has been extensively researched and is mandated as a standard of care by the National Institute for Health and Clinical Excellence.
      • Fang A.H.S.
      • Lim W.T.
      • Balakrishnan T.
      Early warning score validation methodologies and performance metrics: a systematic review.
      ,
      • Oglesby K.J.
      • Sterne J.A.C.
      • Gibbison B.
      Improving early warning scores – more data, better validation, the same response.
      An EWSS is recommended in all hospitals to help guide bedside nurses to know when and how to escalate care to others, such as the CCOS.
      • Liu V.X.
      • Lu Y.
      • Carey K.A.
      • Gilbert E.R.
      • Afshar M.
      • Akel M.
      • et al.
      Comparison of early warning scoring systems for hospitalized patients with and without infection at risk for in-hospital mortality and transfer to the intensive care unit.
      An EWSS has an associated escalation strategy (often referred to as a track-and-trigger system) and uses numerical scores to multiple vital sign parameters to trigger an alert. Vital signs commonly included are heart rate, blood pressure, respiratory rate, level of consciousness, and oxygen saturation. The more deranged the vital sign is, the higher the score becomes, leading to a graded escalation response (efferent limb of the EWSS).
      • Pirret A.M.
      • Takerei S.F.
      • Kazula L.M.
      The effectiveness of a patient at risk team comprised of predominantly ward experienced nurses: a before and after study.
      ,
      • Odell M.
      • Victor C.
      • Oliver D.
      Nurses' role in detecting deterioration in ward patients: systematic literature review.
      Lack of an international EWS is likely related to the view that an EWS in one hospital may not be applicable in another owing to different patient characteristics, which suggests one single EWS may not meet the needs of unique patient populations.
      • Oglesby K.J.
      • Sterne J.A.C.
      • Gibbison B.
      Improving early warning scores – more data, better validation, the same response.
      ,
      • Pirret A.M.
      • Kazula L.M.
      Removing modifications to the New Zealand Early Warning Score- does ethnicity matter? A multimethod research design.
      This is reflected in hospitals that have developed their own EWS, such as a paediatric EWS,
      • Zachariasse J.M.
      • MacOnochie I.K.
      • Nijman R.G.
      • Greber-Platzer S.
      • Smit F.J.
      • Nieboer D.
      • et al.
      Improving the prioritization of children at the emergency department: updating the Manchester Triage System using vital signs.
      or a modified EWS.
      • Pirret A.M.
      • Kazula L.M.
      The impact of a modified New Zealand Early Warning Score (M–NZEWS) and NZEWS on ward patients triggering a medical emergency team activation: a mixed methods sequential design.

      3.1.2 Nurse concern

      Nurse concern or ‘worry factor’ is an indication of clinical deterioration;
      • Romero-Brufau S.
      • Gaines K.
      • Nicolas C.T.
      • Johnson M.G.
      • Hickman J.
      • Huddleston J.M.
      The fifth vital sign? Nurse worry predicts inpatient deterioration within 24 hours.
      hence, a concern criterion is commonly added to an EWSS to enable nurses to escalate their concerns or intuition irrespective of vital signs.
      • Kalliokoski J.
      • Kyngas H.
      • Ala-Kokko T.
      • Merilainen M.
      Insight into hospital ward nurses' concerns about patient health and the corresponding medical emergency team nurse response.
      Nurse concern is a subjective reason for concern irrespective of vital signs. Nurse concern increases with clinical experience,
      • Cioffi J.
      • Conwayt R.
      • Everist L.
      • Scott J.
      • Senior J.
      'Patients of concern' to nurses in acute care settings: a descriptive study.
      which means inexperienced nurses have difficulty using this criterion to escalate deteriorating patients.
      • Cioffi J.
      • Conwayt R.
      • Everist L.
      • Scott J.
      • Senior J.
      'Patients of concern' to nurses in acute care settings: a descriptive study.
      Packaging information enables nurses to use more convincing language when escalating patients
      • Cioffi J.
      • Conwayt R.
      • Everist L.
      • Scott J.
      • Senior J.
      'Patients of concern' to nurses in acute care settings: a descriptive study.
      and is more effective than an isolated vital sign.
      • Andrews T.
      • Waterman H.
      Packaging: a grounded theory of how to report physiological deterioration effectively.
      However, the ability to ‘package’ clinical deterioration effectively to justify escalation depends on nurses’ knowledge, confidence, and experience,
      • Massey D.
      • Chaboyer W.
      • Aitken L.
      Nurses’ perceptions of accessing a medical emergency team: a qualitative study.
      all factors that take time to develop. Douw et al.
      • Douw G.
      • Schoonhoven L.
      • Holwerda T.
      • Huisman-de Waal G.
      • van Zanten A.R.H.
      • van Achterberg T.
      • et al.
      Nurses' worry or concern and early recognition of deteriorating patients on general wards in acute care hospitals: a systematic review.
      identified nine indicators of concern that assist nurses with communicating concerns of patient deterioration to medical staff members. The Dutch-Early-Nurse-Worry Indicator Score combined with the EWS improved unplanned ICU admissions and unexpected mortality and was more predictive than the EWS or ‘nurse worry’ criteria alone.
      • Douw G.
      • Huisman-de Waal G.
      • van Zanten A.R.H.
      • van der Hoeven J.G.
      • Schoonhoven L.
      Nurses' 'worry' as predictor of deteriorating surgical ward patients: a prospective cohort study of the Dutch-Early-Nurse-Worry-Indicator-Score.
      Although “nurse concern” is recognised as an important factor in any escalation process, more studies are required to quantify the value nurse worry indicators add to an EWS.

      3.1.3 ICU patient discharge follow-up

      Jones et al.
      • Jones D.A.
      • Dunbar N.J.
      • Bellomo R.
      Clinical deterioration in hospital inpatients: the need for another paradigm shift.
      and more recently McIntyre et al.
      • McIntyre T.
      • Taylor C.
      • Bailey M.
      • Jones D.
      Differences in the characteristics, treatment, and outcomes of patient groups reviewed by intensive care liaison nurses in Australia: a multicentre prospective study.
      suggest a proactive and pre-emptive approach to clinical deterioration, such as follow-up of patients discharging from the ICU. For some CCOSs, ICU discharge is the most common reason for referral.
      • Alberto L.
      • Gillespie B.M.
      • Green A.
      • Martínez M.D.C.
      • Cañete A.
      • Zotarez H.
      • et al.
      Activities undertaken by intensive care unit liaison nurses in Argentina.
      • Pedersen A.
      • Psirides A.
      • Coombs M.
      Models and activities of critical care outreach in New Zealand hospitals: results of a national census.
      • Elliott S.
      • Chaboyer W.
      • Ernest D.
      • Doric A.
      • Endacott R.
      A national survey of Australian intensive care unit (ICU) liaison nurse (LN) services.
      • Green A.
      • Jones D.
      • McIntyre T.
      • Taylor C.
      • Chaboyer W.
      • Bailey M.
      Characteristics and outcomes of patients reviewed by intensive care unit liaison nurses in Australia: a prospective multicentre study.
      Discharging patients from the ICU to the ward is a vulnerable time for patients, exposing them to anxiety
      • Wong D.J.N.
      • Wickham A.J.
      A survey of intensive care unit discharge communication practices in the UK.
      ,
      • Cuzco C.
      • Delgado-Hito P.
      • Marín Pérez R.
      • Núñez Delgado A.
      • Romero-García M.
      • Martínez-Momblan M.A.
      • et al.
      Patients’ experience while transitioning from the intensive care unit to a ward.
      and risk of adverse events,
      • van Sluisveld N.
      • Hesselink G.
      • van der Hoeven J.G.
      • Westert G.
      • Wollersheim H.
      • Zegers M.
      Improving clinical handover between intensive care unit and general ward professionals at intensive care unit discharge.
      • Elliott M.
      • Page K.
      • Worrall-Carter L.
      • Rolley J.
      Examining adverse events after intensive care unit discharge: outcomes from a pilot questionnaire.
      • Elliott M.
      • Page K.
      • Worrall-Carter L.
      Factors associated with post-intensive care unit adverse events: a clinical validation study.
      including ICU readmission and death.
      • Wong D.J.N.
      • Wickham A.J.
      A survey of intensive care unit discharge communication practices in the UK.
      ,
      • Li P.
      • Boyd J.M.
      • Ghali W.A.
      • Stelfox H.T.
      Stakeholder views regarding patient discharge from intensive care: suboptimal quality and opportunities for improvement.
      The CCOS plays a key role in advanced assessment, technical support, and communication by ensuring written transfer information is available and understood by the ward medical and nursing staff, and by being physically present to guide the ward staff after patient transfer to the ward,
      • Pirret A.
      How well do we transition patients from ICU to the ward? Let our patients tell us.
      ,
      • Peters J.S.
      Role of transitional care measures in the prevention of readmission after critical illness.
      all factors that reduce ward nurses’ anxiety associated with receiving ICU patients.
      • Kauppi W.
      • Proos M.
      • Olausson S.
      Ward nurses' experiences of the discharge process between intensive care unit and general ward.
      Follow-up of patients discharged prematurely or out of hours during ICU bed demand may also play a role in reducing patient anxiety
      • McCairn A.J.
      • Jones C.
      Does time of transfer from critical care to the general wards affect anxiety? A pragmatic prospective cohort study.
      and mortality.
      • Pirret A.
      How well do we transition patients from ICU to the ward? Let our patients tell us.
      Three systematic reviews and meta-analyses show ICU discharge follow-up with or without transition programmes reduce the risk of ICU readmission.
      • Tanner J.
      • Cornish J.
      Routine critical care step-down programmes: systematic review and meta-analysis.
      • Niven D.J.
      • Bastos J.F.
      • Stelfox H.T.
      Critical care transition programs and the risk of readmission or death after discharge from an ICU: a systematic review and meta-analysis.
      • Österlind J.
      • Gerhardsson J.
      • Myrberg T.
      Critical care transition programs on readmission or death: a systematic review and meta-analysis.
      Although a recent systematic review suggests diverging evidence for other benefits of ICU discharge follow-up,
      • Wibrandt I.
      • Lippert A.
      Improving patient safety in handover from intensive care unit to general ward: a systematic review.
      Nates et al.
      • Nates J.L.
      • Nunnally M.
      • Kleinpell R.
      • Blosser S.
      • Goldner J.
      • Birriel B.
      • et al.
      ICU admission, discharge, and triage guidelines: a framework to enhance clinical operations, development of institutional policies, and further research.
      in their evidence-based recommendations suggest ICU discharge follow-up reduces ICU discharge delays, ward adverse advents, mortality, and ICU readmission.

      3.1.4 Patient and family activated call for concern

      Although an EWSS is useful for detecting and escalating deteriorating patients, it is not always acted upon.
      • Odell M.
      Patient- and relative-activated critical care outreach: a 7-year service review.
      This has led to an increasing emphasis on enabling patients and families to escalate their concerns to the CCOS based on the premise that patients and families recognise their deterioration before the ward staff.
      • Odell M.
      Patient- and relative-activated critical care outreach: a 7-year service review.
      • Odell M.
      • Gerber K.
      • Gager M.
      Call 4 concern: patient and relative activated critical care outreach.
      • Gill F.J.
      • Leslie G.D.
      • Marshall A.P.
      The Impact of implementation of family-initiated escalation of care for the deteriorating patient in hospital: a systematic review.
      • Thiele L.
      • Flabouris A.
      • Thompson C.
      Acute clinical deterioration and consumer escalation in the hospital setting: a literature review.
      • Dwyer T.A.
      • Flenady T.
      • Kahl J.
      • Quinney L.
      Evaluation of a patient and family activated escalation system: Ryan's Rule.
      Dwyer et al.
      • Dwyer T.A.
      • Flenady T.
      • Kahl J.
      • Quinney L.
      Evaluation of a patient and family activated escalation system: Ryan's Rule.
      identified an average of 2.5 activations per month over a 2-year period, with 35% resolved by communication alone, nearly half requiring some clinical intervention, and 15% needing transfer to a high level of care. While these services have improved service delivery to patients,
      • Odell M.
      Patient- and relative-activated critical care outreach: a 7-year service review.
      there are concerns the service may be used to respond to issues that are not related to deterioration.
      • Gill F.J.
      • Leslie G.D.
      • Marshall A.P.
      The Impact of implementation of family-initiated escalation of care for the deteriorating patient in hospital: a systematic review.
      ,
      • Thiele L.
      • Flabouris A.
      • Thompson C.
      Acute clinical deterioration and consumer escalation in the hospital setting: a literature review.
      ,
      • Strickland W.
      • Pirret A.
      • Takerei S.
      Patient and/or family activated rapid response service: patients' perceptions of deterioration and need for a service.
      Hence, more research is needed on the impact of this type of service on patient safety.

      3.1.5 Proactive rounding

      Proactive rounding may be a useful strategy when first establishing a CCOS or if the service is being underutilised.
      • Kara A.
      • Dean D.F.
      • Johnson C.S.
      • Hui S.L.
      The impact of proactive rounding on rapid response team calls: an observational study.
      It has been used as a tool to facilitate the ‘worry factor’ and escalate to the CCOS.
      • Guirgis F.W.
      • Gerdik C.
      • Wears R.L.
      • Williams D.J.
      • Kalynych C.J.
      • Sabato J.
      • et al.
      Proactive rounding by the rapid response team reduces inpatient cardiac arrests.
      Identifying ward patients who are at high risk of deteriorating enables earlier intervention and prevents further deterioration.
      • Guirgis F.W.
      • Gerdik C.
      • Wears R.L.
      • Williams D.J.
      • Kalynych C.J.
      • Sabato J.
      • et al.
      Proactive rounding by the rapid response team reduces inpatient cardiac arrests.
      Proactive rounding practices continue to evolve as automated artificial intelligence predictive models are developed.
      • Winterbottom F.A.
      The role of tele-critical care in rescue and resuscitation.

      3.2 Education and training

      Establishing a CCOS provides an opportunity to further develop clinical and interpersonal expertise that fosters support, teamwork, and collaboration.
      • Olsen S.L.
      • Søreide E.
      • Hillman K.
      • Hansen B.S.
      Succeeding with rapid response systems – a never-ending process: a systematic review of how health-care professionals perceive facilitators and barriers within the limbs of the RRS.
      ,
      • Hession C.A.
      • Meaney T.
      Ward nurses’ experiences and perceptions of the critical care outreach service: a qualitative study undertaken in a large teaching hospital in the West of Ireland.
      Building trust and establishing a positive relationship with the ward team is key to the success of a CCOS.
      • Pattison N.
      • McLellan J.
      • Roskelly L.
      • McLeod K.
      • Wiseman T.
      Managing clinical uncertainty: an ethnographic study of the impact of critical care outreach on end-of-life transitions in ward-based critically ill patients with a life-limiting illness.
      The CCON must balance their critical care expertise with the immediate needs of the ward environment, working in collaboration with ward staff to navigate the approach taken to monitor and manage the patient. Being accessible, approachable,
      • Aitken L.M.
      • Chaboyer W.
      • Vaux A.
      • Crouch S.
      • Burmeister E.
      • Daly M.
      • et al.
      Effect of a 2-tier rapid response system on patient outcome and staff satisfaction.
      ,
      • McIntyre T.
      • Taylor C.
      • Eastwood G.M.
      • Jones D.
      • Baldwin I.
      • Bellomo R.
      A survey of ward nurses attitudes to the intensive care nurse consultant service in a teaching hospital.
      ,
      • Wood T.
      • Pirret A.
      • Takerei S.
      • Harford J.
      Staff perceptions of a patient at risk team: a survey design.
      friendly, and knowledgeable
      • Aitken L.M.
      • Chaboyer W.
      • Vaux A.
      • Crouch S.
      • Burmeister E.
      • Daly M.
      • et al.
      Effect of a 2-tier rapid response system on patient outcome and staff satisfaction.
      enables CCONs to have key roles in supporting nurses’ and junior doctors’ decision-making by sharing their knowledge and skills
      • Hession C.A.
      • Meaney T.
      Ward nurses’ experiences and perceptions of the critical care outreach service: a qualitative study undertaken in a large teaching hospital in the West of Ireland.
      ,
      • Wood T.
      • Pirret A.
      • Takerei S.
      • Harford J.
      Staff perceptions of a patient at risk team: a survey design.
      ,
      • Prinsloo C.
      Self-Leadership in a critical care outreach service for quality patient care.
      and building ward nurses’ confidence in managing deteriorating patients.
      • McIntyre T.
      • Taylor C.
      • Eastwood G.M.
      • Jones D.
      • Baldwin I.
      • Bellomo R.
      A survey of ward nurses attitudes to the intensive care nurse consultant service in a teaching hospital.
      ,
      • Prinsloo C.
      Self-Leadership in a critical care outreach service for quality patient care.
      Many interventions performed by the CCOS relate to communication and education
      • McIntyre T.
      • Taylor C.
      • Bailey M.
      • Jones D.
      Differences in the characteristics, treatment, and outcomes of patient groups reviewed by intensive care liaison nurses in Australia: a multicentre prospective study.
      ,
      • Williams G.
      • Rotering L.
      • Samuel A.
      • Du Plessis J.
      • Abdel Khaleq M.H.A.
      • Crilly J.
      Staff's perception of the intensive care outreach nurse role: a multisite cross-sectional study.
      and include patient and family advocacy, which may comprise of assisting the team with early decision-making regarding patient treatment limitations and transition to end-of-life care.
      • Pattison N.
      • McLellan J.
      • Roskelly L.
      • McLeod K.
      • Wiseman T.
      Managing clinical uncertainty: an ethnographic study of the impact of critical care outreach on end-of-life transitions in ward-based critically ill patients with a life-limiting illness.
      ,
      • Hyde-Wyatt J.
      • Garside J.
      Critical care outreach: a valuable resource?.
      • Pattison N.
      • O'Gara G.
      • Wigmore T.
      Negotiating transitions: involvement of critical care outreach teams in end-of-life decision making.
      • Pattison N.
      • O'Gara G.
      Making appropriate decisions about admission to critical care: the role of critical care outreach and medical emergency teams.
      • King A.
      • Botti M.
      • McKenzie D.P.
      • Barrett J.
      • Bloomer M.J.
      End-of-life care and intensive care unit clinician involvement in a private acute care hospital: a retrospective descriptive medical record audit.
      Cross et al.
      • Cross W.M.
      • Moore A.G.
      • Sampson T.
      • Kitch C.
      • Ockerby C.
      Implementing clinical supervision for ICU outreach nurses: a case study of their journey.
      identified nurses new to CCOS need clinical supervision, role clarification, understanding how to deal with personal issues, dedicated time for reflection, and debriefing. Debriefing, managing emotional wellbeing and valuing individuals are identified as factors that reduce CCONs’ moral distress and burnout.
      • Pattison N.
      • Droney J.
      • Gruber P.
      Burnout: caring for critically ill and end-of-life patients with cancer.
      Identifying CCOS roles and responsibilities needs a structured approach. As CCOSs often combine proactive and reactive responsibilities in one role.
      • Kara A.
      • Dean D.F.
      • Johnson C.S.
      • Hui S.L.
      The impact of proactive rounding on rapid response team calls: an observational study.
      Table 1 provides a decision guide for CCOSs when facing competing priorities whilst functioning in a combined role. The CCOS can rapidly provide support to bedside staff reactively in clinical emergencies and also prevent emergencies through proactive clinical review, detection, and referral.
      Table 1Critical care outreach nurse responsibilities.
      PrioritiesRoutine activitiesRationale/measure
      Responding to ward emergencies takes immediate priority over all other CCOS activitiesExpert staff to lead & coach during high-risk events
      • 1.
        Ward emergencies
      • Attend each event & monitor
      • Assist team leader & coach staff
      • Ensure event documentation completion
      • Identify safety issues & rectify/escalate if needed
      Trained staff at each event
      • Currey J.
      • McIntyre T.
      • Taylor C.
      • Allen J.
      • Jones D.
      Critical care nurses' perceptions of essential elements for an intensive care liaison or critical care outreach nurse curriculum.


      Documentation completed

      Safety issues immediately rectified

      Hospital-wide safety support
      • Devita M.A.
      • Bellomo R.
      • Hillman K.
      • Kellum J.
      • Rotondi A.
      • Teres D.
      • et al.
      Findings of the first consensus conference on medical emergency teams.
      ,
      • Winterbottom F.A.
      • Webre H.
      Rapid response system restructure: focus on prevention and early Intervention.
      CCOS referrals
      • Attend each referral
      • Collaborate with interdisciplinary team
      • Follow practice protocols
      • Ensure prescribed orders are completed
      • Ensure appropriate patient transfer or discharge
      • Ensure referrals are documented
        • Winterbottom F.A.
        • Webre H.
        Rapid response system restructure: focus on prevention and early Intervention.
      • Escalate to critical care physician on APP as needed
      Trained staff at each event
      • Currey J.
      • McIntyre T.
      • Taylor C.
      • Allen J.
      • Jones D.
      Critical care nurses' perceptions of essential elements for an intensive care liaison or critical care outreach nurse curriculum.


      Early intervention for high-risk patients (e.g. antibiotics for sepsis)

      Triage patients to right level of care
      • Winterbottom F.A.
      • Webre H.
      Rapid response system restructure: focus on prevention and early Intervention.


      Complete documentation
      • Kara A.
      • Dean D.F.
      • Johnson C.S.
      • Hui S.L.
      The impact of proactive rounding on rapid response team calls: an observational study.
      Proactive rounding
      • Review overnight emergencies
      • Ensure CCOS review is documented in patient record
      • Follow-up on patients with emergency events overnight
      Early intervention for patients at high risk of further deterioration. High-risk patients (e.g., IV access, BiPAP)
      • Kara A.
      • Dean D.F.
      • Johnson C.S.
      • Hui S.L.
      The impact of proactive rounding on rapid response team calls: an observational study.
      ,
      • Guirgis F.W.
      • Gerdik C.
      • Wears R.L.
      • Williams D.J.
      • Kalynych C.J.
      • Sabato J.
      • et al.
      Proactive rounding by the rapid response team reduces inpatient cardiac arrests.
      ,
      • Winterbottom F.A.
      • Webre H.
      Rapid response system restructure: focus on prevention and early Intervention.


      Prevent patient deterioration

      Triage patients to right level of care

      Build relationships between units
      • McIntyre T.
      • Taylor C.
      • Eastwood G.M.
      • Jones D.
      • Baldwin I.
      • Bellomo R.
      A survey of ward nurses attitudes to the intensive care nurse consultant service in a teaching hospital.
      ,
      • Wood T.
      • Pirret A.
      • Takerei S.
      • Harford J.
      Staff perceptions of a patient at risk team: a survey design.


      Liaise with patient flow coordinators
      ∗This table is based on the work of Winterbottom et al.
      • Winterbottom F.A.
      • Webre H.
      Rapid response system restructure: focus on prevention and early Intervention.
      and has been used with permission of the lead author.
      CCOS, critical care outreach service; BiPAP, Bi-level postive airway pressure.
      A specific competency programme based on the Competences for Recognising and Responding to Acutely Ill Patients in Hospital
      Department of Health
      Competences for recognising and responding to acutely ill patients in hospital.
      and knowledge outlined in critical care nursing standards have been used to develop nurses new to CCOS roles.
      • Pirret A.M.
      • Takerei S.F.
      • Kazula L.M.
      The effectiveness of a patient at risk team comprised of predominantly ward experienced nurses: a before and after study.
      Legislation and/or local policy may define CCON's scope of practice; therefore, it is necessary to also include what needs escalating to a critical care physician or APP.
      An Australian study surveying a convenience sample of participants at an RRS conference showed CCONs attending MET calls considered interprofessional training, including clinical deterioration theory and skills, RRS governance, professionalism, and teamwork important.
      • Currey J.
      • Massey D.
      • Allen J.
      • Jones D.
      What nurses involved in a medical emergency teams consider the most vital areas of knowledge and skill when delivering care to the deteriorating ward patient. A nurse-oriented curriculum development project.
      An American before-and-after study demonstrated that a performance improvement–based inpatient resuscitation programme that included concepts of early recognition of clinical deterioration and closed-loop feedback communication decreased hospital mortality and increased survival to discharge.
      • Davis D.P.
      • Graham P.G.
      • Husa R.D.
      • Lawrence B.
      • Minokadeh A.
      • Altieri K.
      • et al.
      A performance improvement-based resuscitation programme reduces arrest incidence and increases survival from in-hospital cardiac arrest.
      Planned study sessions provided to staff to fill knowledge gaps related to detecting, responding, and managing deterioration have been used successfully in some CCOS models.
      • Pirret A.M.
      • Takerei S.F.
      • Kazula L.M.
      The effectiveness of a patient at risk team comprised of predominantly ward experienced nurses: a before and after study.
      ,
      • Williams G.
      • Rotering L.
      • Samuel A.
      • Du Plessis J.
      • Abdel Khaleq M.H.A.
      • Crilly J.
      Staff's perception of the intensive care outreach nurse role: a multisite cross-sectional study.
      Currey et al.
      • Currey J.
      • McIntyre T.
      • Taylor C.
      • Allen J.
      • Jones D.
      Critical care nurses' perceptions of essential elements for an intensive care liaison or critical care outreach nurse curriculum.
      identified CCONs’ theoretical knowledge, advanced assessment skills, and professional attributes as important in their role development.
      • Currey J.
      • McIntyre T.
      • Taylor C.
      • Allen J.
      • Jones D.
      Critical care nurses' perceptions of essential elements for an intensive care liaison or critical care outreach nurse curriculum.
      Hence, sound clinical judgement, experience, and knowledge are an essential element of a CCOS.
      • Hession C.A.
      • Meaney T.
      Ward nurses’ experiences and perceptions of the critical care outreach service: a qualitative study undertaken in a large teaching hospital in the West of Ireland.
      ,
      • Athifa M.
      • Finn J.
      • Brearley L.
      • Williams T.A.
      • Hay B.
      • Laurie K.
      • et al.
      A qualitative exploration of nurse's perception of critical outreach service: a before and after study.
      When implementing a CCOS, organisations need to determine which areas within the hospital will be supported by the CCOS, to better understand skillsets required. A CCON has specialist knowledge although it may be limited for some specialties.
      • Alzghoul M.M.
      The experience of nurses working with trauma patients in critical care and emergency settings: a qualitative study from Scottish nurses' perspective.
      There may be areas within the hospital, such as paediatric, obstetric,
      • Sultan P.
      • Arulkumaran N.
      • Rhodes A.
      Provision of critical care services for the obstetric population.
      or mental health, that may require the CCON to work outside their standard knowledge and skillset. It is therefore essential that the CCOS considers how they can contribute to patients in these specialty areas and consider knowledge and skill gaps that may need addressing.
      • Currey J.
      • Massey D.
      • Allen J.
      • Jones D.
      What nurses involved in a medical emergency teams consider the most vital areas of knowledge and skill when delivering care to the deteriorating ward patient. A nurse-oriented curriculum development project.

      3.3 Organisational engagement

      The period preceding the implementation of a CCOS can be used to engage and prepare key stakeholders. Failure to do this can result in nursing and medical staff resistance to the CCOS
      • Jeddian A.
      • Lindenmeyer A.
      • Marshall T.
      • Howard A.F.
      • Sayadi L.
      • Rashidian A.
      • et al.
      Implementation of a critical care outreach service: a qualitative study.
      and no improved patient outcomes.
      • Jeddian A.
      • Hemming K.
      • Lindenmeyer A.
      • Rashidian A.
      • Sayadi L.
      • Jafari N.
      • et al.
      Evaluation of a critical care outreach service in a middle-income country: a stepped wedge cluster randomized trial and nested qualitative study.
      Engagement activities will likely focus on ward-based registered nurses (RNs) as they frequently escalate to the CCOS
      • Smith D.
      • Cartwright M.
      • Dyson J.
      • Hartin J.
      • Aitken L.M.
      Patterns of behaviour in nursing staff actioning the afferent limb of the rapid response system (RRS): a focused ethnography.
      and junior medical staff who may also refer to the service.
      • Chua W.L.
      • Legido-Quigley H.
      • Jones D.
      • Hassan N.B.
      • Tee A.
      • Liaw S.Y.
      A call for better doctor–nurse collaboration: a qualitative study of the experiences of junior doctors and nurses in escalating care for deteriorating ward patients.
      Initially, it may be preferable for the CCOS lead to deliver a more comprehensive presentation that provides ward staff with a detailed overview of the service. These presentations could be delivered at events such as ward meetings or grand rounds. If several CCOS members are responsible for delivering the information, a standardised presentation may ensure that key messages are consistent. Suggested content for this initial presentation is summarised in Table 2.
      Table 2Suggested content for a comprehensive service overview presentation.
      ItemMinimum suggested content to hospital staffAdditional content
      WhoIntroduce the CCOS team members.
      • Consider including photographs of team members to help ward staff recognise who is who (this may be particularly useful if there are different team members with different roles/skillsets, e.g., RNs, APPs).
        • Lyons P.G.
        • Edelson D.P.
        • Churpek M.M.
        Rapid response systems.
      WhatProvide an overview of the expertise provided by team members (both in terms of knowledge and skills).
      • Emphasise what the team members can provide and, if appropriate, what they cannot.
      • Consider aligning these points to CCON service standard operating procedures.
        • Son Y.J.
        • Kim G.O.
        • Lee Y.M.
        • Oh M
        • Choi J
        Predictors of Early and Late Unplanned Intensive Care Unit Readmission: A Retrospective Cohort Study.
      WhereDescribe the remit and boundaries of the team i.e. where they will attend calls and, if appropriate, where they will not.
      • Address any variant procedures that may be used to escalate care in more remote clinical areas, e.g., satellite units, outpatient departments.
      WhenOutline the circumstances in which a referral to the team can be made. This is likely to include information on objective referral criteria (including EWS).
      • Address expectations relating to staff behaviour if there is concern or ‘worry’ about a patient
        • Douw G.
        • Huisman-de Waal G.
        • van Zanten A.R.H.
        • van der Hoeven J.G.
        • Schoonhoven L.
        Nurses' 'worry' as predictor of deteriorating surgical ward patients: a prospective cohort study of the Dutch-Early-Nurse-Worry-Indicator-Score.
        in the absence of an elevated EWS.
      • Delineate referral to the CCON from existing escalation pathways already used within the organisation, e.g., calling a ward emergency
      • If the CCON is to provide additional services (e.g., following up patients who have been stepped down from a critical care area or supporting ward staff caring for patients with a tracheostomy; receiving noninvasive ventilation; nasal high-flow oxygen therapy) consider outlining these services too.
      HowProvide information about the practicalities of referral including mobile or pager numbers if relevant. Be clear and precise about the ‘go live’ date and when staff can expect a response if they call.
      • Clarify if arrangements are different at night or during a weekend.
      • If a mechanism is going to be provided for patients and/or the relatives to contact CCON directly,
        • Odell M.
        • Gerber K.
        • Gager M.
        Call 4 concern: patient and relative activated critical care outreach.
        this may also be introduced.
      APP, advanced practice provider; EWS, early warning score; CCOS, critical care outreach service; CCON, critical care outreach nurse; RN, registered nurse.
      Many nurses favour approaching colleagues for information to inform their decision-making.
      • Marshall A.P.
      • West S.H.
      • Aitken L.M.
      Preferred information sources for clinical decision making: critical care nurses' perceptions of information accessibility and usefulness.
      Consequently, if capacity for engagement work is limited by a lack of resources, prioritising senior and/or influential personnel within ward areas, such as charge nurses and nurse educators, for the more comprehensive presentation may be useful.
      Some nurses find reviewing text-based sources of information ‘daunting’;
      • Marshall A.P.
      • West S.H.
      • Aitken L.M.
      Preferred information sources for clinical decision making: critical care nurses' perceptions of information accessibility and usefulness.
      therefore, distributing simple-to-read materials with clear and concise information about the CCOS may be helpful. This information could be delivered using a range of media such as fliers, posters, and lanyard cards. Hand delivering these resources to the ward areas potentially provides further opportunities to deliver information ‘bursts’ about the service. The success of the service may be partly contingent on the beliefs that ward staff members hold about the consequences (positive or negative) of referring to the CCOS.
      • Smith D.
      • Cartwright M.
      • Dyson J.
      • Hartin J.
      • Aitken L.M.
      Barriers and enablers of recognition and response to deteriorating patients in the acute hospital setting: a theory-driven interview study using the Theoretical Domains Framework.
      Hence, every interaction with ward staff prior to and after implementation should be an opportunity to increase credibility, build relationships, and establish trust. Digital information sources in the days immediately preceding the service can be used as prompts and cues for referral to the CCOS; this may be through computer screensavers, the staff intranet, or as part of the organisation's daily/weekly electronic bulletins.
      As junior members of the patient's primary medical team respond to a deteriorating patient alongside an external responder, such as the CCOS,
      • Rihari-Thomas J.
      • Digiacomo M.
      • Phillips J.
      • Newton P.
      • Davidson P.M.
      Clinician perspectives of barriers to effective implementation of a rapid response system in an academic health centre: a focus group study.
      establishing relationships with ward-based physicians is important. Building these relationships can be helpful for the CCOS when they are assisting junior medical staff to navigate the often complex hierarchies that can exist within hospitals.
      • McGaughey J.
      • O'Halloran P.
      • Porter S.
      • Blackwood B.
      Early warning systems and rapid response to the deteriorating patient in hospital: a systematic realist review.
      An international study of RRTs showed more than 25% of patients reviewed by a CCOS have new limitations of treatment initiated, such as a do not attempt cardiopulmonary resuscitation.
      • Bannard-Smith J.
      • Lighthall G.K.
      • Subbe C.P.
      • Durham L.
      • Welch J.
      • Bellomo R.
      • et al.
      Clinical outcomes of patients seen by rapid response teams: a template for benchmarking international teams.
      Consequently, engaging with the organisation's palliative care clinicians to agree on referral pathways for patients who initially trigger a CCOS review but whose ongoing needs are best served through palliative care may be useful.
      • King A.
      • Botti M.
      • McKenzie D.P.
      • Barrett J.
      • Bloomer M.J.
      End-of-life care and intensive care unit clinician involvement in a private acute care hospital: a retrospective descriptive medical record audit.

      3.4 Clinical governance

      There are limited studies describing or testing clinical governance models to oversee CCOS implementation, despite the need for administration and governance of RRS being identified as important factors.
      • Allen J.
      • Jones D.
      • Currey J.
      Clinician and manager perceptions of factors leading to ward patient clinical deterioration.
      Nevertheless, building a coalition of key leaders to guide change is necessary to get the CCOS strategy approved and implemented. When planning for a CCOS, a strong rationale including good data demonstrating the magnitude and impact of the current problem and how a CCOS may solve this problem is needed.
      • Williams G.
      • Rotering L.
      • Samuel A.
      • Du Plessis J.
      • Abdel Khaleq M.H.A.
      • Crilly J.
      Staff's perception of the intensive care outreach nurse role: a multisite cross-sectional study.
      Data commonly used to support the need for a CCOS include the numbers of ward cardiac arrests, ICU readmissions, MET escalations,
      • Pirret A.M.
      • Takerei S.F.
      • Kazula L.M.
      The effectiveness of a patient at risk team comprised of predominantly ward experienced nurses: a before and after study.
      ,
      • Pirret A.M.
      The role and effectiveness of a nurse practitioner led critical care outreach service.
      and patient events related to failure to escalate. Other relevant measures could include a staff survey to identify the perceived benefits of a CCOS.
      • Williams G.
      • Rotering L.
      • Samuel A.
      • Du Plessis J.
      • Abdel Khaleq M.H.A.
      • Crilly J.
      Staff's perception of the intensive care outreach nurse role: a multisite cross-sectional study.
      ,
      • Wood T.
      • Pirret A.
      • Takerei S.
      • Harford J.
      Staff perceptions of a patient at risk team: a survey design.
      Establishing a steering committee to lead and provide oversight of the RRS is recommended. This steering committee could include a nursing and medical lead for the hospital, an ICU medical and nursing lead, members of the CCOS, and nursing/medical education department representatives. Expertise from other departments could be seconded as needed, such as the hospital communication department to develop a communication strategy, the hospital informatics department to assist with a data/information strategy, or the afterhours nursing supervisor team to assist with aligning and supporting the service.
      A written draft CCOS model, ideally as part of a multidisciplinary team approach, is recommended.
      • Jones D.A.
      • Dunbar N.J.
      • Bellomo R.
      Clinical deterioration in hospital inpatients: the need for another paradigm shift.
      Significant consultation and debate during this development are critical to ensure the model is fit for purpose, robust, and accepted. The model must meet the needs of the organisation; hence, there will be some variation of models between organisations, for instance, not all CCONs come from the ICU, and in some organisations, a two-tiered approach is more desirable.
      • Pirret A.M.
      • Takerei S.F.
      • Kazula L.M.
      The effectiveness of a patient at risk team comprised of predominantly ward experienced nurses: a before and after study.
      ,
      • Aitken L.M.
      • Chaboyer W.
      • Vaux A.
      • Crouch S.
      • Burmeister E.
      • Daly M.
      • et al.
      Effect of a 2-tier rapid response system on patient outcome and staff satisfaction.
      ,
      • Williams G.
      • Hughes V.
      • Timms J.
      • Raftery C.
      Emergency nurse as hospital clinical team coordinator - shining a light into the night.
      Major activities, action plans, responsible person, and timelines in a Gantt chart format are useful to ensure disciplined and transparent project management. Finally, a communication plan is essential including a written draft summary of the model and expectations of the CCOS and how staff will access and utilise the service. Included in the draft model will be how the CCOSs escalate their concerns to the parent team or critical care APP or physician and how adverse events are managed.
      • Jones D.
      • Warrillow S.
      Clinical deterioration in cancer patients-the role of the rapid response team.
      Following establishment of the service, CCOS representation at a hospital-wide deteriorating patient forum would ensure the CCOS is embedded as part of the organisation's permanent RRS.

      3.5 Monitoring and evaluation

      RRSs have a comprehensive set of measures to evaluate the effectiveness of the services that have been developed and tested over time by various organisations and professional groups.
      • Subbe C.P.
      • Bannard-Smith J.
      • Bunch J.
      • Champunot R.
      • DeVita M.A.
      • Durham L.
      • et al.
      Quality metrics for the evaluation of rapid response systems: proceedings from the third international consensus conference on rapid response systems.
      Before establishing a CCOS, there needs to be clear objectives as to what the service is aiming to achieve. Using national or international recommendations to establish an RRS
      • Devita M.A.
      • Bellomo R.
      • Hillman K.
      • Kellum J.
      • Rotondi A.
      • Teres D.
      • et al.
      Findings of the first consensus conference on medical emergency teams.
      may not be enough to convince individual organisations; hence, local data can be instrumental in supporting business cases and service development requests. Point prevalence surveys of vital sign recording practice may be used to highlight gaps in patient care
      • Chellel A.
      • Fraser J.
      • Fender V.
      • Higgs D.
      • Buras-Rees S.
      • Hook L.
      • et al.
      Nursing observations on ward patients at risk of critical illness.
      and make the case for implementing a CCOS. In addition to more objective patient outcome measures, surveys of CCOS team members and end users of the CCOS after implementation of the CCOS can monitor the effectiveness of the service, enabling experience and feedback to improve components of the service.
      • Williams G.
      • Rotering L.
      • Samuel A.
      • Du Plessis J.
      • Abdel Khaleq M.H.A.
      • Crilly J.
      Staff's perception of the intensive care outreach nurse role: a multisite cross-sectional study.
      ,
      • Wood T.
      • Pirret A.
      • Takerei S.
      • Harford J.
      Staff perceptions of a patient at risk team: a survey design.
      How CCOSs are evaluated will depend on several factors, such as organisational context, model of service, staff expertise, administration resources, information technology (IT) availability, staffing resource, and organisational objectives. A list of the measures commonly used to evaluate the service is found in Box 1; these can be adapted according to the organisation's specific objectives and service model. Collected monthly, the data can give an overview on how the CCOS is being utilised and how work patterns may be affected by hospital admissions, ICU occupancy, seasonal affects, and systemic organisational changes. Data should be collected for at least 12 months to determine the impact of the CCOS on patient care and experience. By measuring the CCOS service activity and outcomes, a foundation can be established on which to build improvements that not only effect patient care, outcome, and experience but can also be shared with other organisations.
      Metrics used to evaluate a CCOS.
      • Number of ward patients to determine
        • number of CCOS referrals
          • McIntyre T.
          • Taylor C.
          • Bailey M.
          • Jones D.
          Differences in the characteristics, treatment, and outcomes of patient groups reviewed by intensive care liaison nurses in Australia: a multicentre prospective study.
          ,
          • Alberto L.
          • Zotárez H.
          • Cañete Á.
          • Niklas J.E.
          • Enriquez J.M.
          • Gerónimo M.R.
          • et al.
          A description of the ICU liaison nurse role in Argentina.
          ,
          • Alberto L.
          • Gillespie B.M.
          • Green A.
          • Martínez M.D.C.
          • Cañete A.
          • Zotarez H.
          • et al.
          Activities undertaken by intensive care unit liaison nurses in Argentina.
          per 1000 admissions
        • number of CCOS reviews
          • McIntyre T.
          • Taylor C.
          • Bailey M.
          • Jones D.
          Differences in the characteristics, treatment, and outcomes of patient groups reviewed by intensive care liaison nurses in Australia: a multicentre prospective study.
          ,
          • Alberto L.
          • Zotárez H.
          • Cañete Á.
          • Niklas J.E.
          • Enriquez J.M.
          • Gerónimo M.R.
          • et al.
          A description of the ICU liaison nurse role in Argentina.
          per 1000 admissions
        • number of ward cardiopulmonary arrests per 1000 admissions
          • Devita M.A.
          • Bellomo R.
          • Hillman K.
          • Kellum J.
          • Rotondi A.
          • Teres D.
          • et al.
          Findings of the first consensus conference on medical emergency teams.
          ,
          • Kara A.
          • Dean D.F.
          • Johnson C.S.
          • Hui S.L.
          The impact of proactive rounding on rapid response team calls: an observational study.
        • number of MET calls per 1000 admissions.
          • Devita M.A.
          • Bellomo R.
          • Hillman K.
          • Kellum J.
          • Rotondi A.
          • Teres D.
          • et al.
          Findings of the first consensus conference on medical emergency teams.
          ,
          • Kara A.
          • Dean D.F.
          • Johnson C.S.
          • Hui S.L.
          The impact of proactive rounding on rapid response team calls: an observational study.
      • Analysis of the CCOS referrals to identify workload that may include:
        • day of week, time of day, specialty, and ward.
        • CCOSs response times according to agreed criteria based on the acuity of the patient
          • Kara A.
          • Dean D.F.
          • Johnson C.S.
          • Hui S.L.
          The impact of proactive rounding on rapid response team calls: an observational study.
        • The number of patients discharged from critical care to the ward and/or followed up within 24 h
          • Olsen S.L.
          • Søreide E.
          • Hillman K.
          • Hansen B.S.
          Succeeding with rapid response systems – a never-ending process: a systematic review of how health-care professionals perceive facilitators and barriers within the limbs of the RRS.
        • CCOS activities such as taking and analysing blood gas samples, intravenous line insertion
          • McIntyre T.
          • Taylor C.
          • Bailey M.
          • Jones D.
          Differences in the characteristics, treatment, and outcomes of patient groups reviewed by intensive care liaison nurses in Australia: a multicentre prospective study.
          ,
          • Alberto L.
          • Zotárez H.
          • Cañete Á.
          • Niklas J.E.
          • Enriquez J.M.
          • Gerónimo M.R.
          • et al.
          A description of the ICU liaison nurse role in Argentina.
          ,
          • Alberto L.
          • Gillespie B.M.
          • Green A.
          • Martínez M.D.C.
          • Cañete A.
          • Zotarez H.
          • et al.
          Activities undertaken by intensive care unit liaison nurses in Argentina.
      • Review of ward cardiac arrest patients to identify potential delays in escalation.
        • Devita M.A.
        • Bellomo R.
        • Hillman K.
        • Kellum J.
        • Rotondi A.
        • Teres D.
        • et al.
        Findings of the first consensus conference on medical emergency teams.
        ,
        • Alberto L.
        • Gillespie B.M.
        • Green A.
        • Martínez M.D.C.
        • Cañete A.
        • Zotarez H.
        • et al.
        Activities undertaken by intensive care unit liaison nurses in Argentina.
        ,
        • Kara A.
        • Dean D.F.
        • Johnson C.S.
        • Hui S.L.
        The impact of proactive rounding on rapid response team calls: an observational study.
      • Number of patient or family activations to the CCOS
        • Kalliokoski J.
        • Kyngas H.
        • Ala-Kokko T.
        • Merilainen M.
        Insight into hospital ward nurses' concerns about patient health and the corresponding medical emergency team nurse response.
        ,
        • Douw G.
        • Schoonhoven L.
        • Holwerda T.
        • Huisman-de Waal G.
        • van Zanten A.R.H.
        • van Achterberg T.
        • et al.
        Nurses' worry or concern and early recognition of deteriorating patients on general wards in acute care hospitals: a systematic review.
        ,
        • Kara A.
        • Dean D.F.
        • Johnson C.S.
        • Hui S.L.
        The impact of proactive rounding on rapid response team calls: an observational study.
      • Number of serious incidents related to sub-optimal care of a deteriorating patient.
        • Alberto L.
        • Gillespie B.M.
        • Green A.
        • Martínez M.D.C.
        • Cañete A.
        • Zotarez H.
        • et al.
        Activities undertaken by intensive care unit liaison nurses in Argentina.
        ,
        • Kara A.
        • Dean D.F.
        • Johnson C.S.
        • Hui S.L.
        The impact of proactive rounding on rapid response team calls: an observational study.
      • The number of readmissions to ICU within a specified period of time (such as 48 h).
        • Olsen S.L.
        • Søreide E.
        • Hillman K.
        • Hansen B.S.
        Succeeding with rapid response systems – a never-ending process: a systematic review of how health-care professionals perceive facilitators and barriers within the limbs of the RRS.
        [
        • Son Y.J.
        • Kim G.O.
        • Lee Y.M.
        • Oh M
        • Choi J
        Predictors of Early and Late Unplanned Intensive Care Unit Readmission: A Retrospective Cohort Study.
        ]
      An electronic database rather than a paper-based system is better for the CCOS data collection; hence, the reason why early discussion with IT services when developing the CCOS is useful. Regular feedback of the data to governance bodies will establish a basis for service development, staffing levels, areas of concern, and where patient care and experience have seen an improvement.

      4. Conclusion

      The interchangeable terms and lack of a single model for a CCOS means research is difficult to interpret in the real-word setting. In addition, limited research into the most effective or appropriate administrative and governance arrangements for an RRS and CCOS required the expert panel–modified Delphi approach to inform commentary; further research into these elements of a CCOS is recommended. Using an expert panel, a rapid review of the literature and a modified Delphi technique to combine evidence, experience, and clinical judgment effectively developed a practical approach to establishing a CCOS. Five key components needed to establish a CCOS were identified and included approaches to service delivery, education and training, organisational engagement, clinical governance, and monitoring and evaluation.
      The expert panel research design successfully synthesised evidence, experience, and clinical judgement to provide a practical approach for those establishing a CCOS, thereby reducing the evidence to clinical practice gap. This method of research will likely be valuable in other areas of practice where terms are used interchangeably, and the literature is diverse and lacking a single approach to practice.

      Funding

      This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

      Copyright

      If the article is accepted, authors assign copyright to Australian Critical Care.

      Conflict of interest

      The authors have no known conflict of interest associated with this work.

      CRediT authorship contribution statement

      Ged Williams: design and work allocation of the article, responsible for at least one section each and worked collaboratively to integrate the article into a whole, Writing – review & editing. Alison Pirret: responsible for at least one section each and worked collaboratively to integrate the article into a whole, Writing – review & editing. Nicki Credland: responsible for at least one section each and worked collaboratively to integrate the article into a whole, Writing – review & editing. Mandy Odell: responsible for at least one section each and worked collaboratively to integrate the article into a whole, Writing – review & editing. Chris Raftery: responsible for at least one section each and worked collaboratively to integrate the article into a whole, Writing – review & editing. Duncan Smith: responsible for at least one section each and worked collaboratively to integrate the article into a whole, Writing – review & editing. Fiona Winterbottom: responsible for at least one section each and worked collaboratively to integrate the article into a whole, Writing – review & editing. Debbie Massey: responsible for at least one section each and worked collaboratively to integrate the article into a whole, Writing – review & editing.

      References

        • Cretikos M.A.
        • Chen J.
        • Hillman K.M.
        • Bellomo R.
        • Finfer S.R.
        • Flabouris A.
        The effectiveness of implementation of the medical emergency team (MET) system and factors associated with use during the MERIT study.
        Crit Care Resusc: J Aust Acad Crit Care Med. 2007; 9: 206-212
        • Bellomo R.
        • Goldsmith D.
        • Uchino S.
        • Buckmaster J.
        • Hart G.
        • Opdam H.
        • et al.
        Prospective controlled trial of effect of medical emergency team on postoperative morbidity and mortality rates.
        Crit Care Med. 2004; 32: 916-921
        • Massey D.
        • Aitken L.M.
        • Chaboyer W.
        The impact of a nurse led rapid response system on adverse, major adverse events and activation of the medical emergency team.
        Intensive Crit Care Nurs. 2015; 31: 83-90
        • Devita M.A.
        • Bellomo R.
        • Hillman K.
        • Kellum J.
        • Rotondi A.
        • Teres D.
        • et al.
        Findings of the first consensus conference on medical emergency teams.
        Crit Care Med. 2006; 34: 2463-2478
        • Lyons P.G.
        • Edelson D.P.
        • Churpek M.M.
        Rapid response systems.
        Resuscitation. 2018; 128: 191-197
        • Garry L.
        • Rohan N.
        • O'Connor T.
        • Patton D.
        • Moore Z.
        Do nurse-led critical care outreach services impact inpatient mortality rates?.
        Nurs Crit Care. 2019; 24: 40-46
        • Lynch J.
        • Cope V.
        • Murray M.
        The intensive care unit liaison nurse and their value in averting clinical deterioration: a qualitative descriptive study.
        Intensive Crit Care Nurs. 2021; 63
        • McIntyre T.
        • Taylor C.
        • Bailey M.
        • Jones D.
        Differences in the characteristics, treatment, and outcomes of patient groups reviewed by intensive care liaison nurses in Australia: a multicentre prospective study.
        Aust Crit Care. 2019; 32: 403-409
        • Alberto L.
        • Zotárez H.
        • Cañete Á.
        • Niklas J.E.
        • Enriquez J.M.
        • Gerónimo M.R.
        • et al.
        A description of the ICU liaison nurse role in Argentina.
        Intensive Crit Care Nurs. 2014; 30: 31-37
        • Pirret A.M.
        • Takerei S.F.
        • Kazula L.M.
        The effectiveness of a patient at risk team comprised of predominantly ward experienced nurses: a before and after study.
        Intensive Crit Care Nurs. 2015; 31: 133-140
        • Olsen S.L.
        • Søreide E.
        • Hillman K.
        • Hansen B.S.
        Succeeding with rapid response systems – a never-ending process: a systematic review of how health-care professionals perceive facilitators and barriers within the limbs of the RRS.
        Resuscitation. 2019; 144: 75-90
        • Pirret A.M.
        The role and effectiveness of a nurse practitioner led critical care outreach service.
        Intensive Crit Care Nurs. 2008; 24: 327-384
        • Kapu A.N.
        Addition of acute care nurse practitioners to medical and surgical rapid response teams: a pilot project.
        Crit Care Nurse. 2014; 34: 51-59
        • So H.M.
        • Yan W.W.
        • Chair S.Y.
        A nurse-led critical care outreach program to reduce readmission to the intensive care unit: a quasi-experimental study with a historical control group.
        Aust Crit Care. 2019; 32: 494-501
        • Kovacs C.
        Outreach and early warning systems for the prevention of intensive care admission and death of critically ill adult patients on general hospital wards.
        Int J Nurs Pract. 2016; 22: 523-525
        • Aitken L.M.
        • Chaboyer W.
        • Vaux A.
        • Crouch S.
        • Burmeister E.
        • Daly M.
        • et al.
        Effect of a 2-tier rapid response system on patient outcome and staff satisfaction.
        Aust Crit Care. 2015; 28: 107-114
        • Tillmann B.W.
        • Klingel M.L.
        • McLeod S.L.
        • Anderson S.
        • Haddara W.
        • Parry N.G.
        The impact of delayed critical care outreach team activation on in-hospital mortality and other patient outcomes: a historical cohort study.
        Can J Anesth. 2018; 65: 1210-1217
        • Pattison N.
        • Eastham E.
        Critical care outreach referrals: a mixed-method investigative study of outcomes and experiences.
        Nurs Crit Care. 2012; 17: 71-82
        • Allen E.
        • Elliott D.
        • Jackson D.
        Recognising and responding to in-hospital clinical deterioration: an integrative review of interprofessional practice issues.
        J Clin Nurs. 2017; 26: 3990-4012
        • Haugland H.
        • Rehn M.
        • Klepstad P.
        • Krüger A.
        • Albrektsen G.E.
        • Berlac P.A.
        • et al.
        Developing quality indicators for physician-staffed emergency medical services: a consensus process.
        Scand J Trauma Resusc Emerg Med. 2017; 25
        • Khangura S.
        • Konnyu K.
        • Cushman R.
        • Grimshaw J.
        • Moher D.
        Evidence summaries: the evolution of a rapid review approach.
        Syst Rev. 2012; 1
        • Keeney S.
        • Hasson F.
        • McKenna H.P.
        A critical review of the Delphi technique as a research methodology for nursing.
        Int J Nurs Stud. 2001; 38: 195-200
        • Khodyakov D.
        • Hempel S.
        • Rubenstein L.
        • Shekelle P.
        • Foy R.
        • Salem-Schatz S.
        • et al.
        Conducting online expert panels: a feasibility and experimental replicability study.
        BMC Med Res Methodol. 2011; 11
        • Coulter I.
        • Elfenbaum P.
        • Jain S.
        • Jonas W.
        SEaRCH™ expert panel process: streamlining the link between evidence and practice.
        BMC Res Notes. 2016; 9
        • O'Leary D.F.
        • Casey M.
        • O'Connor L.
        • Stokes D.
        • Fealy G.M.
        • O'Brien D.
        • et al.
        Using rapid reviews: an example from a study conducted to inform policy-making.
        J Adv Nurs. 2017; 73: 742-752
        • Haby M.M.
        • Chapman E.
        • Clark R.
        • Barreto J.
        • Reveiz L.
        • Lavis J.N.
        What are the best methodologies for rapid reviews of the research evidence for evidence-informed decision making in health policy and practice: a rapid review.
        Health Res Pol Syst. 2016; 14
        • Aronson J.K.
        • Heneghan C.
        • Mahtani K.R.
        • Plüddemann A.
        A word about evidence: ‘rapid reviews’ or ‘restricted reviews’?.
        BMJ Evidence Based Med. 2018; 23: 204-205
        • Campbell F.
        • Booth A.
        • Weeks L.
        • Kaunelis D.
        • Smith A.
        A scoping review found increasing examples of rapid qualitative evidence syntheses and no methodological guidance.
        J Clin Epidemiol. 2019; 115: 160-171
        • Gill F.J.
        • Kendrick T.
        • Davies H.
        • Greenwood M.
        A two phase study to revise the Australian practice standards for specialist critical care nurses.
        Aust Crit Care. 2017; 30: 173-181
        • Bloomer M.J.
        • Ranse K.
        • Butler A.
        • Brooks L.
        A national position statement on adult end-of-life care in critical care.
        Aust Crit Care. 2021;
        • Hohmann E.
        • Brand J.C.
        • Rossi M.J.
        • Lubowitz J.H.
        Expert opinion Is necessary: Delphi panel methodology facilitates a scientific approach to consensus.
        Arthrosc J Arthrosc Relat Surg. 2018; 34: 349-351
        • Gainforth H.L.
        • Hoekstra F.
        • McKay R.
        • McBride C.B.
        • Sweet S.N.
        • Martin Ginis K.A.
        • et al.
        Integrated knowledge translation: guiding principles for conducting and disseminating spinal cord injury research in partnership.
        Arch Phys Med Rehabil. 2021; 102: 656-663
        • Ball C.
        Critical care outreach services--do they make a difference?.
        Intensive Crit Care Nurs. 2002; 18: 257-260
        • Story D.A.
        • Shelton A.C.
        • Poustie S.J.
        • Colin-Thome N.J.
        • McIntyre R.E.
        • McNichol P.L.
        Effect of an anaesthesia department led critical care outreach and acute pain service on postoperative serious adverse events.
        Anaesthesia. 2006; 61: 24-28
        • Jones D.A.
        • Dunbar N.J.
        • Bellomo R.
        Clinical deterioration in hospital inpatients: the need for another paradigm shift.
        Med J Aust. 2012; 196: 97-100
        • Williams G.
        • Rotering L.
        • Samuel A.
        • Du Plessis J.
        • Abdel Khaleq M.H.A.
        • Crilly J.
        Staff's perception of the intensive care outreach nurse role: a multisite cross-sectional study.
        J Nurs Care Qual. 2019; 34: 352-357
        • Hession C.A.
        • Meaney T.
        Ward nurses’ experiences and perceptions of the critical care outreach service: a qualitative study undertaken in a large teaching hospital in the West of Ireland.
        Nurs Crit Care. 2022; 27: 19-26
        • Odell M.
        • Victor C.
        • Oliver D.
        Nurses' role in detecting deterioration in ward patients: systematic literature review.
        J Adv Nurs. 2009; 65: 1992-2006
        • Fang A.H.S.
        • Lim W.T.
        • Balakrishnan T.
        Early warning score validation methodologies and performance metrics: a systematic review.
        BMC Med Inf Decis Making. 2020; : 20
        • Oglesby K.J.
        • Sterne J.A.C.
        • Gibbison B.
        Improving early warning scores – more data, better validation, the same response.
        Anaesthesia. 2020; 75: 149-151
        • Liu V.X.
        • Lu Y.
        • Carey K.A.
        • Gilbert E.R.
        • Afshar M.
        • Akel M.
        • et al.
        Comparison of early warning scoring systems for hospitalized patients with and without infection at risk for in-hospital mortality and transfer to the intensive care unit.
        JAMA Netw Open. 2020; 3e205191
        • Pirret A.M.
        • Kazula L.M.
        Removing modifications to the New Zealand Early Warning Score- does ethnicity matter? A multimethod research design.
        Intensive Crit Care Nurs. 2021; : 103141
        • Zachariasse J.M.
        • MacOnochie I.K.
        • Nijman R.G.
        • Greber-Platzer S.
        • Smit F.J.
        • Nieboer D.
        • et al.
        Improving the prioritization of children at the emergency department: updating the Manchester Triage System using vital signs.
        PLos One. 2021; 16
        • Pirret A.M.
        • Kazula L.M.
        The impact of a modified New Zealand Early Warning Score (M–NZEWS) and NZEWS on ward patients triggering a medical emergency team activation: a mixed methods sequential design.
        Intensive Crit Care Nurs. 2021; 62: 102963
        • Romero-Brufau S.
        • Gaines K.
        • Nicolas C.T.
        • Johnson M.G.
        • Hickman J.
        • Huddleston J.M.
        The fifth vital sign? Nurse worry predicts inpatient deterioration within 24 hours.
        JAMIA Open. 2019; 2: 465-470
        • Kalliokoski J.
        • Kyngas H.
        • Ala-Kokko T.
        • Merilainen M.
        Insight into hospital ward nurses' concerns about patient health and the corresponding medical emergency team nurse response.
        Intensive Crit Care Nurs. 2019; 53: 100-108
        • Cioffi J.
        • Conwayt R.
        • Everist L.
        • Scott J.
        • Senior J.
        'Patients of concern' to nurses in acute care settings: a descriptive study.
        Aust Crit Care. 2009; 22: 178-186
        • Andrews T.
        • Waterman H.
        Packaging: a grounded theory of how to report physiological deterioration effectively.
        J Adv Nurs. 2005; 52: 473-481
        • Massey D.
        • Chaboyer W.
        • Aitken L.
        Nurses’ perceptions of accessing a medical emergency team: a qualitative study.
        Aust Crit Care. 2014; 27: 133-138
        • Douw G.
        • Schoonhoven L.
        • Holwerda T.
        • Huisman-de Waal G.
        • van Zanten A.R.H.
        • van Achterberg T.
        • et al.
        Nurses' worry or concern and early recognition of deteriorating patients on general wards in acute care hospitals: a systematic review.
        Crit Care. 2015; 19
        • Douw G.
        • Huisman-de Waal G.
        • van Zanten A.R.H.
        • van der Hoeven J.G.
        • Schoonhoven L.
        Nurses' 'worry' as predictor of deteriorating surgical ward patients: a prospective cohort study of the Dutch-Early-Nurse-Worry-Indicator-Score.
        Int J Nurs Stud. 2016; 59: 134-140
        • Alberto L.
        • Gillespie B.M.
        • Green A.
        • Martínez M.D.C.
        • Cañete A.
        • Zotarez H.
        • et al.
        Activities undertaken by intensive care unit liaison nurses in Argentina.
        Aust Crit Care. 2017; 30: 74-78
        • Pedersen A.
        • Psirides A.
        • Coombs M.
        Models and activities of critical care outreach in New Zealand hospitals: results of a national census.
        Nurs Crit Care. 2016; 21: 233-242
        • Elliott S.
        • Chaboyer W.
        • Ernest D.
        • Doric A.
        • Endacott R.
        A national survey of Australian intensive care unit (ICU) liaison nurse (LN) services.
        Aust Crit Care. 2012; 25: 253-262
        • Green A.
        • Jones D.
        • McIntyre T.
        • Taylor C.
        • Chaboyer W.
        • Bailey M.
        Characteristics and outcomes of patients reviewed by intensive care unit liaison nurses in Australia: a prospective multicentre study.
        Crit Care Resusc. 2015; 17: 244-252
        • Wong D.J.N.
        • Wickham A.J.
        A survey of intensive care unit discharge communication practices in the UK.
        J Intensive Care Soc. 2013; 14: 330-333
        • Cuzco C.
        • Delgado-Hito P.
        • Marín Pérez R.
        • Núñez Delgado A.
        • Romero-García M.
        • Martínez-Momblan M.A.
        • et al.
        Patients’ experience while transitioning from the intensive care unit to a ward.
        Nurs Crit Care. 2016; https://doi.org/10.1111/nicc.12697
        • van Sluisveld N.
        • Hesselink G.
        • van der Hoeven J.G.
        • Westert G.
        • Wollersheim H.
        • Zegers M.
        Improving clinical handover between intensive care unit and general ward professionals at intensive care unit discharge.
        Intensive Care Med. 2015; 41: 589-604
        • Elliott M.
        • Page K.
        • Worrall-Carter L.
        • Rolley J.
        Examining adverse events after intensive care unit discharge: outcomes from a pilot questionnaire.
        Int J Nurs Pract. 2013; 19: 479-486
        • Elliott M.
        • Page K.
        • Worrall-Carter L.
        Factors associated with post-intensive care unit adverse events: a clinical validation study.
        Nurs Crit Care. 2014; 19: 228-235
        • Li P.
        • Boyd J.M.
        • Ghali W.A.
        • Stelfox H.T.
        Stakeholder views regarding patient discharge from intensive care: suboptimal quality and opportunities for improvement.
        Can Respir J. 2015; 22: 109-118
        • Pirret A.
        How well do we transition patients from ICU to the ward? Let our patients tell us.
        Intensive Crit Care Nurs. 2020; 58
        • Peters J.S.
        Role of transitional care measures in the prevention of readmission after critical illness.
        Crit Care Nurse. 2017; 37: e10-e17
        • Kauppi W.
        • Proos M.
        • Olausson S.
        Ward nurses' experiences of the discharge process between intensive care unit and general ward.
        Nurs Crit Care. 2018; 23: 127-133
        • McCairn A.J.
        • Jones C.
        Does time of transfer from critical care to the general wards affect anxiety? A pragmatic prospective cohort study.
        Intensive Crit Care Nurs. 2014; 30: 219-225
        • Tanner J.
        • Cornish J.
        Routine critical care step-down programmes: systematic review and meta-analysis.
        Nurs Crit Care. 2021; 26: 118-127
        • Niven D.J.
        • Bastos J.F.
        • Stelfox H.T.
        Critical care transition programs and the risk of readmission or death after discharge from an ICU: a systematic review and meta-analysis.
        Crit Care Med. 2014; 42: 179-187
        • Österlind J.
        • Gerhardsson J.
        • Myrberg T.
        Critical care transition programs on readmission or death: a systematic review and meta-analysis.
        Acta Anaesthesiol Scand. 2020; 64: 870-883
        • Wibrandt I.
        • Lippert A.
        Improving patient safety in handover from intensive care unit to general ward: a systematic review.
        J Patient Saf. 2020; 16: 199-210
        • Nates J.L.
        • Nunnally M.
        • Kleinpell R.
        • Blosser S.
        • Goldner J.
        • Birriel B.
        • et al.
        ICU admission, discharge, and triage guidelines: a framework to enhance clinical operations, development of institutional policies, and further research.
        Crit Care Med. 2016; 44: 1553-1602
        • Odell M.
        Patient- and relative-activated critical care outreach: a 7-year service review.
        Br J Nurs. 2019; 28: 116-121
        • Odell M.
        • Gerber K.
        • Gager M.
        Call 4 concern: patient and relative activated critical care outreach.
        Br J Nurs. 2010; 19: 1390-1395
        • Gill F.J.
        • Leslie G.D.
        • Marshall A.P.
        The Impact of implementation of family-initiated escalation of care for the deteriorating patient in hospital: a systematic review.
        Worldviews Evidence-Based Nurs. 2016; 13: 303-313
        • Thiele L.
        • Flabouris A.
        • Thompson C.
        Acute clinical deterioration and consumer escalation in the hospital setting: a literature review.
        Resuscitation. 2020; 156: 72-83
        • Dwyer T.A.
        • Flenady T.
        • Kahl J.
        • Quinney L.
        Evaluation of a patient and family activated escalation system: Ryan's Rule.
        Aust Crit Care. 2020; 33: 39-46
        • Strickland W.
        • Pirret A.
        • Takerei S.
        Patient and/or family activated rapid response service: patients' perceptions of deterioration and need for a service.
        Intensive Crit Care Nurs. 2019; 51: 20-26
        • Kara A.
        • Dean D.F.
        • Johnson C.S.
        • Hui S.L.
        The impact of proactive rounding on rapid response team calls: an observational study.
        Hosp Pract. 2019; 47: 42-45
        • Guirgis F.W.
        • Gerdik C.
        • Wears R.L.
        • Williams D.J.
        • Kalynych C.J.
        • Sabato J.
        • et al.
        Proactive rounding by the rapid response team reduces inpatient cardiac arrests.
        Resuscitation. 2013; 84: 1668-1673
        • Winterbottom F.A.
        The role of tele-critical care in rescue and resuscitation.
        Crit Care Nurs Clin N Am. 2021; 33: 357-368
        • Pattison N.
        • McLellan J.
        • Roskelly L.
        • McLeod K.
        • Wiseman T.
        Managing clinical uncertainty: an ethnographic study of the impact of critical care outreach on end-of-life transitions in ward-based critically ill patients with a life-limiting illness.
        J Clin Nurs. 2018; 27: 3900-3912
        • McIntyre T.
        • Taylor C.
        • Eastwood G.M.
        • Jones D.
        • Baldwin I.
        • Bellomo R.
        A survey of ward nurses attitudes to the intensive care nurse consultant service in a teaching hospital.
        Aust Crit Care. 2012; 25: 100-109
        • Wood T.
        • Pirret A.
        • Takerei S.
        • Harford J.
        Staff perceptions of a patient at risk team: a survey design.
        Intensive Crit Care Nurs. 2017; 43: 94-100
        • Prinsloo C.
        Self-Leadership in a critical care outreach service for quality patient care.
        Afr J Nurs Midwifery. 2020; 22
        • Hyde-Wyatt J.
        • Garside J.
        Critical care outreach: a valuable resource?.
        Nurs Crit Care. 2020; 25: 16-23
        • Pattison N.
        • O'Gara G.
        • Wigmore T.
        Negotiating transitions: involvement of critical care outreach teams in end-of-life decision making.
        Am J Crit Care. 2015; 24: 232-240
        • Pattison N.
        • O'Gara G.
        Making appropriate decisions about admission to critical care: the role of critical care outreach and medical emergency teams.
        Nurs Crit Care. 2014; 19: 4-6
        • King A.
        • Botti M.
        • McKenzie D.P.
        • Barrett J.
        • Bloomer M.J.
        End-of-life care and intensive care unit clinician involvement in a private acute care hospital: a retrospective descriptive medical record audit.
        Aust Crit Care. 2021; 34: 452-459
        • Cross W.M.
        • Moore A.G.
        • Sampson T.
        • Kitch C.
        • Ockerby C.
        Implementing clinical supervision for ICU outreach nurses: a case study of their journey.
        Aust Crit Care. 2012; 25: 263-270
        • Pattison N.
        • Droney J.
        • Gruber P.
        Burnout: caring for critically ill and end-of-life patients with cancer.
        Nurs Crit Care. 2020; 25: 93-101
        • Department of Health
        Competences for recognising and responding to acutely ill patients in hospital.
        2008 (London)
        • Currey J.
        • Massey D.
        • Allen J.
        • Jones D.
        What nurses involved in a medical emergency teams consider the most vital areas of knowledge and skill when delivering care to the deteriorating ward patient. A nurse-oriented curriculum development project.
        Nurse Educ Today. 2018; 67: 77-82
        • Davis D.P.
        • Graham P.G.
        • Husa R.D.
        • Lawrence B.
        • Minokadeh A.
        • Altieri K.
        • et al.
        A performance improvement-based resuscitation programme reduces arrest incidence and increases survival from in-hospital cardiac arrest.
        Resuscitation. 2015; 92: 63-69
        • Currey J.
        • McIntyre T.
        • Taylor C.
        • Allen J.
        • Jones D.
        Critical care nurses' perceptions of essential elements for an intensive care liaison or critical care outreach nurse curriculum.
        Aust Crit Care. 2021;
        • Athifa M.
        • Finn J.
        • Brearley L.
        • Williams T.A.
        • Hay B.
        • Laurie K.
        • et al.
        A qualitative exploration of nurse's perception of critical outreach service: a before and after study.
        Aust Crit Care. 2011; 24: 39-47
        • Alzghoul M.M.
        The experience of nurses working with trauma patients in critical care and emergency settings: a qualitative study from Scottish nurses' perspective.
        Int J Orthop Trauma Nurs. 2014; 18: 13-22
        • Sultan P.
        • Arulkumaran N.
        • Rhodes A.
        Provision of critical care services for the obstetric population.
        Best Pract Res Clin Obstet Gynaecol. 2013; 27: 803-809
        • Jeddian A.
        • Lindenmeyer A.
        • Marshall T.
        • Howard A.F.
        • Sayadi L.
        • Rashidian A.
        • et al.
        Implementation of a critical care outreach service: a qualitative study.
        Int Nurs Rev. 2017; 64: 353-362
        • Jeddian A.
        • Hemming K.
        • Lindenmeyer A.
        • Rashidian A.
        • Sayadi L.
        • Jafari N.
        • et al.
        Evaluation of a critical care outreach service in a middle-income country: a stepped wedge cluster randomized trial and nested qualitative study.
        J Crit Care. 2016; 36: 212-217
        • Smith D.
        • Cartwright M.
        • Dyson J.
        • Hartin J.
        • Aitken L.M.
        Patterns of behaviour in nursing staff actioning the afferent limb of the rapid response system (RRS): a focused ethnography.
        J Adv Nurs. 2020; 76: 3548-3562
        • Chua W.L.
        • Legido-Quigley H.
        • Jones D.
        • Hassan N.B.
        • Tee A.
        • Liaw S.Y.
        A call for better doctor–nurse collaboration: a qualitative study of the experiences of junior doctors and nurses in escalating care for deteriorating ward patients.
        Aust Crit Care. 2020; 33: 54-61
        • Marshall A.P.
        • West S.H.
        • Aitken L.M.
        Preferred information sources for clinical decision making: critical care nurses' perceptions of information accessibility and usefulness.
        Worldviews Evidence-Based Nurs. 2011; 8: 224-235
        • Smith D.
        • Cartwright M.
        • Dyson J.
        • Hartin J.
        • Aitken L.M.
        Barriers and enablers of recognition and response to deteriorating patients in the acute hospital setting: a theory-driven interview study using the Theoretical Domains Framework.
        J Adv Nurs. 2021; 77: 2813-2844https://doi.org/10.1111/jan.14830
        • Rihari-Thomas J.
        • Digiacomo M.
        • Phillips J.
        • Newton P.
        • Davidson P.M.
        Clinician perspectives of barriers to effective implementation of a rapid response system in an academic health centre: a focus group study.
        Int J Health Pol Manag. 2017; 6: 447-456
        • McGaughey J.
        • O'Halloran P.
        • Porter S.
        • Blackwood B.
        Early warning systems and rapid response to the deteriorating patient in hospital: a systematic realist review.
        J Adv Nurs. 2017; 73: 2877-2891
        • Bannard-Smith J.
        • Lighthall G.K.
        • Subbe C.P.
        • Durham L.
        • Welch J.
        • Bellomo R.
        • et al.
        Clinical outcomes of patients seen by rapid response teams: a template for benchmarking international teams.
        Resuscitation. 2016; 107: 7-12
        • Allen J.
        • Jones D.
        • Currey J.
        Clinician and manager perceptions of factors leading to ward patient clinical deterioration.
        Aust Crit Care. 2018; 31: 369-375
        • Williams G.
        • Hughes V.
        • Timms J.
        • Raftery C.
        Emergency nurse as hospital clinical team coordinator - shining a light into the night.
        Australas Emerg Nurs J. 2012; 15: 245-251
        • Jones D.
        • Warrillow S.
        Clinical deterioration in cancer patients-the role of the rapid response team.
        Crit Care Med. 2014; 42: 997-998
        • Subbe C.P.
        • Bannard-Smith J.
        • Bunch J.
        • Champunot R.
        • DeVita M.A.
        • Durham L.
        • et al.
        Quality metrics for the evaluation of rapid response systems: proceedings from the third international consensus conference on rapid response systems.
        Resuscitation. 2019; 141: 1-12
        • Chellel A.
        • Fraser J.
        • Fender V.
        • Higgs D.
        • Buras-Rees S.
        • Hook L.
        • et al.
        Nursing observations on ward patients at risk of critical illness.
        Nurs Times. 2002; 98: 36-39
        • Winterbottom F.A.
        • Webre H.
        Rapid response system restructure: focus on prevention and early Intervention.
        Crit Care Nurs Q. 2021; 44: 424-430
        • Son Y.J.
        • Kim G.O.
        • Lee Y.M.
        • Oh M
        • Choi J
        Predictors of Early and Late Unplanned Intensive Care Unit Readmission: A Retrospective Cohort Study.
        J Nurs Scholarsh. 2021; 53: 400-407