Advertisement

Psychometric properties of the Nursing Intensive Care Satisfaction Scale: A multicentre cross-sectional study

  • Marta Romero-García
    Affiliations
    Department of Fundamental and Medical-Surgical Nursing, School of Nursing. University of Barcelona, L'Hospitalet de Llobregat, Barcelona, Spain

    GRIN-IDIBELL, Institute of Biomedical Research, L'Hospitalet de Llobregat, Barcelona, Spain

    International Research Project for the Humanization of Health Care, Proyecto HU-CI. Collado Villalba, Madrid, Spain
    Search for articles by this author
  • Isidro Alcalà-Jimenez
    Affiliations
    Hospital Vall D'Hebrón, Barcelona, Spain
    Search for articles by this author
  • María Antonia Martínez-Momblan
    Affiliations
    Department of Fundamental and Medical-Surgical Nursing, School of Nursing. University of Barcelona, L'Hospitalet de Llobregat, Barcelona, Spain

    Biomedical Research Networking Centre of Rare Diseases (CIBER-ER), Unit 747 ISCIII, Madrid, Spain
    Search for articles by this author
  • Laura de la Cueva-Ariza
    Affiliations
    Department of Fundamental and Medical-Surgical Nursing, School of Nursing. University of Barcelona, L'Hospitalet de Llobregat, Barcelona, Spain

    GRIN-IDIBELL, Institute of Biomedical Research, L'Hospitalet de Llobregat, Barcelona, Spain

    International Research Project for the Humanization of Health Care, Proyecto HU-CI. Collado Villalba, Madrid, Spain
    Search for articles by this author
  • Cecilia Cuzco
    Affiliations
    Department of Fundamental and Medical-Surgical Nursing, School of Nursing. University of Barcelona, L'Hospitalet de Llobregat, Barcelona, Spain
    Search for articles by this author
  • Sergio Alonso
    Affiliations
    Department of Fundamental and Medical-Surgical Nursing, School of Nursing. University of Barcelona, L'Hospitalet de Llobregat, Barcelona, Spain

    GRIN-IDIBELL, Institute of Biomedical Research, L'Hospitalet de Llobregat, Barcelona, Spain
    Search for articles by this author
  • Llúcia Benito-Aracil
    Correspondence
    Corresponding author at: Pavelló de Govern, 3° pl. 08907 L'Hospitalet de Llobregat, Spain. Tel.: +34 934 024 225.
    Affiliations
    Department of Fundamental and Medical-Surgical Nursing, School of Nursing. University of Barcelona, L'Hospitalet de Llobregat, Barcelona, Spain

    GRIN-IDIBELL, Institute of Biomedical Research, L'Hospitalet de Llobregat, Barcelona, Spain
    Search for articles by this author
  • Pilar Delgado-Hito
    Affiliations
    Department of Fundamental and Medical-Surgical Nursing, School of Nursing. University of Barcelona, L'Hospitalet de Llobregat, Barcelona, Spain

    GRIN-IDIBELL, Institute of Biomedical Research, L'Hospitalet de Llobregat, Barcelona, Spain

    International Research Project for the Humanization of Health Care, Proyecto HU-CI. Collado Villalba, Madrid, Spain
    Search for articles by this author
Open AccessPublished:April 27, 2022DOI:https://doi.org/10.1016/j.aucc.2022.03.012

      Abstract

      Background

      Patient satisfaction with nursing care is an indicator of patient satisfaction with the hospital stay in general. The Nursing Intensive Care Satisfaction Scale is the only scale about patient satisfaction with nursing care received in an intensive care unit that incorporates the critically ill patient's perspective into its design and validation. We validated the scale nationally, incorporating intensive care units at public and private hospitals of different levels of complexity in Spain.

      Objectives

      The objective of this study was to validate in Spanish intensive care units the Nursing Intensive Care Satisfaction Scale, a patient-centred questionnaire that evaluates recently discharged intensive care patients' satisfaction with the nursing care they received.

      Design

      We used a psychometric quantitative methodology and a descriptive cross-sectional design.

      Setting and participants

      The study was conducted in intensive care units at level II and III public and private hospitals throughout Spain. The study population was all patients discharged from intensive care units from December 2018 to December 2019 from the 19 participating hospitals. We used consecutive sampling until reaching a sample size of 677 patients. The assessment instruments were given to patients at discharge and 48 h later to measure temporal stability.

      Methods

      The validation process included the analysis of internal consistency (Cronbach's α coefficient), temporal stability (test-retest), construct validity through a confirmatory factor analysis, and criterion validity using the Pearson correlation coefficient and three criterion items that assessed similar constructs.

      Results

      The reliability of the scale was 0.97, and the factors obtained values between 0.87 and 0.96. The intraclass correlation coefficient for the total scale was 0.83, indicating good temporal stability. Construct validity showed a good fit and a four-factor structure, in accordance with the theoretical model. Criterion validity presented a correlation that was between moderate and high (range: 0.46 to 0.57).

      Conclusions

      The Nursing Intensive Care Satisfaction Scale has good psychometric properties, demonstrating its ability to accurately measure patient satisfaction across a range of contexts in Spain. Continuous monitoring of satisfaction will allow nurses to identify areas for improvement that can increase the quality of care.

      Keywords

      1. Introduction

      Patients are no longer seen—or see themselves—as mere objects of health care. Rather, they are understood as active subjects of their own health. Therefore, we must know patients’ opinions of the care they receive, including their views of the context, processes, and results so that services can be adjusted to their needs, expectations, and priorities.
      • Romero-García M.
      • Trujols-Albet J.
      Towards increased incorporation of the patient perspective in the design of evaluation tools for the evaluation of efficiency and quality of patient care.
      ,
      • Romero-García M.
      • de la Cueva-Ariza L.
      • Benito-Aracil L.
      • Lluch-Canut T.
      • Trujols-Albet J.
      • Martínez-Momblan M.A.
      • et al.
      Nursing intensive-care satisfaction scale [NICSS]: development and validation of a patient-centred instrument.
      Patients who have received individualised care participate more in their care, which favours compliance with treatment and attendance at follow-up visits.
      • Konerding U.
      • Bowen T.
      • Elkhuizen S.G.
      • Faubel R.
      • Forte P.
      • Karampli E.
      • et al.
      Development of a universal short patient satisfaction questionnaire on the basis of SERVQUAL: psychometric analyses with data of diabetes and stroke patients from six different European countries.
      ,
      • Ng J.H.Y.
      • Luk B.H.K.
      Patient satisfaction: concept analysis in the healthcare context.
      In turn, patients who attend their follow-up visits have higher satisfaction.
      • Iftikhar A.
      • Allah N.
      • Shadiullah K.
      • Habibullah K.
      • Muhammad A.,R.
      • Muhammad H.,K.
      Predictors of patient satisfaction.
      The opposite occurs when patients perceive a lack of nursing care, which is associated with nursing understaffing and poor hospital work environments. In such situations, patient satisfaction is low.
      • Aiken L.H.
      • Sloane D.M.
      • Ball J.
      • Bruyneel L.
      • Rafferty A.M.
      • Griffiths P.
      Patient satisfaction with hospital care and nurses in England: an observational study.
      The concept of satisfaction in relation to health care has been subject to a paradigm shift, leaving behind a model in which professionals assumed that they knew what satisfied patients, regardless of whether these assumptions coincided with patients' priorities. The paradigm shift necessitated incorporating the patient's perspective into service delivery to improve patient satisfaction.
      • Ng J.H.Y.
      • Luk B.H.K.
      Patient satisfaction: concept analysis in the healthcare context.
      ,
      • Bull C.
      • Byrnes J.
      • Hettiarachchi R.
      • Downes M.
      A systematic review of the validity and reliability of patient-reported experience measures.
      ,
      • Romero M.
      • de la Cueva L.
      • Jover C.
      • Delgado P.
      • Acosta B.
      • Sola M.
      • et al.
      Perception of the critical patient on nursing cares: an approach to the concept of satisfaction.
      At the same time, satisfaction is subjective and contextual, and several authors have noted that it varies across social groups. Therefore, satisfaction can only be defined in relation to the social context and to the patient's previous experiences.
      • Ng J.H.Y.
      • Luk B.H.K.
      Patient satisfaction: concept analysis in the healthcare context.
      ,
      • Batbaatar E.
      • Dorjdagva J.
      • Luvsannyam A.
      • Amenta P.
      Conceptualisation of patient satisfaction: a systematic narrative literature review.
      • Goh M.L.
      • Ang E.N.K.
      • Chan Y.H.
      • He H.G.
      • Vehviläinen-Julkunen K.
      A descriptive quantitative study on multi-ethnic patient satisfaction with nursing care measured by the Revised Humane Caring Scale.
      • Otani K.
      • Waterman B.
      • Dunagan W.C.
      Patient satisfaction: how patient health conditions influence their satisfaction.
      Importantly, a patient's satisfaction with nursing care is the best predictor of his or her overall satisfaction with the hospital stay.
      • Otani K.
      • Waterman B.
      • Faulkner K.M.
      • Boslaugh S.
      • Burroughs T.E.
      • Dunagan W.C.
      Patient satisfaction: focusing on "excellent.

      2. Background

      Patient satisfaction is defined as the degree of convergence between the patients' expectations of ideal nursing care and the patients' perception of the care they actually receive.
      • Iftikhar A.
      • Allah N.
      • Shadiullah K.
      • Habibullah K.
      • Muhammad A.,R.
      • Muhammad H.,K.
      Predictors of patient satisfaction.
      ,
      • Donabedian A.
      The quality of medical care: how can it be assessed?.
      Nursing care is directed towards and centred on the patient, making patient satisfaction an important measure of the quality of care.
      • Goh M.L.
      • Ang E.N.K.
      • Chan Y.H.
      • He H.G.
      • Vehviläinen-Julkunen K.
      A descriptive quantitative study on multi-ethnic patient satisfaction with nursing care measured by the Revised Humane Caring Scale.
      ,
      • Laschinger H.S.
      • Hall L.M.
      • Pedersen C.
      • Almost J.
      A psychometric analysis of the patient satisfaction with nursing care quality questionnaire: an actionable approach to measuring patient satisfaction.
      ,
      • Reck D.L.
      Can and should nurses be aware of patients' expectations for their nursing care?.
      Intensive care patients consider nursing care to be satisfactory when it is characterised by an outlook that is both humanistic and scientific, is offered continuously, and is aimed at providing them with security, well-being, and confidence.
      • Romero M.
      • de la Cueva L.
      • Jover C.
      • Delgado P.
      • Acosta B.
      • Sola M.
      • et al.
      Perception of the critical patient on nursing cares: an approach to the concept of satisfaction.
      Numerous authors have designed patient satisfaction questionnaires that have been recognised and validated for evaluating various types of care in a range of settings.
      • Romero-García M.
      • de la Cueva-Ariza L.
      • Benito-Aracil L.
      • Lluch-Canut T.
      • Trujols-Albet J.
      • Martínez-Momblan M.A.
      • et al.
      Nursing intensive-care satisfaction scale [NICSS]: development and validation of a patient-centred instrument.
      ,
      • Laschinger H.S.
      • Hall L.M.
      • Pedersen C.
      • Almost J.
      A psychometric analysis of the patient satisfaction with nursing care quality questionnaire: an actionable approach to measuring patient satisfaction.
      ,
      • Babakus E.
      • Mangold W.G.
      Adapting the SERVQUAL scale to hospital services: an empirical investigation.
      • Davis B.
      • Bush H.
      Developing effective measurement tools: a case study of the consumer emergency care satisfaction scale.
      • La-Monica E.L.
      • Oberst M.T.
      • Madea A.R.
      • Wolf R.M.
      Development of a patient satisfaction scale.
      • Larson P.J.
      • Ferketich S.L.
      Patients' satisfaction with nurses' caring during hospitalization.
      • Lynn M.R.
      • McMillen B.J.
      • Sidani S.
      Understanding and measuring patients' assessment of the quality of nursing care.
      • Thomas L.H.
      • McColl E.
      • Priest J.
      • Bond S.
      • Boys R.J.
      Newcastle satisfaction with nursing scales: an instrument for quality assessments of nursing care.
      • Risser N.L.
      Development of an instrument to measure patient satisfaction with nurses and nursing care in primary care settings.
      Patient satisfaction assessment questionnaires can be classified into four categories:
      • Trujols J.
      • Iraurgi I.
      • Oviedo-Joekes E.
      • Guàrdia-Olmos J.
      A critical analysis of user satisfaction surveys in addiction services: opioid maintenance treatment as a representative case study.
      patient-generated (developed entirely from the patient's perspective), patient-centred (developed in a way that incorporates the patient's perspective), valued by the patient (developed without the patients' participation but valued by patients as relevant), and irrelevant to the patient (developed without the patients' participation and considered by patients to be irrelevant to them). The first two categories incorporate the patient's perspective by using mixed methods. The qualitative component gives patients a voice and makes it possible to understand their perspective.
      • de la Cueva-Ariza L.
      • Romero-García M.,M.
      • Delgado-Hito P.
      • Acosta-Mejuto B.
      • Jover-Sancho C.
      • Ricart-Basagaña M.T.
      • et al.
      Development of an instrument to measure the degree of critical patient's satisfaction with nursing care: research protocol.
      ,
      • Lasch K.E.
      • Marquis P.
      • Vigneux M.
      • Abetz L.
      • Arnould B.
      • Bayliss M.
      • et al.
      PRO development rigorous qualitative research as the crucial foundation.
      The quantitative component makes it possible to study the psychometric properties of the instrument and measure the results.
      Delivering truly patient-centred care requires an equally patient-centred evaluation of that care. Such an evaluation will only be possible if, among other requirements such as psychometric robustness, the patients' perspective is incorporated.
      • Romero-García M.
      • de la Cueva-Ariza L.
      • Benito-Aracil L.
      • Lluch-Canut T.
      • Trujols-Albet J.
      • Martínez-Momblan M.A.
      • et al.
      Nursing intensive-care satisfaction scale [NICSS]: development and validation of a patient-centred instrument.
      ,
      • Bull C.
      • Byrnes J.
      • Hettiarachchi R.
      • Downes M.
      A systematic review of the validity and reliability of patient-reported experience measures.
      In this sense, only the Nursing Intensive Care Satisfaction Scale (NICSS)
      • Romero-García M.
      • de la Cueva-Ariza L.
      • Benito-Aracil L.
      • Lluch-Canut T.
      • Trujols-Albet J.
      • Martínez-Momblan M.A.
      • et al.
      Nursing intensive-care satisfaction scale [NICSS]: development and validation of a patient-centred instrument.
      has been specifically designed and validated to evaluate the satisfaction of critically ill patients from their perspective (Trujol's category 2). In an earlier phase, we validated the scale in Spanish with a sample of 200 patients, and the results of the confirmatory factor analysis showed correlations between some factors (F1, F2, and F3), with values slightly higher than 1. This relatively weak outcome may be the result of the sample size, as pointed out by Uriel & Aldás-Manzano.
      • Uriel E.
      • Aldás-Manzano J.
      Análisis multivariante aplicado: aplicaciones al marketing, investigación de mercados, economía, dirección de empresas y turismo (Applied multivariate analysis: applications in marketing, market research, economy, business management and tourism).
      The study also revealed that item sixdid not present statistically significant differences and item 11 did not present sufficient factorial load (r = 0.147). Understanding the validation of an instrument as a continuous and dynamic process, we decided to perform a study with a larger sample to evaluate more fully the factorial structure of the NICSS satisfaction model. Specifically, we undertook a study to validate the NICSS in intensive care units (ICUs) across Spain to test its fit with the factorial structure of the Multifactorial Model of Nursing Intensive Care Satisfaction (MM-NICS).
      • Romero M.
      • de la Cueva L.
      • Jover C.
      • Delgado P.
      • Acosta B.
      • Sola M.
      • et al.
      Perception of the critical patient on nursing cares: an approach to the concept of satisfaction.

      3. Methods

      3.1 Study design and participants

      We used a psychometric quantitative method and a descriptive cross-sectional design. The study was carried out in the adult ICUs of level II and III public and private hospitals throughout Spain. The study population was all patients discharged from ICUs from December 2018 to December 2019 from the 19 participating hospitals.
      We calculated the sample size by considering the number of items that make up the NICSS. Because the scale had more than 20 items, we needed 5–10 participants per item.
      • Streiner D.L.
      Starting at the beginning: an introduction to coefficient alpha and internal consistency.
      We also predicted a 38% dropout rate. Therefore, the necessary sample size was 677. Sampling was nonprobabilistic and proceeded consecutively until we had achieved the necessary number of patients meeting the following inclusion criteria: (i) oriented to time, place, and person and (ii) able to read and write. The exclusion criterion was being discharged to another hospital or directly to the patient's home. We considered the number of annual admissions at each ICU to obtain the necessary sample size.

      3.2 Data collection and procedure

      The following instruments were used.
      The NICSS
      • Romero-García M.
      • de la Cueva-Ariza L.
      • Benito-Aracil L.
      • Lluch-Canut T.
      • Trujols-Albet J.
      • Martínez-Momblan M.A.
      • et al.
      Nursing intensive-care satisfaction scale [NICSS]: development and validation of a patient-centred instrument.
      has a total of 49 items distributed in four factors. The first three are related to the experiences of the critically ill patient in relation to the nursing care received—Factor 1: Holistic Care (20 items), Factor 2: Communication Modes (six items), and Factor 3: Professional Behaviors (11 items). The final factor covers patients’ experiences of and feelings about the nursing care received: Factor 4: Consequences (12 items). Each of the items is measured on a Likert-type scale, with six answer options (1 = “strongly disagree” to 6 = “strongly agree”). The score for each factor is obtained by adding the points of the items as follows—Factor 1: Holistic Care (minimum of 20 and maximum of 120), Factor 2: Communication Modes (minimum of 6 and maximum of 36), Factor 3: Professional Behaviors (minimum of 11 and maximum of 66), and Factor 4: Consequences (minimum of 12 and maximum of 66). In Factor four, there are three items (44, 48, and 49) formulated positively but expressing an unfavourable opinion; therefore, the scoring is reversed. Three criterion items evaluate similar constructs with respect to general satisfaction (item 50), the intent to return to the same ICU, if necessary (item 51), and expectations met (item 52).
      An NICSS total score between 49 and 130 is interpreted as meaning that the patient is not satisfied, from 131 to 212 that he or she is quite satisfied, and from 213 to 294 that he or she is very satisfied with the nursing care received in the ICU. The reliability of the total NICSS is 0.95, and that of the factors is between 0.7 and 0.91. The NICSS presents good temporal stability with an intraclass correlation coefficient for the total scale of 0.83. Construct validity shows acceptable fit and a four-factor structure. Criterion validity presented a correlation between low and high (range: 0.42–0.68).
      • Romero-García M.
      • de la Cueva-Ariza L.
      • Benito-Aracil L.
      • Lluch-Canut T.
      • Trujols-Albet J.
      • Martínez-Momblan M.A.
      • et al.
      Nursing intensive-care satisfaction scale [NICSS]: development and validation of a patient-centred instrument.
      We used an ad hoc form to collect sociodemographic and clinical information about the patients and organisational information about the participating hospitals. The sociodemographic variables were age, sex, marital status, employment status, and level of education. The clinical variables were the days of stay in the ICU, diagnosis at admission, previous admissions to the ICU, perceived health state (Likert-type scale with 10 response options, ranging from 1 “terrible” to 10 “excellent”), and perception of the degree of recovery (Likert-type scale with 10 response options, ranging from 1 “I haven't improved at all” to 10 “I've fully recovered”). The organisational variables were hospital type, ICU cubicle type, and nurse-to-patient ratio.
      Once the patient was discharged from the ICU to general care and within a period not exceeding 48 h, the collaborating investigator of each of the participating ICUs contacted the patient to offer him or her detailed information about the study and to seek informed consent. If the patient agreed to participate, he or she was given the inclusion and exclusion criteria, the NICSS scale, and the sociodemographic data sheet to fill out on his or her own. The collaborating researcher collected the completed questionnaires within 24 h. The collaborating researcher completed the organisational data form and the clinical data form from the patient's medical history. Forty-eight hours after the patient filled out the study questionnaires, he or she was contacted to fill out the NICSS again so that we could analyse the temporal stability of the responses (Fig. 1).
      Figure 1
      Fig. 1Procedure for data collection. ICU, intensive care unit; NICSS, Nursing Intensive Care Satisfaction Scale.
      We removed names and assigned a code to the completed questionnaires to guarantee confidentiality. The information collected was entered into Microsoft Access 2007.

      3.3 Statistical analysis

      We evaluated whether any of the items had a nonresponse rate ≥20%, a level of variability <10% (variance <0.225),
      • Streiner D.
      • Norman G.
      • Carney J.
      Health measurement scales a practical guide to their development and use.
      a response rate of >90% in any of the valid response categories, and an item–total correlation <0.3. In addition, following the recommendation of Ferketich,
      • Ferketich S.
      Focus on psychometrics aspects of item analysis.
      we evaluated whether any items of the questionnaire presented either of the following descriptions: (i) the mean correlation between the item and the other items was not between 0.30 and 0.70 or (ii) less than 25% of the correlations were between 0.30 and 0.70. None of the items showed inadequate values, and therefore, none were eliminated. The factorial structure was considered statistically valid to explain the responses obtained through the NICSS since this structure coincided with the MM-NICS.
      • Anderson J.
      • Gerbing D.
      Structural equation modeling in practice: a review and recommended two-step approach.
      ,
      • Boomsma A.
      Reporting analyses of covariance structures.
      To evaluate the goodness of fit of the model, we used the following indicators.
      • Hu L.
      • Bentler P.M.
      Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives.
      To quantify absolute fit, we used (i) goodness of fit index, with values >0.90 suggesting an acceptable fit and values >0.95 an excellent fit
      • Schermelleh-Engel K.
      • Moosbrugger H.
      • y Müller H.
      Evaluating the fit of structural equation models: tests of significance and descriptive goodness-of-fit measures.
      and (ii) root mean standard error, with appropriate values between 0.05 and 0.08.
      • Hair J.
      • Anderson R.
      • Tatham R.
      • Black W.
      Análisis multivariante (multivariate data analysis).
      To quantify incremental fit, we used (iii) normed fit index with recommended values >0.95.
      • Bentler P.M.
      • Bonett D.G.
      Significance tests and goodness of fit in the analysis of covariance structures.
      Criterion validity was analysed using the Spearman correlation coefficient (r > 0.5), correlating the NICSS scores with the three criterion items: general satisfaction (item 50), the intent to return to the same ICU, if necessary (item 51), and expectations met (item 52).
      Reliability of the instrument was determined by studying its internal consistency using Cronbach's α, in which values >0.7 indicate good consistency,
      • Bland J.M.
      • Altman D.G.
      ,
      • Nunnally J.C.
      • Bernstein I.H.
      Psychometric theory.
      and the test-retest was evaluated using the intraclass correlation coefficient, in which values >0.7 show acceptable consistency and >0.8 good consistency.
      • Cicchetti D.V.
      Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in Psychology.
      Data were analysed with R, version 3.1.2, for Windows and EQS, version 6.1, for confirmatory factor analysis. Statistical significance was established at a probability of p < 0.05.

      3.4 Ethical considerations

      The study was conducted in accordance with the principles of the Declaration of Helsinki and approved by the Clinical Research Ethics Committee of the hospital (Code nº2018/7818/l). Confidentiality was protected by assigning participants a code, which was used to identify them instead of their names. Subjects demonstrated their voluntary agreement to participate in the study by signing the informed consent form. Prior to completing the questionnaire, all participants gave informed consent, confirming that they understood that participation was voluntary and anonymous and that they could withdraw at any time, and that their data would be completely confidential, stored and analysed on a secure computer, and used only for the study.

      4. Results

      4.1 Sample characteristics

      The participating institutions were 19 hospitals in Spain: 15 public hospitals (5 level II and 10 level III) and four private hospitals (three level II and one level III). The nurse-to-patient ratio was 1:2, except in 10.5% (n = 2) of the hospitals, where it was 1:1.
      The final sample was comprised of 677 participants, with a mean age of 59.7 (standard deviation: 15.7 years). Four hundred twenty-six participants (62.9%) were male. Table 1 shows the sociodemographic, clinical, and organisational variables.
      Table 1Sociodemographic, clinical, and organisational characteristics of the sample (n = 677).
      n (%)
      AgeMedium (SD)59.7 (15.7)
      SexFemale251 (37.1%)
      Male426 (62.9%)
      Employment statusUnemployed51 (7.53%)
      Employed206 (30.4%)
      In early retirement36 (5.32%)
      Retired304 (44.9%)
      Student12 (1.77%)
      Other situations68 (10.0%)
      Marital statusMarried404 (59.7%)
      Lives with a romantic partner49 (7.24%)
      Single101 (14.9%)
      Separated22 (3.25%)
      Divorced40 (5.91%)
      Widowed61 (9.01%)
      EducationDid not finish secondary school91 (13.4%)
      Finished secondary school586 (86.6%)
      Perception of health statusMedian [Q1; Q3]7.00 [5.00; 8.00]
      Perception of degree of recoveryMedian [Q1; Q3]8.00 [7.00; 9.00]
      Previous ICU admissionsYes217 (32.1%)
      No460 (67.9%)
      Length of ICU stay (days)Median [Q1; Q3]4.00 [2.00; 7.00]
      Diagnosis at admissionNeurological47 (6.94%)
      Respiratory151 (22.3%)
      Traumatological45 (6.65%)
      Hepatic25 (3.69%)
      Immunological4 (0.59%)
      Coronary140 (20.7%)
      Cardiac103 (15.2%)
      Renal25 (3.69%)
      Digestive61 (9.01%)
      Urinary18 (2.66%)
      Other58 (8.57%)
      Type of ICUPolyvalent564 (83.3%)
      Coronary60 (8.86%)
      Hepatic11 (1.62%)
      Surgical2 (0.30%)
      Respiratory13 (1.92%)
      Trauma14 (2.07%)
      Cardiac13 (1.92%)
      Type of patient cubicleOpen153 (22.6%)
      Individual closed487 (71.9%)
      Double closed37 (5.47%)
      ICU: intensive care unit; SD: standard deviation; Median [Q1; Q3].

      4.2 Validity

      None of the items showed a response rate greater than 90% in any of the response categories or a variability of less than 10% (variance <0.225). In addition, the different factors presented adequate homogeneity (correlation <0.3). Under these criteria, all items of the questionnaire were maintained.
      The factorial structure was analysed using the least squares method. All items had a factorial load >0.3. Likewise, all items revealed statistically significant saturation, and the factors were optimally correlated with the rest of the factors (extremes: 0.84–0.99) (Fig. 2).
      Figure 2
      Fig. 2Standardised parameters of the model.
      The absolute and incremental fit indices presented a good fit to the model with a goodness of fit index of 1, a root mean standard error of 0, and a normed fit index of 0.99. Regarding criterion validity (Table 2), the four factors of the NICSS presented a moderate to high correlation with the proposed criterion items (range: 0.46–0.57).
      Table 2Spearman multitrait-multimethod correlations matrix.
      Item 50: Overall satisfactionItem 51: Intent to returnItem 52: Expectations met
      rp-valuerp-valuerp-value
      Factor 1: Holistic care0.55<0.001
      p < 0.05.
      0.52<0.001
      p < 0.05.
      0.52<0.001
      p < 0.05.
      Factor 2: Communication Modes0.55<0.001
      p < 0.05.
      0.46<0.001
      p < 0.05.
      0.5<0.001
      p < 0.05.
      Factor 3: Professional Behaviors0.57<0.001
      p < 0.05.
      0.52<0.001
      p < 0.05.
      0.52<0.001
      p < 0.05.
      Factor 4: Consequences0.54<0.001
      p < 0.05.
      0.46<0.001
      p < 0.05.
      0.48<0.001
      p < 0.05.
      NICSS Total0.56<0.001
      p < 0.05.
      0.51<0.001
      p < 0.05.
      0.51<0.001
      p < 0.05.
      NICSS: Nursing Intensive Care Satisfaction Scale; r: rho, Spearman's rank correlation coefficient.
      a p < 0.05.

      4.3 Reliability

      Cronbach's α internal consistency coefficient was 0.97 for the total NICSS scale and >0.7 for all factors (range: 0.87–0.96) and >0.7 for all items (range: 0.72–0.93), except for item 42, which had a Cronbach's α of 0.64. A test-retest analysis was performed with 80.5% (n = 545) of the sample, with an interval of >48 h between administrations. One hundred thirty-two patients (19.5%) did not complete the NICSS again, mainly because they had been discharged from the hospital in 48.5% (n = 64) or because they did not want to continue participating in the study in 12.9% (n = 17). The total intraclass correlation coefficient was 0.83, indicating good consistency for the entire scale. The difference in mean score on the test and the retest was statistically significant for Factor 4: Consequences. For factors 1 and 3, the intraclass correlation coefficient was >0.8 (range: 0.73–0.83), indicating good consistency. For factors 2 and 4, the intraclass correlation coefficient was >0.7 and therefore acceptable (Table 3).
      Table 3Test-retest analysis of the factors of the NICSS.
      ItemsNTestRetestp-valueICC
      MeanSDMeanSD
      F 1: Holistic Care20545113.58.9113.79.10.570.83
      F 2: Communication Modes654534.12.834.22.80.380.79
      F 3: Professional Behaviors1154562.84.562.94.70.670.82
      F 4: Consequences12545675.966.56.40.02∗0.73
      OVERALL NICSS SCALE49545275.820.6276.521.20.170.83
      p-value: Wilcoxon signed rank-sum test for paired data.
      ICC: intraclass correlation coefficient; NICSS: Nursing Intensive Care Satisfaction Scale; F: Factor.

      5. Discussion

      The objective of the NICSS is to evaluate the level of patient satisfaction with nursing intensive care using a self-completed instrument whose design and validation have incorporated the perspective of the critically ill patient.
      • Romero-García M.
      • de la Cueva-Ariza L.
      • Benito-Aracil L.
      • Lluch-Canut T.
      • Trujols-Albet J.
      • Martínez-Momblan M.A.
      • et al.
      Nursing intensive-care satisfaction scale [NICSS]: development and validation of a patient-centred instrument.
      ,
      • Romero M.
      • de la Cueva L.
      • Jover C.
      • Delgado P.
      • Acosta B.
      • Sola M.
      • et al.
      Perception of the critical patient on nursing cares: an approach to the concept of satisfaction.
      Until now, most instruments for measuring patient satisfaction have focused on the physical and structural aspects of hospitals that are of interest to health professionals and researchers. They have not incorporated the perspective of patients themselves.
      • Trujols J.
      • Portella M.J.
      • Iraurgi I.
      • Campins M.J.
      • Siñol N.
      • de los Cobos J.P.
      Patient-reported outcome measures: are they patient-generated, patient-centred or patient-valued?.
      If patients’ perspectives are not considered, the validity of the resulting data will be questionable.
      • Trujols J.
      • Iraurgi I.
      • Oviedo-Joekes E.
      • Guàrdia-Olmos J.
      A critical analysis of user satisfaction surveys in addiction services: opioid maintenance treatment as a representative case study.

      5.1 Psychometric characteristics

      The score for internal consistency using Cronbach's α for the overall NICSS scale was 0.97 and > 0.7 for all factors and for all items, meaning that the instrument showed good internal consistency. Likewise, the coefficient of homogeneity of the items for each of the factors showed values >0.3, and all items revealed statistically significant saturation.
      If we compare the reliability results obtained from the NICSS with those of other scales that measure patient satisfaction and that have solid psychometric and clinical recognition, such as the Risser Patient Satisfaction Scale,
      • Risser N.L.
      Development of an instrument to measure patient satisfaction with nurses and nursing care in primary care settings.
      ,
      • Charalambous A.
      • Adamakidou T.
      Risser patient satisfaction scale: a validation study in Greek cancer patients.
      the Patient Satisfaction Index,
      • Hinshaw A.
      • Atwood J.
      A patient satisfaction instrument: precision by replication.
      the Consumer Emergency Care Satisfaction Scale,
      • Davis B.
      • Bush H.
      Developing effective measurement tools: a case study of the consumer emergency care satisfaction scale.
      the Caring Assessment Report Evaluation Q-sort,
      • Larson P.J.
      Important nurse caring behaviors perceived by patients with cancer.
      the Patient Satisfaction with Nursing Care Quality Questionnaire,
      • Laschinger H.S.
      • Hall L.M.
      • Pedersen C.
      • Almost J.
      A psychometric analysis of the patient satisfaction with nursing care quality questionnaire: an actionable approach to measuring patient satisfaction.
      and the Newcastle Satisfaction with Nursing Scale,
      • McColl E.
      • Thomas L.
      • Bond S.
      A study to determine patient satisfaction with nursing care.
      we see that the NICCS’ results are similar and in some cases better.
      The intraclass correlation coefficient showed a value of 0.83, indicating good temporal stability for the total scale, good consistency for Factors 1and 3, and acceptable consistency for Factors 2 and 4. These results differ from those of our first attempt to validate the scale, in which Factor 2: Communication Modes obtained an inacceptable degree of consistency, leading us to propose the current research.
      • Romero-García M.
      • de la Cueva-Ariza L.
      • Benito-Aracil L.
      • Lluch-Canut T.
      • Trujols-Albet J.
      • Martínez-Momblan M.A.
      • et al.
      Nursing intensive-care satisfaction scale [NICSS]: development and validation of a patient-centred instrument.
      Of the instruments reviewed, not all studies report temporal stability. Of those that do, the most recent studies show similar values to those of the NICSS, such as the Chinese version of the Caring Assessment Report Evaluation Q-sort,
      • Chen S.-Y.
      • Yen W.-J.
      • Lin Y.-J.
      • Lee C.-H.
      • Lu Y.-C.
      A Chinese version of the Caring Assessment Report Evaluation Q-sort scale for measuring patients ‘perception on nurses' caring behaviours: reliability and validity assessment.
      the Patient Satisfaction with Nursing Care Quality Questionnaire,
      • Laschinger H.S.
      • Hall L.M.
      • Pedersen C.
      • Almost J.
      A psychometric analysis of the patient satisfaction with nursing care quality questionnaire: an actionable approach to measuring patient satisfaction.
      and the Risser Scale.
      • Charalambous A.
      • Adamakidou T.
      Risser patient satisfaction scale: a validation study in Greek cancer patients.
      Confirmatory factor analysis allowed us to identify four factors coinciding with the MM-NICS, which describes the construct of satisfaction with nursing care on the basis of four dimensions.
      • Romero M.
      • de la Cueva L.
      • Jover C.
      • Delgado P.
      • Acosta B.
      • Sola M.
      • et al.
      Perception of the critical patient on nursing cares: an approach to the concept of satisfaction.
      Each of the factors presented a high degree of correlation with the other factors, indicating a good fit with the theoretical model, in which the satisfaction of critically ill patients depends on care that is based on a combination of humanistic and scientific viewpoints, is provided continuously, and supports patients’ feeling of safety, well-being, and trust. The improvement in the correlations between the factors in the confirmatory factor analysis with respect to the previous validation study
      • Romero-García M.
      • de la Cueva-Ariza L.
      • Benito-Aracil L.
      • Lluch-Canut T.
      • Trujols-Albet J.
      • Martínez-Momblan M.A.
      • et al.
      Nursing intensive-care satisfaction scale [NICSS]: development and validation of a patient-centred instrument.
      shows that a larger sample was necessary to determine the fit with the theoretical model,
      • Uriel E.
      • Aldás-Manzano J.
      Análisis multivariante aplicado: aplicaciones al marketing, investigación de mercados, economía, dirección de empresas y turismo (Applied multivariate analysis: applications in marketing, market research, economy, business management and tourism).
      as hypothesised.
      All the items presented an adequate factorial load and statistically significant differences from each other, unlike in our previous study, in which item sixdid not present statistically significant differences and item 11 had an insufficient factorial load (r = 0.147). The goodness of fit indices analysed from the confirmatory model presented a good fit to the MM-NICS.
      • Romero-García M.
      • de la Cueva-Ariza L.
      • Benito-Aracil L.
      • Lluch-Canut T.
      • Trujols-Albet J.
      • Martínez-Momblan M.A.
      • et al.
      Nursing intensive-care satisfaction scale [NICSS]: development and validation of a patient-centred instrument.
      These indices included the goodness of fit index and the root mean standard error (for measuring absolute fit) and the normed fit index (for measuring incremental fit).
      Unlike the NICSS, some scales submitted to a factor analysis have been shown to have only a single factor, such as the Patient Satisfaction Index
      • Hinshaw A.
      • Atwood J.
      A patient satisfaction instrument: precision by replication.
      and the Patient Satisfaction with Nursing Care Quality Questionnaire.
      • Laschinger H.S.
      • Hall L.M.
      • Pedersen C.
      • Almost J.
      A psychometric analysis of the patient satisfaction with nursing care quality questionnaire: an actionable approach to measuring patient satisfaction.
      ,
      • Milutinovic D.
      • Simin D.
      • Brkic N.
      • Brkic S.
      The patient satisfaction with nursing care quality: the psicometric study of the Serbian version of PSNCQ questionnaire.
      Because satisfaction is multidimensional, it makes sense to use a multifactorial instrument to evaluate it so that specific areas for improvement can be defined. In addition to the NICSS,
      • Romero-García M.
      • de la Cueva-Ariza L.
      • Benito-Aracil L.
      • Lluch-Canut T.
      • Trujols-Albet J.
      • Martínez-Momblan M.A.
      • et al.
      Nursing intensive-care satisfaction scale [NICSS]: development and validation of a patient-centred instrument.
      other multifactorial satisfaction scales include the Caring Assessment Report Evaluation Q-sort,
      • Chen S.-Y.
      • Yen W.-J.
      • Lin Y.-J.
      • Lee C.-H.
      • Lu Y.-C.
      A Chinese version of the Caring Assessment Report Evaluation Q-sort scale for measuring patients ‘perception on nurses' caring behaviours: reliability and validity assessment.
      the Monica-Oberst Patient Satisfaction Scale,
      • La-Monica E.L.
      • Oberst M.T.
      • Madea A.R.
      • Wolf R.M.
      Development of a patient satisfaction scale.
      and the SERVQUAL Scale,
      • Calixto-Olalde M.G.
      • Okino N.O.
      • Hayashida M.
      • Costa I.A.
      • Trevizan M.A.
      • Godoy S.
      SERVQUA scale: validation in the mexican population.
      but these scales do not focus on critically ill patients.
      Regarding criterion validity, the correlations of the different factors of the NICSS with the criterion items about overall satisfaction, intent to be treated again at the same ICU, and expectations met, a moderate to high correlation is obtained with the proposed items (range: 0.46–0.57). These results coincide with those of other studies that show positive associations between the level of satisfaction and the criterion items used (14, 42, 46, 48, and 49).

      6. Limitations

      Due to the characteristics of the method and the scale, this study has some limitations. There may be a survival bias because patients who died during their stay were not able to complete the scale, for obvious reasons. It is also possible that dissatisfied patients chose not to complete the questionnaire. Additionally, public hospitals were overrepresented in the sample.
      Finally, the retest presents the difficulty that participants may have remembered their responses from the first administration. The retest was also limited by the fact that some patients did not fill out the questionnaire a second time, because they either had been discharged or chose not to continue participating.

      7. Conclusion

      We have presented the psychometric properties of the NICSS and evaluated its ability to measure the satisfaction of a sample of critically ill patients from 19 ICUs of public and private level II and III hospitals throughout Spain. Construct validity showed a four-factor structure and robust fit with the MM-NICS theoretical model. The criterion validity of the NICSS was moderate to high when correlated with the level of overall satisfaction, the intent to be treated again at the same ICU, and expectations met. The NICSS is feasible and easy to complete for patients who have recently been discharged from the ICU.
      The NICSS is a useful instrument for guiding clinical practice because it enables clinicians, to evaluate and monitor critically ill patients’ satisfaction with the care they receive. Because the NICSS makes it possible to identify areas for improvement, health professionals—specifically nurses, managers, and administrators—can use it to create action plans for increasing the quality of care by modifying behaviours, skills, attitudes, or situations that play a role in nursing care in particular, and in health care in general. Additionally, its use encourages the ongoing improvement of humanised care from a perspective of professional commitment.
      Our study expands the existing literature on satisfaction with nursing care in the ICU and provides a stronger case for the validity of the NICSS. Although we have focused on the Spanish health system, the NICSS is currently being adapted and validated for use in other countries, including Japan, Turkey, the United States, and Iran. Future studies could contextualise the instrument in other countries or other areas outside ICUs. This instrument is of international interest because it makes it possible to measure the satisfaction of critically ill patients in a way that incorporates the patient's perspective both in the design and validation processes.

      Conflict of interest

      The authors declare no conflict of interest.

      Funding

      This study had been funded by the Nurse and Society Foundation as part of the Nurse Research Projects Grants ( PR-282/18 ), as well as financed by the Carlos III Health Institute (Ministry of Science and Innovation) and co-financed by the European Regional Development Fund, whose motto is “A way to build Europe”, FIS (PI18/00935).

      CRediT authorship contribution statement

      Marta Romero-García: Conceptualisation, Methodology, Supervision, Project administration, Formal analysis, Writing e-original draft, Writing e-review and editing.
      Isidro Alcalà-Jimenez: Methodology, Data curation, Software, Writing e-review and editing.
      María Antonia Martínez-Momblan: Writing e-review and editing.
      Laura de la Cueva-Ariza: Writing e-review and editing.
      Cecilia Cuzco: Writing e-review and editing.
      Sergio Alonso: Writing e-review and editing.
      Llúcia Benito-Aracil: Conceptualisation, Methodology, Formal analysis, Writing e-original draft, Writing e-review and, editing.
      Pilar Delgado-Hito: Conceptualisation, Methodology, Supervision, Project administration, Formal analysis, Writing e-original draft, Writing e-review, and editing.

      Acknowledgements

      The authors thank to the patients for their help, patience and participation.
      The authors also thank all the main research nurses from each of the centers: Irene Otero (Hospital del Mar), Gemma Martinez (Hospital Clínico de Barcelona), Jordi Adamuz and Carlos Santos (Hospital Universitari de Bellvitge), Pilar Muñoz (Hospital Germans Trias i Pujol), Rocío Tabernero and Elisabet Gallart (Vall d'Hebron Hospital), Àngels Piñol (Hospital Verge de la Cinta Tortosa), Anna Custodi (Vic Hospital), Celia Sánchez (Son Espases University Hospital), Marisol Holanda (Marqués de Valdecilla Hospital), Ignacio Zaragoza (12 Octubre University Hospital), Cristina Diez (Gregorio Marañón Hospital), Susana Arias and Mar Sánchez (Getafe University Hospital), Marga García and Ángela Alonso (Fuenlabrada University Hospital), Gabriel Heras (University Hospital of Torrejón), Antonio Coba (San Juan de Dios Hospital of Córdoba), Ana Cristina Álvarez (Mutual Accident Hospital of Zaragoza), Juana María Gutiérrez (University Hospital of Palencia), Dolores Saiz (Albacete Hospital), Álvaro Ortega (Quironsalud Málaga Hospital) and Concepción Zaforteza (Comarcal d'Inca Hospital); as well as all the collaborating research nurses who participated in the data collection.
      We thank CERCA Programme / Generalitat de Catalunya for institutional support.
      Susan Frekko provided feedback on the manuscript and translated it from the original Spanish.

      Appendix A. Supplementary data

      The following is/are the Supplementary data to this article:

      References

        • Romero-García M.
        • Trujols-Albet J.
        Towards increased incorporation of the patient perspective in the design of evaluation tools for the evaluation of efficiency and quality of patient care.
        Enfermería Intensiva. 2015; 26: 1-2https://doi.org/10.1016/j.enfi.2015.01.001
        • Romero-García M.
        • de la Cueva-Ariza L.
        • Benito-Aracil L.
        • Lluch-Canut T.
        • Trujols-Albet J.
        • Martínez-Momblan M.A.
        • et al.
        Nursing intensive-care satisfaction scale [NICSS]: development and validation of a patient-centred instrument.
        J Adv Nurs. 2018; 74: 1423-1435https://doi.org/10.1111/jan.13546
        • Konerding U.
        • Bowen T.
        • Elkhuizen S.G.
        • Faubel R.
        • Forte P.
        • Karampli E.
        • et al.
        Development of a universal short patient satisfaction questionnaire on the basis of SERVQUAL: psychometric analyses with data of diabetes and stroke patients from six different European countries.
        PLoS One. 2019; 14: 1-24https://doi.org/10.1371/journal.pone.0197924
        • Ng J.H.Y.
        • Luk B.H.K.
        Patient satisfaction: concept analysis in the healthcare context.
        Patient Educ Counsel. 2019; 102: 790-796https://doi.org/10.1016/j.pec.2018.11.013
        • Iftikhar A.
        • Allah N.
        • Shadiullah K.
        • Habibullah K.
        • Muhammad A.,R.
        • Muhammad H.,K.
        Predictors of patient satisfaction.
        Gomal J Med Sci. 2011; 9: 183-188
        • Aiken L.H.
        • Sloane D.M.
        • Ball J.
        • Bruyneel L.
        • Rafferty A.M.
        • Griffiths P.
        Patient satisfaction with hospital care and nurses in England: an observational study.
        BMJ Open. 2018; 8e019189https://doi.org/10.1136/bmjopen-2017-019189
        • Bull C.
        • Byrnes J.
        • Hettiarachchi R.
        • Downes M.
        A systematic review of the validity and reliability of patient-reported experience measures.
        Health Serv Res. 2019; 54: 1023-1035https://doi.org/10.1111/1475-6773.13187
        • Romero M.
        • de la Cueva L.
        • Jover C.
        • Delgado P.
        • Acosta B.
        • Sola M.
        • et al.
        Perception of the critical patient on nursing cares: an approach to the concept of satisfaction.
        Enfermería Intensiva. 2013; 24: 51-62https://doi.org/10.1016/j.enfi.2012.09.003
        • Batbaatar E.
        • Dorjdagva J.
        • Luvsannyam A.
        • Amenta P.
        Conceptualisation of patient satisfaction: a systematic narrative literature review.
        Perspect. Public Health. 2015; 135: 243-250https://doi.org/10.1177/1757913915594196
        • Goh M.L.
        • Ang E.N.K.
        • Chan Y.H.
        • He H.G.
        • Vehviläinen-Julkunen K.
        A descriptive quantitative study on multi-ethnic patient satisfaction with nursing care measured by the Revised Humane Caring Scale.
        Appl Nurs Res. 2016; 31: 126-131https://doi.org/10.1016/j.apnr.2016.02.002
        • Otani K.
        • Waterman B.
        • Dunagan W.C.
        Patient satisfaction: how patient health conditions influence their satisfaction.
        J Healthc Manag. 2012; 57: 276-292
        • Otani K.
        • Waterman B.
        • Faulkner K.M.
        • Boslaugh S.
        • Burroughs T.E.
        • Dunagan W.C.
        Patient satisfaction: focusing on "excellent.
        J Healthc Manag. 2009; 54: 93-102
        • Donabedian A.
        The quality of medical care: how can it be assessed?.
        Rev Calid Asist. 2001; 16: 580-587
        • Laschinger H.S.
        • Hall L.M.
        • Pedersen C.
        • Almost J.
        A psychometric analysis of the patient satisfaction with nursing care quality questionnaire: an actionable approach to measuring patient satisfaction.
        J Nurs Care Qual. 2005; 20: 220-230https://doi.org/10.1097/00001786-200507000-00006
        • Reck D.L.
        Can and should nurses be aware of patients' expectations for their nursing care?.
        Nurs Adm Q. 2013; 37: 109-115https://doi.org/10.1097/NAQ.0b013e3182869e03
        • Babakus E.
        • Mangold W.G.
        Adapting the SERVQUAL scale to hospital services: an empirical investigation.
        Health Serv Res. 1992; 26: 767-786
        • Davis B.
        • Bush H.
        Developing effective measurement tools: a case study of the consumer emergency care satisfaction scale.
        J Nurs Care Qual. 1995; 9: 26-35https://doi.org/10.1097/00001786-199501000-00007
        • La-Monica E.L.
        • Oberst M.T.
        • Madea A.R.
        • Wolf R.M.
        Development of a patient satisfaction scale.
        Res Nurs Health. 1986; 9: 43-50https://doi.org/10.1002/nur.4770090108
        • Larson P.J.
        • Ferketich S.L.
        Patients' satisfaction with nurses' caring during hospitalization.
        West J Nurs Res. 1993; 15: 690-707https://doi.org/10.1177/019394599301500603
        • Lynn M.R.
        • McMillen B.J.
        • Sidani S.
        Understanding and measuring patients' assessment of the quality of nursing care.
        Nurs Res. 2007; 56: 159-166https://doi.org/10.1097/01.NNR.0000270025.52242.70
        • Thomas L.H.
        • McColl E.
        • Priest J.
        • Bond S.
        • Boys R.J.
        Newcastle satisfaction with nursing scales: an instrument for quality assessments of nursing care.
        Qual Health Care. 1996; 5: 67-72https://doi.org/10.1136/qshc.5.2.67
        • Risser N.L.
        Development of an instrument to measure patient satisfaction with nurses and nursing care in primary care settings.
        Nurs Res. 1975; 24: 45-52https://doi.org/10.1097/00006199-197501000-00011
        • Trujols J.
        • Iraurgi I.
        • Oviedo-Joekes E.
        • Guàrdia-Olmos J.
        A critical analysis of user satisfaction surveys in addiction services: opioid maintenance treatment as a representative case study.
        Patient Prefer Adherence. 2014; 8: 107-117https://doi.org/10.2147/PPA.S52060
        • de la Cueva-Ariza L.
        • Romero-García M.,M.
        • Delgado-Hito P.
        • Acosta-Mejuto B.
        • Jover-Sancho C.
        • Ricart-Basagaña M.T.
        • et al.
        Development of an instrument to measure the degree of critical patient's satisfaction with nursing care: research protocol.
        J Adv Nurs. 2014; 70: 201-210https://doi.org/10.1111/jan.12184
        • Lasch K.E.
        • Marquis P.
        • Vigneux M.
        • Abetz L.
        • Arnould B.
        • Bayliss M.
        • et al.
        PRO development rigorous qualitative research as the crucial foundation.
        Qual Life Res. 2010; 19: 1087-1096https://doi.org/10.1007/s11136-010-9677-6
        • Uriel E.
        • Aldás-Manzano J.
        Análisis multivariante aplicado: aplicaciones al marketing, investigación de mercados, economía, dirección de empresas y turismo (Applied multivariate analysis: applications in marketing, market research, economy, business management and tourism).
        Editorial Thomson, Madrid2005
        • Streiner D.L.
        Starting at the beginning: an introduction to coefficient alpha and internal consistency.
        J Pers Assess. 2003; 80: 99-103https://doi.org/10.1207/S15327752JPA8001_18
        • Streiner D.
        • Norman G.
        • Carney J.
        Health measurement scales a practical guide to their development and use.
        5 ed. Oxford University Press Publications, New York2014https://doi.org/10.1093/med/9780199685219.001.0001
        • Ferketich S.
        Focus on psychometrics aspects of item analysis.
        Res Nurs Health. 1991; 14: 165-168https://doi.org/10.1002/nur.4770140211
        • Anderson J.
        • Gerbing D.
        Structural equation modeling in practice: a review and recommended two-step approach.
        Psychol Bull. 1988; 103: 411-423https://doi.org/10.1037/0033-2909.103.3.411
        • Boomsma A.
        Reporting analyses of covariance structures.
        Struct Equ Model. 2000; 7: 461-483https://doi.org/10.1207/S15328007SEM0703_6
        • Hu L.
        • Bentler P.M.
        Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives.
        Struct Equ Model. 1999; 6: 1-55https://doi.org/10.1080/10705519909540118
        • Schermelleh-Engel K.
        • Moosbrugger H.
        • y Müller H.
        Evaluating the fit of structural equation models: tests of significance and descriptive goodness-of-fit measures.
        Methods of Psychological Research Online. 2003; 8: 23-74
        • Hair J.
        • Anderson R.
        • Tatham R.
        • Black W.
        Análisis multivariante (multivariate data analysis).
        5th ed. Prentice-Hall, Madrid1999
        • Bentler P.M.
        • Bonett D.G.
        Significance tests and goodness of fit in the analysis of covariance structures.
        Psychol Bull. 1980; 88: 588-606https://doi.org/10.1037/0033-2909.88.3.588
        • Bland J.M.
        • Altman D.G.
        Cronbach’s alpha. B.M.J. 1997; 314: 572https://doi.org/10.1136/bmj.314.7080.572
        • Nunnally J.C.
        • Bernstein I.H.
        Psychometric theory.
        3rd ed. McGraw-Hill, Mexico1995
        • Cicchetti D.V.
        Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in Psychology.
        Psychol Assess. 1994; 6: 284-290https://doi.org/10.1037/1040-3590.6.4.284
        • Trujols J.
        • Portella M.J.
        • Iraurgi I.
        • Campins M.J.
        • Siñol N.
        • de los Cobos J.P.
        Patient-reported outcome measures: are they patient-generated, patient-centred or patient-valued?.
        J Ment Health. 2013; 22: 555-562https://doi.org/10.3109/09638237.2012.734653
        • Charalambous A.
        • Adamakidou T.
        Risser patient satisfaction scale: a validation study in Greek cancer patients.
        BMC Nurs. 2012; 11: 27https://doi.org/10.1186/1472-6955-11-27
        • Hinshaw A.
        • Atwood J.
        A patient satisfaction instrument: precision by replication.
        Nurs Res. 1982; 31: 170-191
        • Larson P.J.
        Important nurse caring behaviors perceived by patients with cancer.
        Oncol Nurs Forum. 1984; 11: 46-50
        • McColl E.
        • Thomas L.
        • Bond S.
        A study to determine patient satisfaction with nursing care.
        Nurs Stand. 1996; 10: 34-38
        • Chen S.-Y.
        • Yen W.-J.
        • Lin Y.-J.
        • Lee C.-H.
        • Lu Y.-C.
        A Chinese version of the Caring Assessment Report Evaluation Q-sort scale for measuring patients ‘perception on nurses' caring behaviours: reliability and validity assessment.
        Int J Nurs Pract. 2012; 18: 388-395https://doi.org/10.1111/j.1440-172X.2012.02042.x
        • Milutinovic D.
        • Simin D.
        • Brkic N.
        • Brkic S.
        The patient satisfaction with nursing care quality: the psicometric study of the Serbian version of PSNCQ questionnaire.
        Scand J Caring Sci. 2012; 26: 598-606https://doi.org/10.1111/j.1471-6712.2012.00969.x
        • Calixto-Olalde M.G.
        • Okino N.O.
        • Hayashida M.
        • Costa I.A.
        • Trevizan M.A.
        • Godoy S.
        SERVQUA scale: validation in the mexican population.
        Texto Contexto Enferm. 2011; 20: 526-533https://doi.org/10.1590/S0104-07072011000300014