Competency assessment of residents of Intensive Care Medicine through a simulation-based objective structured clinical evaluation (OSCE). A multicenter observational study
Objectives
The current official model of training in Intensive Care Medicine (ICM) in Spain is based on exposure to experiences through clinical rotations. The main objective was to determine the level of competency (I novice to V independent practitioner) achieved by the residents at the end of the 3rd year of training (R3) in ICM through a simulation-based OSCE. Secondary objectives were: (1) To identify gaps in performance, and (2) To investigate the reliability and feasibility of conducting simulation-based assessment at multiple sites.
Design
Observational multicenter study.
Setting
Thirteen Spanish ICU Departments.
Participants
Thirty six R3.
Intervention
The participants performed on five, 15-min, high-fidelity crisis scenarios in four simulation centers. The performances were video recorded for later scoring by trained raters.
Main variables of interest
Via a Delphi technique, an independent panel of expert intensivists identified critical essential performance elements (CEPE) for each scenario to define the levels of competency.
Results
A total of 176 performances were analyzed. The internal consistency of the check-lists were adequate (KR-20 range 0.64–0.79). Inter-rater reliability was strong [median Intraclass Correlation Coefficient across scenarios: 0.89 (0.65–0.97)]. Competency levels achieved by R3 were: Level I (18.8%), II (35.2%), III (42.6%), IV/V (3.4%). Overall, a great heterogeneity in performance was observed.
Competency assessment of residents of Intensive Care Medicine through a simulation-based objective structured clinical evaluation (OSCE). A multicenter observational study
Objectives
The current official model of training in Intensive Care Medicine (ICM) in Spain is based on exposure to experiences through clinical rotations. The main objective was to determine the level of competency (I novice to V independent practitioner) achieved by the residents at the end of the 3rd year of training (R3) in ICM through a simulation-based OSCE. Secondary objectives were: (1) To identify gaps in performance, and (2) To investigate the reliability and feasibility of conducting simulation-based assessment at multiple sites.
Design
Observational multicenter study.
Setting
Thirteen Spanish ICU Departments.
Participants
Thirty six R3.
Intervention
The participants performed on five, 15-min, high-fidelity crisis scenarios in four simulation centers. The performances were video recorded for later scoring by trained raters.
Main variables of interest
Via a Delphi technique, an independent panel of expert intensivists identified critical essential performance elements (CEPE) for each scenario to define the levels of competency.
Results
A total of 176 performances were analyzed. The internal consistency of the check-lists were adequate (KR-20 range 0.64–0.79). Inter-rater reliability was strong [median Intraclass Correlation Coefficient across scenarios: 0.89 (0.65–0.97)]. Competency levels achieved by R3 were: Level I (18.8%), II (35.2%), III (42.6%), IV/V (3.4%). Overall, a great heterogeneity in performance was observed.
Competency assessment of residents of Intensive Care Medicine through a simulation-based objective structured clinical evaluation (OSCE). A multicenter observational study
Objectives
The current official model of training in Intensive Care Medicine (ICM) in Spain is based on exposure to experiences through clinical rotations. The main objective was to determine the level of competency (I novice to V independent practitioner) achieved by the residents at the end of the 3rd year of training (R3) in ICM through a simulation-based OSCE. Secondary objectives were: (1) To identify gaps in performance, and (2) To investigate the reliability and feasibility of conducting simulation-based assessment at multiple sites.
Design
Observational multicenter study.
Setting
Thirteen Spanish ICU Departments.
Participants
Thirty six R3.
Intervention
The participants performed on five, 15-min, high-fidelity crisis scenarios in four simulation centers. The performances were video recorded for later scoring by trained raters.
Main variables of interest
Via a Delphi technique, an independent panel of expert intensivists identified critical essential performance elements (CEPE) for each scenario to define the levels of competency.
Results
A total of 176 performances were analyzed. The internal consistency of the check-lists were adequate (KR-20 range 0.64–0.79). Inter-rater reliability was strong [median Intraclass Correlation Coefficient across scenarios: 0.89 (0.65–0.97)]. Competency levels achieved by R3 were: Level I (18.8%), II (35.2%), III (42.6%), IV/V (3.4%). Overall, a great heterogeneity in performance was observed.
Competency assessment of residents of Intensive Care Medicine through a simulation-based objective structured clinical evaluation (OSCE). A multicenter observational study
Objectives
The current official model of training in Intensive Care Medicine (ICM) in Spain is based on exposure to experiences through clinical rotations. The main objective was to determine the level of competency (I novice to V independent practitioner) achieved by the residents at the end of the 3rd year of training (R3) in ICM through a simulation-based OSCE. Secondary objectives were: (1) To identify gaps in performance, and (2) To investigate the reliability and feasibility of conducting simulation-based assessment at multiple sites.
Design
Observational multicenter study.
Setting
Thirteen Spanish ICU Departments.
Participants
Thirty six R3.
Intervention
The participants performed on five, 15-min, high-fidelity crisis scenarios in four simulation centers. The performances were video recorded for later scoring by trained raters.
Main variables of interest
Via a Delphi technique, an independent panel of expert intensivists identified critical essential performance elements (CEPE) for each scenario to define the levels of competency.
Results
A total of 176 performances were analyzed. The internal consistency of the check-lists were adequate (KR-20 range 0.64–0.79). Inter-rater reliability was strong [median Intraclass Correlation Coefficient across scenarios: 0.89 (0.65–0.97)]. Competency levels achieved by R3 were: Level I (18.8%), II (35.2%), III (42.6%), IV/V (3.4%). Overall, a great heterogeneity in performance was observed.
Competency assessment of residents of Intensive Care Medicine through a simulation-based objective structured clinical evaluation (OSCE). A multicenter observational study
Objectives
The current official model of training in Intensive Care Medicine (ICM) in Spain is based on exposure to experiences through clinical rotations. The main objective was to determine the level of competency (I novice to V independent practitioner) achieved by the residents at the end of the 3rd year of training (R3) in ICM through a simulation-based OSCE. Secondary objectives were: (1) To identify gaps in performance, and (2) To investigate the reliability and feasibility of conducting simulation-based assessment at multiple sites.
Design
Observational multicenter study.
Setting
Thirteen Spanish ICU Departments.
Participants
Thirty six R3.
Intervention
The participants performed on five, 15-min, high-fidelity crisis scenarios in four simulation centers. The performances were video recorded for later scoring by trained raters.
Main variables of interest
Via a Delphi technique, an independent panel of expert intensivists identified critical essential performance elements (CEPE) for each scenario to define the levels of competency.
Results
A total of 176 performances were analyzed. The internal consistency of the check-lists were adequate (KR-20 range 0.64–0.79). Inter-rater reliability was strong [median Intraclass Correlation Coefficient across scenarios: 0.89 (0.65–0.97)]. Competency levels achieved by R3 were: Level I (18.8%), II (35.2%), III (42.6%), IV/V (3.4%). Overall, a great heterogeneity in performance was observed.
Competency assessment of residents of Intensive Care Medicine through a simulation-based objective structured clinical evaluation (OSCE). A multicenter observational study
Objectives
The current official model of training in Intensive Care Medicine (ICM) in Spain is based on exposure to experiences through clinical rotations. The main objective was to determine the level of competency (I novice to V independent practitioner) achieved by the residents at the end of the 3rd year of training (R3) in ICM through a simulation-based OSCE. Secondary objectives were: (1) To identify gaps in performance, and (2) To investigate the reliability and feasibility of conducting simulation-based assessment at multiple sites.
Design
Observational multicenter study.
Setting
Thirteen Spanish ICU Departments.
Participants
Thirty six R3.
Intervention
The participants performed on five, 15-min, high-fidelity crisis scenarios in four simulation centers. The performances were video recorded for later scoring by trained raters.
Main variables of interest
Via a Delphi technique, an independent panel of expert intensivists identified critical essential performance elements (CEPE) for each scenario to define the levels of competency.
Results
A total of 176 performances were analyzed. The internal consistency of the check-lists were adequate (KR-20 range 0.64–0.79). Inter-rater reliability was strong [median Intraclass Correlation Coefficient across scenarios: 0.89 (0.65–0.97)]. Competency levels achieved by R3 were: Level I (18.8%), II (35.2%), III (42.6%), IV/V (3.4%). Overall, a great heterogeneity in performance was observed.
Competency assessment of residents of Intensive Care Medicine through a simulation-based objective structured clinical evaluation (OSCE). A multicenter observational study
Objectives
The current official model of training in Intensive Care Medicine (ICM) in Spain is based on exposure to experiences through clinical rotations. The main objective was to determine the level of competency (I novice to V independent practitioner) achieved by the residents at the end of the 3rd year of training (R3) in ICM through a simulation-based OSCE. Secondary objectives were: (1) To identify gaps in performance, and (2) To investigate the reliability and feasibility of conducting simulation-based assessment at multiple sites.
Design
Observational multicenter study.
Setting
Thirteen Spanish ICU Departments.
Participants
Thirty six R3.
Intervention
The participants performed on five, 15-min, high-fidelity crisis scenarios in four simulation centers. The performances were video recorded for later scoring by trained raters.
Main variables of interest
Via a Delphi technique, an independent panel of expert intensivists identified critical essential performance elements (CEPE) for each scenario to define the levels of competency.
Results
A total of 176 performances were analyzed. The internal consistency of the check-lists were adequate (KR-20 range 0.64–0.79). Inter-rater reliability was strong [median Intraclass Correlation Coefficient across scenarios: 0.89 (0.65–0.97)]. Competency levels achieved by R3 were: Level I (18.8%), II (35.2%), III (42.6%), IV/V (3.4%). Overall, a great heterogeneity in performance was observed.
Competency assessment of residents of Intensive Care Medicine through a simulation-based objective structured clinical evaluation (OSCE). A multicenter observational study
Objectives
The current official model of training in Intensive Care Medicine (ICM) in Spain is based on exposure to experiences through clinical rotations. The main objective was to determine the level of competency (I novice to V independent practitioner) achieved by the residents at the end of the 3rd year of training (R3) in ICM through a simulation-based OSCE. Secondary objectives were: (1) To identify gaps in performance, and (2) To investigate the reliability and feasibility of conducting simulation-based assessment at multiple sites.
Design
Observational multicenter study.
Setting
Thirteen Spanish ICU Departments.
Participants
Thirty six R3.
Intervention
The participants performed on five, 15-min, high-fidelity crisis scenarios in four simulation centers. The performances were video recorded for later scoring by trained raters.
Main variables of interest
Via a Delphi technique, an independent panel of expert intensivists identified critical essential performance elements (CEPE) for each scenario to define the levels of competency.
Results
A total of 176 performances were analyzed. The internal consistency of the check-lists were adequate (KR-20 range 0.64–0.79). Inter-rater reliability was strong [median Intraclass Correlation Coefficient across scenarios: 0.89 (0.65–0.97)]. Competency levels achieved by R3 were: Level I (18.8%), II (35.2%), III (42.6%), IV/V (3.4%). Overall, a great heterogeneity in performance was observed.
