Competency-based medical education (CBME) is
changing the way physicians are educated,1–4 with a
heavy emphasis on quantifying and qualifying their
performance via robust assessments. The central tenet
of CBME is that trainees must demonstrate competence in applying acquired skills during patient care
activities,5 and CBME requires assessment strategies
that ensure trainees apply their knowledge, skills, and
abilities in authentic or simulated environments.6 This
necessity for direct observation and assessment of
trainees’ performance has resulted in a shift toward
workplace-based assessments (WBAs) as a primary
method of assessment.
There is no single comprehensive WBA assessment
tool, and experts have argued that decisions regarding
trainees’ progression should be based on aggregates of
multiple measures of performance using both qualitative and quantitative methods.6,7 The implementation of WBAs allows educators to identify patterns in
the development of knowledge, skills, and performance. While educators and researchers have paid
considerable attention to understanding patterns in
WBA data, less attention has been paid to missing
data. Identifying and understanding potential patterns
underlying missing data is an important step in
accurately interpreting WBA data.
While mechanisms exist to deal with missing data
(eg, multiple imputation and maximum likelihood
methods), many of these presume that data are
missing at random.8 This may not be the case in the
context of WBA portfolios. For example, residents
may be more likely to complete WBAs for tasks that
they enjoy and/or perform well; consequently, there
may be missing data for more poorly developed
knowledge, skills, and abilities. Similarly, certain
WBA tools (eg, multi-source feedback) may be
particularly challenging to complete because of the
logistics of collecting the data. Nonrandomly missing
data could threaten the inherent validity of WBA
portfolios.
The purpose of this study is twofold. First, we
examined whether data are, in fact, missing at
random across various competencies within the
context of our local WBA system. Second, we assessed
whether the amount of missing data correlated with
overall resident performance as determined by a panel
of faculty from the residency education committee
Competency-based medical education (CBME) is
changing the way physicians are educated,1–4 with a
heavy emphasis on quantifying and qualifying their
performance via robust assessments. The central tenet
of CBME is that trainees must demonstrate competence in applying acquired skills during patient care
activities,5 and CBME requires assessment strategies
that ensure trainees apply their knowledge, skills, and
abilities in authentic or simulated environments.6 This
necessity for direct observation and assessment of
trainees’ performance has resulted in a shift toward
workplace-based assessments (WBAs) as a primary
method of assessment.
There is no single comprehensive WBA assessment
tool, and experts have argued that decisions regarding
trainees’ progression should be based on aggregates of
multiple measures of performance using both qualitative and quantitative methods.6,7 The implementation of WBAs allows educators to identify patterns in
the development of knowledge, skills, and performance. While educators and researchers have paid
considerable attention to understanding patterns in
WBA data, less attention has been paid to missing
data. Identifying and understanding potential patterns
underlying missing data is an important step in
accurately interpreting WBA data.
While mechanisms exist to deal with missing data
(eg, multiple imputation and maximum likelihood
methods), many of these presume that data are
missing at random.8 This may not be the case in the
context of WBA portfolios. For example, residents
may be more likely to complete WBAs for tasks that
they enjoy and/or perform well; consequently, there
may be missing data for more poorly developed
knowledge, skills, and abilities. Similarly, certain
WBA tools (eg, multi-source feedback) may be
particularly challenging to complete because of the
logistics of collecting the data. Nonrandomly missing
data could threaten the inherent validity of WBA
portfolios.
The purpose of this study is twofold. First, we
examined whether data are, in fact, missing at
random across various competencies within the
context of our local WBA system. Second, we assessed
whether the amount of missing data correlated with
overall resident performance as determined by a panel
of faculty from the residency education committee
Competency-based medical education (CBME) is
changing the way physicians are educated,1–4 with a
heavy emphasis on quantifying and qualifying their
performance via robust assessments. The central tenet
of CBME is that trainees must demonstrate competence in applying acquired skills during patient care
activities,5 and CBME requires assessment strategies
that ensure trainees apply their knowledge, skills, and
abilities in authentic or simulated environments.6 This
necessity for direct observation and assessment of
trainees’ performance has resulted in a shift toward
workplace-based assessments (WBAs) as a primary
method of assessment.
There is no single comprehensive WBA assessment
tool, and experts have argued that decisions regarding
trainees’ progression should be based on aggregates of
multiple measures of performance using both qualitative and quantitative methods.6,7 The implementation of WBAs allows educators to identify patterns in
the development of knowledge, skills, and performance. While educators and researchers have paid
considerable attention to understanding patterns in
WBA data, less attention has been paid to missing
data. Identifying and understanding potential patterns
underlying missing data is an important step in
accurately interpreting WBA data.
While mechanisms exist to deal with missing data
(eg, multiple imputation and maximum likelihood
methods), many of these presume that data are
missing at random.8 This may not be the case in the
context of WBA portfolios. For example, residents
may be more likely to complete WBAs for tasks that
they enjoy and/or perform well; consequently, there
may be missing data for more poorly developed
knowledge, skills, and abilities. Similarly, certain
WBA tools (eg, multi-source feedback) may be
particularly challenging to complete because of the
logistics of collecting the data. Nonrandomly missing
data could threaten the inherent validity of WBA
portfolios.
The purpose of this study is twofold. First, we
examined whether data are, in fact, missing at
random across various competencies within the
context of our local WBA system. Second, we assessed
whether the amount of missing data correlated with
overall resident performance as determined by a panel
of faculty from the residency education committee
Competency-based medical education (CBME) is
changing the way physicians are educated,1–4 with a
heavy emphasis on quantifying and qualifying their
performance via robust assessments. The central tenet
of CBME is that trainees must demonstrate competence in applying acquired skills during patient care
activities,5 and CBME requires assessment strategies
that ensure trainees apply their knowledge, skills, and
abilities in authentic or simulated environments.6 This
necessity for direct observation and assessment of
trainees’ performance has resulted in a shift toward
workplace-based assessments (WBAs) as a primary
method of assessment.
There is no single comprehensive WBA assessment
tool, and experts have argued that decisions regarding
trainees’ progression should be based on aggregates of
multiple measures of performance using both qualitative and quantitative methods.6,7 The implementation of WBAs allows educators to identify patterns in
the development of knowledge, skills, and performance. While educators and researchers have paid
considerable attention to understanding patterns in
WBA data, less attention has been paid to missing
data. Identifying and understanding potential patterns
underlying missing data is an important step in
accurately interpreting WBA data.
While mechanisms exist to deal with missing data
(eg, multiple imputation and maximum likelihood
methods), many of these presume that data are
missing at random.8 This may not be the case in the
context of WBA portfolios. For example, residents
may be more likely to complete WBAs for tasks that
they enjoy and/or perform well; consequently, there
may be missing data for more poorly developed
knowledge, skills, and abilities. Similarly, certain
WBA tools (eg, multi-source feedback) may be
particularly challenging to complete because of the
logistics of collecting the data. Nonrandomly missing
data could threaten the inherent validity of WBA
portfolios.
The purpose of this study is twofold. First, we
examined whether data are, in fact, missing at
random across various competencies within the
context of our local WBA system. Second, we assessed
whether the amount of missing data correlated with
overall resident performance as determined by a panel
of faculty from the residency education committee
Competency-based medical education (CBME) is
changing the way physicians are educated,1–4 with a
heavy emphasis on quantifying and qualifying their
performance via robust assessments. The central tenet
of CBME is that trainees must demonstrate competence in applying acquired skills during patient care
activities,5 and CBME requires assessment strategies
that ensure trainees apply their knowledge, skills, and
abilities in authentic or simulated environments.6 This
necessity for direct observation and assessment of
trainees’ performance has resulted in a shift toward
workplace-based assessments (WBAs) as a primary
method of assessment.
There is no single comprehensive WBA assessment
tool, and experts have argued that decisions regarding
trainees’ progression should be based on aggregates of
multiple measures of performance using both qualitative and quantitative methods.6,7 The implementation of WBAs allows educators to identify patterns in
the development of knowledge, skills, and performance. While educators and researchers have paid
considerable attention to understanding patterns in
WBA data, less attention has been paid to missing
data. Identifying and understanding potential patterns
underlying missing data is an important step in
accurately interpreting WBA data.
While mechanisms exist to deal with missing data
(eg, multiple imputation and maximum likelihood
methods), many of these presume that data are
missing at random.8 This may not be the case in the
context of WBA portfolios. For example, residents
may be more likely to complete WBAs for tasks that
they enjoy and/or perform well; consequently, there
may be missing data for more poorly developed
knowledge, skills, and abilities. Similarly, certain
WBA tools (eg, multi-source feedback) may be
particularly challenging to complete because of the
logistics of collecting the data. Nonrandomly missing
data could threaten the inherent validity of WBA
portfolios.
The purpose of this study is twofold. First, we
examined whether data are, in fact, missing at
random across various competencies within the
context of our local WBA system. Second, we assessed
whether the amount of missing data correlated with
overall resident performance as determined by a panel
of faculty from the residency education committee
Competency-based medical education (CBME) is
changing the way physicians are educated,1–4 with a
heavy emphasis on quantifying and qualifying their
performance via robust assessments. The central tenet
of CBME is that trainees must demonstrate competence in applying acquired skills during patient care
activities,5 and CBME requires assessment strategies
that ensure trainees apply their knowledge, skills, and
abilities in authentic or simulated environments.6 This
necessity for direct observation and assessment of
trainees’ performance has resulted in a shift toward
workplace-based assessments (WBAs) as a primary
method of assessment.
There is no single comprehensive WBA assessment
tool, and experts have argued that decisions regarding
trainees’ progression should be based on aggregates of
multiple measures of performance using both qualitative and quantitative methods.6,7 The implementation of WBAs allows educators to identify patterns in
the development of knowledge, skills, and performance. While educators and researchers have paid
considerable attention to understanding patterns in
WBA data, less attention has been paid to missing
data. Identifying and understanding potential patterns
underlying missing data is an important step in
accurately interpreting WBA data.
While mechanisms exist to deal with missing data
(eg, multiple imputation and maximum likelihood
methods), many of these presume that data are
missing at random.8 This may not be the case in the
context of WBA portfolios. For example, residents
may be more likely to complete WBAs for tasks that
they enjoy and/or perform well; consequently, there
may be missing data for more poorly developed
knowledge, skills, and abilities. Similarly, certain
WBA tools (eg, multi-source feedback) may be
particularly challenging to complete because of the
logistics of collecting the data. Nonrandomly missing
data could threaten the inherent validity of WBA
portfolios.
The purpose of this study is twofold. First, we
examined whether data are, in fact, missing at
random across various competencies within the
context of our local WBA system. Second, we assessed
whether the amount of missing data correlated with
overall resident performance as determined by a panel
of faculty from the residency education committee
Competency-based medical education (CBME) is
changing the way physicians are educated,1–4 with a
heavy emphasis on quantifying and qualifying their
performance via robust assessments. The central tenet
of CBME is that trainees must demonstrate competence in applying acquired skills during patient care
activities,5 and CBME requires assessment strategies
that ensure trainees apply their knowledge, skills, and
abilities in authentic or simulated environments.6 This
necessity for direct observation and assessment of
trainees’ performance has resulted in a shift toward
workplace-based assessments (WBAs) as a primary
method of assessment.
There is no single comprehensive WBA assessment
tool, and experts have argued that decisions regarding
trainees’ progression should be based on aggregates of
multiple measures of performance using both qualitative and quantitative methods.6,7 The implementation of WBAs allows educators to identify patterns in
the development of knowledge, skills, and performance. While educators and researchers have paid
considerable attention to understanding patterns in
WBA data, less attention has been paid to missing
data. Identifying and understanding potential patterns
underlying missing data is an important step in
accurately interpreting WBA data.
While mechanisms exist to deal with missing data
(eg, multiple imputation and maximum likelihood
methods), many of these presume that data are
missing at random.8 This may not be the case in the
context of WBA portfolios. For example, residents
may be more likely to complete WBAs for tasks that
they enjoy and/or perform well; consequently, there
may be missing data for more poorly developed
knowledge, skills, and abilities. Similarly, certain
WBA tools (eg, multi-source feedback) may be
particularly challenging to complete because of the
logistics of collecting the data. Nonrandomly missing
data could threaten the inherent validity of WBA
portfolios.
The purpose of this study is twofold. First, we
examined whether data are, in fact, missing at
random across various competencies within the
context of our local WBA system. Second, we assessed
whether the amount of missing data correlated with
overall resident performance as determined by a panel
of faculty from the residency education committee
Competency-based medical education (CBME) is
changing the way physicians are educated,1–4 with a
heavy emphasis on quantifying and qualifying their
performance via robust assessments. The central tenet
of CBME is that trainees must demonstrate competence in applying acquired skills during patient care
activities,5 and CBME requires assessment strategies
that ensure trainees apply their knowledge, skills, and
abilities in authentic or simulated environments.6 This
necessity for direct observation and assessment of
trainees’ performance has resulted in a shift toward
workplace-based assessments (WBAs) as a primary
method of assessment.
There is no single comprehensive WBA assessment
tool, and experts have argued that decisions regarding
trainees’ progression should be based on aggregates of
multiple measures of performance using both qualitative and quantitative methods.6,7 The implementation of WBAs allows educators to identify patterns in
the development of knowledge, skills, and performance. While educators and researchers have paid
considerable attention to understanding patterns in
WBA data, less attention has been paid to missing
data. Identifying and understanding potential patterns
underlying missing data is an important step in
accurately interpreting WBA data.
While mechanisms exist to deal with missing data
(eg, multiple imputation and maximum likelihood
methods), many of these presume that data are
missing at random.8 This may not be the case in the
context of WBA portfolios. For example, residents
may be more likely to complete WBAs for tasks that
they enjoy and/or perform well; consequently, there
may be missing data for more poorly developed
knowledge, skills, and abilities. Similarly, certain
WBA tools (eg, multi-source feedback) may be
particularly challenging to complete because of the
logistics of collecting the data. Nonrandomly missing
data could threaten the inherent validity of WBA
portfolios.
The purpose of this study is twofold. First, we
examined whether data are, in fact, missing at
random across various competencies within the
context of our local WBA system. Second, we assessed
whether the amount of missing data correlated with
overall resident performance as determined by a panel
of faculty from the residency education committee
