Kogan et al (2011) emphasise the importance of precise observation of skills demonstrated and of giving first-rate feedback as essential aspects for the development of competence.
In clinical training many tools have been developed and published for the direct observation of the skills of residents with patients (Kogan et al, 2011). Despite this, performance ratings are often subjective and subject to rating errors, such as inaccuracy and unreliability. However, there has not been much investigation into causal factors of rater variability. Explanations by raters themselves for low inter-rater agreement include “differences in observer gender, ethnicity, experience or clinical competence” (p. 1049). In a study by Kogan et al (2011: 1051), four themes were identified to explain
the variability of judgements and ratings, namely:
(i.) “the use of variable frames of reference during observation and rating;
(ii.) the role of inference;
(iii.) the use of variable approaches to synthesising judgements into numerical ratings; and
(iv.) factors external to resident performance that influence ratings.”
Kogan et al (2011: 1051) found that many direct observers use their own outlook—how they do things—or what they consider a respected practitioner would, as criteria for making judgements and giving ratings.
Based on the four themes, Kogan et al (2011: 1056) created a descriptive model of the direct observation process. Present within observers are an amalgam of characteristics, values, expertise and emotions that could impact their observations. Observation is done through two lenses, namely the frame of reference as yardstick lens; and the other lens represents inference, which shapes the meaning and interpretation of the observations. The observation is furthermore done within a context of practice. The observations and interpretations are synthesised into a rating. Unfortunately reality is not neatly, foreseeable or straightforward. There are often other influences and complexities present that may sway ratings. Feedback to the observer, further contribute to modify current and future ratings. The model can be supported by situated cognition theory, namely “an individual’s thinking, knowing and processing are uniquely tied to and inextricably situated within (and cannot be completely separated from) the specific social situations within which those thoughts and actions occur” (pp. 1056-7).
Kogan, J.R., Conforti, L., Bernabeo, E. Iobst, W. & Holmboe, E. 2011. Opening the black box of clinical skills assessment via observation: a conceptual model.
Medical Education, 45, 1048–1060.