The Case for Integrating Multi-Source Data for a Fairer and Holistic Judgement of Competence in Medical Education & Training: Tackling Differential Attainment in Medical Professions - Bridging the Gap Workshop Series 2020
Published 2020-11-10
Keywords
- Differential attainment,
- formative assessment,
- summative assessment,
- safe and competent doctor
How to Cite
Abstract
Being a doctor in the 21st Century requires a diverse range of skills, a broad base of knowledge and a suite of professional values and attitudes that enable the clinical practice to be safe, effective and caring. Doctors, irrespective of their speciality, need to be knowledgeable and skilful not just in their area of expertise but also need a range of generic skills and capabilities such as communication, leadership, academic scholarship and research, teaching, quality improvement, advocacy, digital literacy to name a few. These capabilities, all relevant to clinical practice, are assessed routinely in clinical settings. This rich information about trainees, available from their formative assessments, does not inform high-stakes judgements about progression. Instead, these judgements are usually made on the basis of summative examinations conducted in simulated settings.
Unfortunately, these summative assessments have consistently delivered results with a large magnitude of the differential between the outcomes of candidates, based on factors such as ethnicity, gender, other protected characteristics and also the country of primary medical qualification. Formative assessment during training, however, is individualised and tends not to show this level of difference; leading to a situation where failure in summative examinations comes as a surprise to both trainees and to training programme directors.
There is evidence that periodic assessment of trainees’ acquisition of core capabilities can help make balanced, informed judgements about readiness for progression. The move from a pass/fail categorisation to a yet/not yet categorisation when coupled with appropriate remedial measures can improve, both the validity, as well as the fairness of assessments.
The large magnitude of the differential in outcomes of high-stakes assessments cannot be fixed by tweaking current assessment systems. Instead, there needs to be a recognition that high-level of capabilities consistently demonstrated in the workplace need to play a role in judgements about progression. Failure to do so is unfair, wasteful of public finances, and in breach of the trust places by the public, in training safe and competent clinicians.
References
2. Improving Assessment: Further Guidance and Recommendations [Internet]. Academy of Medical Royal Colleges. 2016 [cited 2020 Nov 1]. Available from: https://www.aomrc.org.uk/reports-guidance/improving-assessment-guidance-recommendations/
3. Cost of Training 2017 [Internet]. Academy of Medical Royal Colleges. 2017 [cited 2020 Nov 1]. Available from: https://www.aomrc.org.uk/reports-guidance/cost-training-2017/
4. Participation E. Medical Act 1983 [Internet]. Statute Law Database; [cited 2020 Nov 1]. Available from: https://www.legislation.gov.uk/ukpga/1983/54/contents
5. Woolf K. Fair Training Pathways for All: Understanding Experiences of Progression - Final Report. :69.
6. Woolf K, Potts HWW, McManus IC. Ethnicity and academic performance in UK trained doctors and medical students: systematic review and meta-analysis. BMJ. 2011 Mar 8;342:d901.
7. NHS Staff Surveys - NHS Staff Survey Results [Internet]. [cited 2020 Jul 8]. Available from: https://www.nhsstaffsurveys.com/Page/1085/Latest-Results/NHS-Staff-Survey-Results/
8. Dave S, Chakravorty I, Menon G, Sidhu K, Bamrah JS, Mehta R. Differential Attainment in Summative Assessments within Postgraduate Medical Education & Training: Sushruta J Health Policy Opin [Internet]. 2020 Aug 9 [cited 2020 Sep 28];13(3). Available from: https://sushrutajnl.net/index.php/sushruta/article/view/89
9. Gibbs G, Simpson C. Conditions Under Which Assessment Supports Students’ Learning. :30.
10. Denholm-Price J, Page N, Williams N, Dourado L. Exploring differential attainment by assessment type in mathematics, chemistry and life sciences [Internet]. Presentation presented at: Horizons in STEM Higher Education Conference?: Making Connections and Sharing Pedagogy; 2019 Jul 3 [cited 2020 Oct 23]; Kingston-upon-Thames, U.K. Available from: https://eprints.kingston.ac.uk/id/eprint/43637/
11. Bloom BS, Airasian PW, Cruikshank KA, Mayer RE, Pintrich PR, Raths J, et al. A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives. Longman; 2001. 342 p.
12. Bennett RE. Formative assessment: a critical review. Assess Educ Princ Policy Pract. 2011 Feb 1;18(1):5–25.
13. Schuwirth LWT, Van der Vleuten CPM. Programmatic assessment: From assessment of learning to assessment for learning. Med Teach. 2011;33(6):478–85.
14. Holmboe ES, Call S, Ficalora RD. Milestones and Competency-Based Medical Education in Internal Medicine. JAMA Intern Med. 2016 Nov 1;176(11):1601–2.
15. Mountford-Zimdars A, Sanders J, Moore J, Sabri D, Jones S, Higham L. What can universities do to support all their students to progress successfully throughout their time at university? Perspect Policy Pract High Educ. 2017 Jul 3;21(2–3):101–10.
16. Generic professional capabilities framework [Internet]. [cited 2020 Jul 8]. Available from: https://www.gmc-uk.org/education/standards-guidance-and-curricula/standards-and-outcomes/generic-professional-capabilities-framework
17. Differential attainment in medical education and training | The BMJ [Internet]. [cited 2020 Nov 2]. Available from: https://www.bmj.com/content/368/bmj.m339
18. Good medical practice [Internet]. [cited 2020 Oct 27]. Available from: https://www.gmc-uk.org/ethical-guidance/ethical-guidance-for-doctors/good-medical-practice
19. Promoting excellence: standards for medical education and training. :51.
20. Coping with complexity: educating for capability | The BMJ [Internet]. [cited 2020 Oct 27]. Available from: https://www.bmj.com/content/323/7316/799
21. Slowther A, Lewando Hundt GA, Purkis J, Taylor R. Experiences of non-UK-qualified doctors working within the UK regulatory framework: a qualitative study. J R Soc Med. 2012 Apr;105(4):157–65.
22. evaluation-of-gmc-welcome-to-uk-practice---january-2019_pdf-79429900.pdf [Internet]. [cited 2020 Nov 2]. Available from: https://www.gmc-uk.org/-/media/documents/evaluation-of-gmc-welcome-to-uk-practice---january-2019_pdf-79429900.pdf
23. Hawkins RE, Margolis MJ, Durning SJ, Norcini JJ. Constructing a validity argument for the mini-Clinical Evaluation Exercise: a review of the research. Acad Med J Assoc Am Med Coll. 2010 Sep;85(9):1453–61.
24. Wolcott MD, Zeeman JM, Cox WC, McLaughlin JE. Using the multiple mini interview as an assessment strategy within the first year of a health professions curriculum. BMC Med Educ [Internet]. 2018 May 3 [cited 2020 Nov 2];18. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5934879/
25. Schuwirth LWT, van der Vleuten CPM. The use of progress testing. Perspect Med Educ. 2012 Mar 1;1(1):24–30.
26. Van Der Vleuten CPM, Schuwirth LWT, Driessen EW, Govaerts MJB, Heeneman S. Twelve Tips for programmatic assessment. Med Teach. 2015 Jul 3;37(7):641–6.
27. Rethans JJ, Sturmans F, Drop R, van der Vleuten C, Hobus P. Does competence of general practitioners predict their performance? Comparison between examination setting and actual practice. BMJ. 1991 Nov 30;303(6814):1377–80.
28. Epstein RM, Hundert EM. Defining and Assessing Professional Competence. JAMA. 2002 Jan 9;287(2):226–35.
29. Warwick C. How international medical graduates view their learning needs for UK GP training. Educ Prim Care. 2014 Jan 1;25(2):84–90.
30. Batalden P, Leach D, Swing S, Dreyfus H, Dreyfus S. General competencies and accreditation in graduate medical education. Health Aff Proj Hope. 2002 Oct;21(5):103–11.
31. Brittlebank A, Archer J, Longson D, Malik A, Bhugra DK. Workplace-Based Assessments in Psychiatry: Evaluation of a Whole Assessment System. Acad Psychiatry. 2013 Sep 1;37(5):301–7.
32. The State of physicianly training in the UK - 2019 report [Internet]. JRCPTB. 2019 [cited 2020 Nov 7]. Available from: https://www.jrcptb.org.uk/documents/state-physicianly-training-uk-2019-report
33. Academic support for the Assessment and Appraisal workstream ofHealth Education Englands review of the ARCP.pdf [Internet]. [cited 2020 Nov 7]. Available from: https://www.hee.nhs.uk/sites/default/files/documents/Academic%20support%20for%20the%20Assessment%20and%20Appraisal%20workstream%20ofHealth%20Education%20Englands%20review%20of%20the%20ARCP.pdf
34. Gold Guide - 8th Edition - Conference Of Postgraduate Medical Deans [Internet]. [cited 2020 Nov 7]. Available from: https://www.copmed.org.uk/gold-guide-8th-edition/
35. a_guide_to_entrustable_professional_activities.pdf [Internet]. [cited 2020 Nov 7]. Available from: https://www.rcr.ac.uk/sites/default/files/a_guide_to_entrustable_professional_activities.pdf
36. Roe, V., Patterson, F., Kerrin, M., Edwards, H. (2019) “What supported your success in training?” A qualitative exploration of the factors associated with an absence of an ethnic attainment gap in post-graduate specialty training. https://www.gmc-uk.org/-/media/documents/final-report-success-factors-in-training-211119-pdf_pdf-81634780.pdf [Internet] [cited 2020 Nov 10]
37. Future Doctor [Internet]. Health Education England. 2019 [cited 2020 Nov 7]. Available from: https://www.hee.nhs.uk/our-work/future-doctor