|
Blended programmatic assessment for competency based curricula
R Mahajan1, S Saiyad2, A Virk3, A Joshi4, T Singh5,
1 Department of Pharmacology, Adesh Institute of Medical Sciences and Research, Bathinda, Gujarat, India
2 Department of Physiology, Smt. NHL Municipal Medical College, Ahmedabad, Gujarat, India
3 Department of Community Medicine, Adesh Medical College and Hospital, Mohri, Kurukshetra, Haryana, India
4 Department of Pharmacology, Pramukhswami Medical College, Karamsad, Anand, Gujarat, India
5 Department of Medical Education, Sri Guru Ram Das University of Health Sciences, Amritsar, Punjab, India
Correspondence Address:
T Singh
Department of Medical Education, Sri Guru Ram Das University of Health Sciences, Amritsar, Punjab
India
Abstract
The uncertainty in all spheres of higher education due to the COVID-19 pandemic has had an unprecedented impact on teaching–learning and assessments in medical colleges across the globe. The conventional ways of assessment are now neither possible nor practical for certifying medical graduates. This has necessitated thoughtful considerations in making adjustments to the assessment system, with most institutions transitioning to online assessments that so far have remained underutilized. Programmatic assessment encourages the deliberate and longitudinal use of diverse assessment methods to maximize learning and assessment and at present can be utilized optimally as it ensures the collection of multiple low-stake assessment data which can be aggregated for high-stake pass/fail decisions by making use of every opportunity for formative feedback to improve performance. Though efforts have been made to introduce programmatic assessment in the competency-based undergraduate curriculum, transitioning to online assessment can be a potential opportunity if the basic tenets of programmatic assessment, choice of online assessment tools, strategies, good practices of online assessments and challenges are understood and explored explicitly for designing and implementing online assessments. This paper explores the possibility of introducing online assessment with face-to-face assessment and structuring a blended programmatic assessment in competency-based medical education.
How to cite this article:
Mahajan R, Saiyad S, Virk A, Joshi A, Singh T. Blended programmatic assessment for competency based curricula.J Postgrad Med 2021;67:18-23
|
How to cite this URL:
Mahajan R, Saiyad S, Virk A, Joshi A, Singh T. Blended programmatic assessment for competency based curricula. J Postgrad Med [serial online] 2021 [cited 2023 May 31 ];67:18-23
Available from: https://www.jpgmonline.com/text.asp?2021/67/1/18/308636 |
Full Text
Introduction
Effective assessment requires tasks that assess cognitive, psychomotor, and communication skills, while also assessing professional attributes. Students' learning depends largely on how they think they will be assessed, especially when embedded within “authentic” learning activities; hence, the assessment takes the shape of a strategic program for enriching teaching and learning in higher education, thus making way for the implementation of programmatic assessment.
As online education takes center stage, one question continues to linger in the minds of medical educators: “How do I know what my online students have learnt?” For which there indeed are no simple answers, just as there are not in face-to-face teaching–learning; however, with a little creativity and flexibility, an online learning environment can open up a host of new student assessment possibilities. And, just as with conventional courses, the answer lies in finding the right combination that works best for a particular course.
The rise of e-learning and web-based education to provide or improve educational experiences has led to a growth in the use of online assessment, which will further increase as the use of e-learning becomes more widespread. E-assessments when compared with conventional methods possess distinct characteristics due to the asynchronous nature of interactivity among the online participants. Therefore, teachers need to rethink the strategies of online pedagogy and develop effective assessment techniques that can support meaningful learning. Meaningful exchange and cooperation between students, peers, and teachers with a common purpose can create opportunities for ongoing and ample learner support. This can ultimately foster impactful engagement and deep learning in online education. Effective application of assessment in an online learning environment can offer a creative pedagogical strategy to facilitate such opportunities.
Are there any ways and means of introducing programmatic assessment through online modes for competency-based medical curriculum; if not fully at least partially? This review tries to explore the possibilities.
Programmatic Assessment and Competency-Based Assessment—The Linkage
Programmatic assessment (PA) is a program of assessment, wherein information about learner's competence and progress is collected, analyzed, and acted upon longitudinally. It shifts the focus away from individual tools and assessments, which serve as data points, which are used to provide feedback and mentoring but not for summative decision making. Decisions are made on the collection of data points, proportional to the stakes.[1] Since individual assessments are not used for high-stake decisions, it allows tools with lesser reliability but high educational impact to be used. It also allows lowering the stress of examinations. Delinking of decision making from each assessment by using it for formative purposes and making high-stake decisions on the collection of such data points can be considered the hallmark of PA.[2]
PA encompasses enough opportunities for assessment for learning and not merely depends upon the assessment of learning. Originally proposed by van der Vleuten and Schuwirth (2005),[3] PA assessment gels well with the concept of competency-based medical education (CBME). CBME advocates the longitudinal development of the learner by mastering competencies and by incorporating more formative assessment strategies, feedback, and continuous monitoring.[4] Principles of PA and Competency-Based Assessment (CBA) are congruent to a large extent.[5],[6],[7],[8] Under these circumstances, it would not be an overstatement to claim that CBME can be implemented properly, only when supplemented by PA [Figure 1].{Figure 1}
Online Assessment—Context
Online assessment can be liberally used for the assessment of knowledge through online multiple-choice questions, assessment of skills by using online objective-structured clinical examination or virtual patients, assessment of competency and performance by using e-logbooks and e-portfolios, and assessment of affective domain through contributions and discussion in online team project works.[9],[10]
As CBME is being adopted in the undergraduate medical curriculum in many institutes, the unparalleled fall-out of the COVID 19 pandemic has dealt a heavy blow on teaching–learning and assessments in medical colleges across the globe. The conventional ways of assessment based on high-stake clinical examination(s) are neither possible nor practical in the face of such adversity, as institutions today are faced with a dilemma of certifying medical graduates in the absence of summative examinations.[11]
The current testing times due to COVID-19 have necessitated exceptional modifications to the assessment system, with most institutions adopting e-learning and e-assessments as an indispensable mode of learning and assessment that so far remained underutilized. The triage of resources in medical education at all levels has recently been suggested as a policy to tide over the COVID-19 pandemic.[12] Even in the post-pandemic era, online modes of learning and assessment are going to stay in medical education. We just need to devise ways to tap their full potential.
In such challenging circumstances, a programme of assessment incorporating frequent online assessment activities is the most feasible option fit for the purpose of assessment. The programmatic approach to assessment encourages the deliberate and longitudinal use of diverse assessment methods to maximize learning and assessment. Such an assessment approach ensures the collection of multiple low-stake assessment data which can be aggregated for higher stake pass/fail decisions by making use of every opportunity for formative feedback to improve performance.[13] But can both CBA and PA be connected through online mode, via e-assessment strategies, particularly so in the face of the COVID 19 crisis? For this to happen, the basic principles of programmatic assessment, choice of online assessment methods, and their relevance to the curriculum need to be prioritized while developing e-assessment strategies for CBME[14] that can enrich the quality and range of student assessment [Table 1].{Table 1}
Online Assessment – Practical Considerations
We need to reconsider and re-review the assessment modalities in medical education looking into the current pandemic.[15] Though the prospects of introducing online assessment may look bright, some practical aspects do need consideration.
First, online assessment cannot be introduced for the entire undergraduate medical course. As experiential learning through patient contact is the focus of medical training, so is the assessment. Second, online teaching must precede online assessment, which literally means one need to develop modules for imparting online teaching and assessment in areas of feasibility. Third, online assessment must be aligned with competency objectives and online teaching-learning methods. Fourth, while implementing online assessment at the institutional level, the opportunities provided by the CBME curriculum, like self-directed learning and facility of virtual classrooms and skill labs must be tapped.
The emphasis within online assessment should be placed on the employment of multimodal tools for formative and summative assessments.[9] These tools must focus upon mastering clinical reasoning, problem-solving, and decision-making skills, enabling the transition to a competency-based curriculum. Effective assessment in a digital epoch offers a comprehensible foundation to the applicability of technology for assessment and feedback. A list of online tools specific for online assessment in different domains is given in [Table 2].{Table 2}
Blended Programmatic Assessment—The Buy-In
As stated earlier, introducing an entire program of online assessment in the CBME curriculum is neither feasible nor practical; nonetheless, it must be made part of the traditional PA of any institute and areas must be identified to develop online modules, thus making the introduction of online assessment feasible in that particular content/expertize area. Since online clinical assessment is a challenge, innovative tools have to be generated like using zoom video communications for virtual clinical examination. However, it is very difficult to replace the tests of clinical, communication, and practical skills with formats that do not allow the examiner to observe a student's performance directly.[16]
Considering all such arguments, the “Blended Programmatic Assessment” will definitely give leverage to the teachers to have a multimodal face-to-face assessment interspersed with online assessment activities [Figure 2].{Figure 2}
As individual assessments are never used for making final decisions in PA and low-stake assessments are very much the part of PA, the immense potential of online assessment can be used to plan low-stake assessments during the entire course. The whole program needs to be supplemented with opportunities for online feedback and reflections. The key task shall be the timely creation and monitoring of a balanced face-to-face and online assessment program that focuses on resource availability, faculty buy-in, and support of regulatory and licensing bodies, fortified with the provision of continuous mentoring and quality assurance. As such, “Blended Programmatic Assessment” will be a new dimension in the implementation of CBME curriculum for medical training.
Blended Programmatic Assessment—Fundamental Concerns
The statement, “Students can, with difficulty, escape from the effects of poor teaching, they cannot (by definition if they want to graduate) escape the effects of poor assessment” signifies the importance of effective assessment very well.[17] The broad perspectives laid down by van der Vleuten and Schuwirth (2005)[3] to defend different forms of assessment are in congruity with the aforementioned statement including blended programmatic assessment. These perspectives address two key concerns—intellectual and logistics.
Intellectual concerns
Given the importance assigned to students' assessment, it is critical to examine the fundamental concerns of assessment within the blended context, i.e., validity, reliability, acceptability, feasibility, and educational impact, to control the associated risks and realize its desirable outcomes. The model of the utility of assessment given by van der Vleuten implies that it is possible to compensate for any lacunae in assessment tools by their strengths.[18] The same model needs to be kept in mind when using online assessment tools in medical education. When we translate the concept to the entire program of blended assessment, it will be better to look at the utility or for that matter, individual characteristics like the validity and reliability of the blended PA itself, rather than looking at the utility/validity/reliability of individual online assessments and/or traditional assessment tools. This allows us to use a variety of online and face-to-face methods to assess competencies.[19]
Though content and construct validity are important, face validity needs to be accorded equal weightage, particularly when introducing online assessment for the first time to students who may be unfamiliar with its processes and may require reassurance. The content validity of online assessment can be enhanced by introducing additional features: like animations, videos, and sound (with headphones) added to online questions and “Hotspot” questions (which require students to place a mark anywhere on an image or diagram dragging labels directly over an image) etc.[20] The online nature and technological aspects of such assessments can significantly impact the authenticity in comparison to other forms of paper-based assessment media.[21]
The reliability of an assessment refers to the dependability of results. It can be seen in terms of marker consistency, temporal stability, and context specificity.[22] The reliability attributes of blended PA are detailed in [Table 3].{Table 3}
Acceptability of an online assessment method by all stakeholders, vis-a-vis teachers, students, institutions, and society is a key attribute of blended PA that influences the faculty motivation to use the method and enhances the student's trust of the results. Similarly, it is pertinent to recognize the various feasibility issues like affordability that govern the use of an assessment method. The acceptability of blended PA will definitely vary at the level of different stakeholders—both interclass and intraclass variations will be seen. Similarly, feasibility will be an issue across institutes, depending upon infrastructure.
For blended PA, the educational impact must be vouched for. During online learning and assessment, the ease of reverting to previous steps while using and being assessed on virtual platforms, thus providing an opportunity for corrections and skill enhancement must be weighed against the possibility of development of careless behavior and actual translation while dealing with real-life situations. The best way to ensure the utility of blended programmatic assessment is to have multiple online tests on multiple content areas by multiple examiners using multiple online tools [as given in [Table 2]] in multiple settings.
Logistic concerns
Just as the intellectual concerns warrant due consideration there are many legal, technical, and economic issues that require rigorous and comprehensive planning to cater to the design and implementation of quality assessment in e-learning.[23] thus having an impact on the blended programmatic assessment [Table 4].{Table 4}
Tips for implementation
Though COVID-19 pandemic has provided opportunities to the medical teachers to innovate and use hitherto underutilized online assessment methods and entwines them with existing assessment methods, and the opportunities thus provided can be utilized to develop a framework of “Blended Programmatic Assessment” particularly when in many countries competency-based medical education and programmatic assessment are newer concepts and are still in their introductory phase and are amenable to experimentation; the paradigm shift is never easy.
For implementing such a program, immense mobilization of resources will be required, besides investing time and efforts for faculty development and capacity building. Faculty development for implementing blended programmatic assessment will have a dual focus—”training in teaching technology” and “training in information technology.” We feel that initially, this will certainly add to the cost of the assessment but a sustained program will eventually lead to better allocation and utilization of resources, thus nullifying the enhanced cost factor. The tips as outlined in [Table 5] will help in introducing the “Blended Programmatic Assessment.”{Table 5}
Tools focusing more on designing realistic digital simulations should also be developed. Still further, with the advent of technology, augmented and virtual reality may be incorporated with online assessment for highly engaging experiences. Efforts must be made to make online assessment suitable for effective as well as skills domain, and using it along with traditional means so as to compute a blended program of assessment in the competency-based medical curriculum.
Conclusion
COVID pandemic has forced the use of online modes for teaching and assessment. Lockdown and closure of institutes even in the post-lockdown phase have made the use of online tools for teaching, learning, and assessment activities inevitable. Teaching–learning and assessment through this mode is likely to continue, though with lesser encounters, even during the postpandemic phase. A framework of “Blended Programmatic Assessment” will provide ways and means to the medical teachers and stakeholders for the seamless incorporation of online assessment with face-to-face traditional assessment in competency-based curricula.
Financial support and sponsorship
Nil.
Conflicts of interest
There are no conflicts of interest.
References
| 1 | Wilkinson TJ, Tweed MJ. Deconstructing programmatic assessment. Adv Med Educ Pract 2018;9:191-7. |
| 2 | Singh T, Gupta P, Singh D. Student assessment. In Principles of Medical Education. 5th ed. N Delhi: Jaypee Brothers; 2021. p 69-80. |
| 3 | van der Vleuten CPM, Schuwirth LWT. Assessing professional competence: From methods to programs. Med Educ 2005;39:309-17. |
| 4 | Frank JR, Snell L, Englander R, Holmboe ES. Implementing competency-based medical education: Moving forward. Med Teach 2017;39:568-73. |
| 5 | Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR. The role of assessment in competency-based medical education. Med Teach 2010;32:676-82. |
| 6 | Schuwirth LWT, van der Vleuten CPM. Programmatic assessment: From assessment of learning to assessment for learning. Med Teach 2011;33:478-85. |
| 7 | van der Vleuten CPM, Schuwirth LWT, Driessen EW, Govaerts MJB, Heeneman S. Twelve tips for programmatic assessment. Med Teach 2015;37:641-6. |
| 8 | Torre DM, Schuwirth LWT, van der Vleuten CPM. Theoretical considerations on programmatic assessment. Med Teach 2010;42:213-20. |
| 9 | Ellaway R, Masters K. AMEE guide 32: E-learning in medical education Part 1: Learning, teaching and assessment. Med Teach 2008;30:455-73. |
| 10 | Gordon JA, Campbell CM. The role of ePortfolios in supporting continuing professional development in practice. Med Teach 2013;35:287-94. |
| 11 | Sahi P, Mishra D, Singh T. Medical education amid the COVID-19 pandemic. Indian Pediatr 2020;57:652-7. |
| 12 | Tolsgaard MG, Cleland J, Wilkinson T, Ellaway RH. How we make choices and sacrifices in medical education during the COVID-19 pandemic. Med Teach 2020;42:741-3. |
| 13 | Humphrey-Murto S, Wood TJ, Ross S, Tavares W, Kvern B, Sidhu R, et al. Assessment pearls for competency-based medical education. J Grad Med Educ 2017;9:688-91. |
| 14 | Bala L, van der Vleuten C, Freeman A, Torre D, Heeneman S, Sam AH. COVID-19 and programmatic assessment. Clin Teach 2020;17:420-2. |
| 15 | Sabzwari S. Rethinking assessment in medical education in the time of COVID-19. MedEdPublish 2020;9:80. |
| 16 | Fuller R, Joynes V, Cooper J, Boursicot K, Roberts T. Could COVID-19 be our 'There is no alternative' (TINA) opportunity to enhance assessment? Med Teach 2020;42:781-6. |
| 17 | Boud D. Assessment and learning: Contradictory or complimentary. In: Knight P, editor. Assessment for Learning in Higher Education. London: Kogan Page; 1995. p. 35-48. |
| 18 | van der Vleuten CPM. The assessment of professional competence: Developments, research and practical implications. Adv Health Sci Educ Theory Pract 1996;1:41-67. |
| 19 | Singh T. Student assessment: Moving over to programmatic assessment. Int J Appl Basic Med Res 2016;6:149-50. |
| 20 | Dennick R, Wilkinson S, Purcell N. Online eAssessment: AMEE Guide no. 39. Med Teach 2009;31:192-206. |
| 21 | Sim G, Holifield P, Brown M. Implementation of computer assisted assessment: Lessons from the literature. ALT-J: Res Learn Technol 2004;12:215-29. |
| 22 | Singh T. Basics of assessment. In: Singh T, Anshu, editors. Principles of Assessment in Medical Education. 1st ed. New Delhi: Jaypee Publishers; 2012. p. 1-13. |
| 23 | Raaheim A, Mathiassen K, Moen V, Lona I, Gynnild V, Bunæs BR, et al. Digital assessment – how does it challenge local practices and national law? A Norwegian case study. Eur J Higher Educ 2019;9:219-31. |
|
