|
Acceptability and feasibility of mini-clinical evaluation exercise as a formative assessment tool for workplace-based assessment for surgical postgraduate students
MK Joshi1, T Singh2, DK Badyal3,
1 Department of Surgical Disciplines, All India Institute of Medical Sciences, New Delhi, India
2 Department of Paediatrics, CMCL-FAIMER Regional Institute, Medical Council of India Nodal Centre for Faculty Development, Christian Medical College and Hospital, Ludhiana, Punjab, India
3 Department of Pharmacology, CMCL-FAIMER Regional Institute, Medical Council of India Nodal Centre for Faculty Development, Christian Medical College and Hospital, Ludhiana, Punjab, India
Correspondence Address:
M K Joshi
Department of Surgical Disciplines, All India Institute of Medical Sciences, New Delhi
India
Abstract
Background: Despite an increasing emphasis on workplace-based assessment (WPBA) during medical training, the existing assessment system largely relies on summative assessment while formative assessment is less valued. Various tools have been described for WPBA, mini-clinical evaluation exercise (mini-CEX) being one of them. Mini-CEX is well accepted in Western countries, however, reports of its use in India are scarce. We conducted this study to assess acceptability and feasibility of mini-CEX as a formative assessment tool for WPBA of surgical postgraduate students in an Indian setting. Methods: Faculty members and 2nd year surgical residents were sensitized toward mini-CEX and requisite numbers of exercises were conducted. The difficulties during conduction of these exercises were identified, recorded, and appropriate measures were taken to address them. At the conclusion, the opinion of residents and faculty members regarding their experience with mini-CEX was taken using a questionnaire. The results were analyzed using simple statistical tools. Results: Nine faculty members out of 11 approached participated in the study (81.8%). All 16 2nd year postgraduate surgical residents participated (100%). Sixty mini-CEX were conducted over 7 months. Each resident underwent 3–5 encounters. The mean time taken by the assessor for observation was 12.3 min (8–30 min) while the mean feedback time was 4.2 min (3–10 min). The faculty reported good overall satisfaction with mini-CEX and found it acceptable as a formative assessment tool. Three faculty members (33.3%) reported mini-CEX as more time-consuming while 2 (22.2%) found it difficult to carry the exercises often. All residents accepted mini-CEX and most of them reported good to high satisfaction with the exercises conducted. Conclusions: Mini-CEX is well accepted by residents and faculty as a formative assessment tool. It is feasible to utilize mini-CEX for WPBA of postgraduate students of surgery.
How to cite this article:
Joshi M K, Singh T, Badyal D K. Acceptability and feasibility of mini-clinical evaluation exercise as a formative assessment tool for workplace-based assessment for surgical postgraduate students.J Postgrad Med 2017;63:100-105
|
How to cite this URL:
Joshi M K, Singh T, Badyal D K. Acceptability and feasibility of mini-clinical evaluation exercise as a formative assessment tool for workplace-based assessment for surgical postgraduate students. J Postgrad Med [serial online] 2017 [cited 2023 Sep 28 ];63:100-105
Available from: https://www.jpgmonline.com/text.asp?2017/63/2/100/201411 |
Full Text
Introduction
Workplace-based assessment (WPBA) has emerged as an attractive concept in the field of medical education. It has many advantages as the assessment is conducted under direct supervision in a real environment, thus resembling the day-to-day clinical encounters more closely than traditional ways of assessment. It focuses more on actual performance rather than mere knowledge. Although it can be used both for formative and summative assessment, the strength lies in its use for formative assessment to promote learning. Since its introduction many tools have been developed and added to armamentarium of WPBA, namely, mini-clinical evaluation exercise (mini-CEX), direct observation of procedural skills, case-based discussions, clinical work samples, and multi-source feedback.
Mini-CEX is a useful tool that can be used for formative assessment and hence for WPBA. First introduced by the American Board of Internal Medicine in 1995 for assessment of postgraduate students, mini-CEX has been designed to provide feedback following direct observation of a clinical encounter.[1] The assessor scores the performance using a generic form that is applicable to most clinical scenarios which is followed by a feedback session. The duration of encounters varies from 10 to 15 min including the feedback. Thus, mini-CEX appears promising as it is conducted in a real environment, in a short time with feedback incorporated. The traditional way of assessment does not allow opportunity to provide feedback and thereby avoids the use of this important influence on learning. Mini-CEX is being used extensively for formative assessment in West [2],[3],[4],[5] but has been used sparingly in India.[6],[7],[8],[9],[10],[11]
Although appealing, WPBA has not enjoyed a uniform level of acceptability and success,[8] and there are concerns regarding the feasibility of carrying out multiple assessments amidst busy clinical schedules by multiple assessors to ensure reliability and simultaneously justifying the formative nature of the exercises being conducted.[12],[13]
We attempted to assess the feasibility of carrying out mini-CEX as a formative assessment tool in Indian setting and also tried to find out whether it is acceptable to our students and faculty in surgery.
Methods
The study was carried out at a Government Medical College and Tertiary Care Teaching Hospital after approval from the ethical clearance committee and consent from the participating faculty and residents. Purposeful sampling including all 16 2nd year postgraduate surgical residents (total population sampling) was done. Focus group technique was employed to evaluate the shortcomings of the existing system of internal assessment (IA) and to sensitize the residents to mini-CEX. Two focus group discussions (FGDs) were conducted for residents. In the first FGD, the views of the residents regarding the existing system of IA were taken including the responses to their reaction to the marks displayed after the assessment. The advantages and disadvantages of the existing system were discussed in detail in a friendly atmosphere. They were encouraged to suggest alternatives to offset the disadvantages of the existing system of IA. Mini-CEX was then introduced as a formative assessment tool that could overcome some of the disadvantages inherent to the traditional way of IA. At the end of the first FGD, the students were provided with printed material on mini-CEX to help it understand better.
In the second FGD, residents were exposed to mini-CEX in detail and were told how the exercises would be conducted and what is expected out of them. The mini-CEX evaluation form was discussed in detail. Simultaneously, one to one discussion was carried out with the faculty members to sensitize them toward mini-CEX with emphasis on how to provide feedback. They were also provided with the printed material on mini-CEX, and the mini-CEX evaluation form was discussed in detail with them.
Informal feedback was taken from the students and faculty members at regular intervals to assess difficulties in running the project, and appropriate measures were taken to address them. After achieving the requisite number of exercises, feedback from residents and faculty members was taken using a predesigned and validated questionnaire on a five-point Likert scale [Figure 1] and [Figure 2]. The questionnaire was used to generate responses to assess the acceptability of mini-CEX as a formative assessment tool for WPBA.{Figure 1}{Figure 2}
Results
The results of the study are as follows.
Analysis of first focus group discussion
The responses of postgraduate students on display of performance following (conventional) IA varied from “works as a stimulant to study hard” to the feeling of “unpleasantness, depression and being worthless.” Many of them admitted being “surprised with the result” as they thought that the “case presentation” went fine and they wanted to know “what went wrong” to which they had no opportunity.
The following shortcomings of the existing conventional IA were identified during the conversation.
The assessment is grossly inadequate as it is conducted once in 6 months as a single case presentationThere is chance of getting an easy case and performing well by the candidate and vice versaThere is chance/luck of getting a strict or lenient examiner that may affect the scoreThere is no provision of feedback to help the candidate know about his performance.
Feasibility to conduct mini-clinical evaluation exercise
A total of 60 mini-CEX were conducted by 9 faculty members for 16 2nd year postgraduate students of the department over the period of 7 months. Each resident underwent at least three and a maximum of five encounters during this period. This amounted to nearly seven exercises conducted by each faculty member over 7 months, i.e., one mini-CEX per month per faculty. This was therefore not difficult for the faculty members and does not seem to increase their workload significantly. The initial plan of distribution of students among faculty members for mini-CEX was found not working well as the students or faculties were not available on same day and time because of their professional commitments. The distribution was then rearranged to ensure that the student remained available to the assigned faculty member.
No extra resources were required to carry out the exercises except printed evaluation sheets. Despite difficulties, mainly limited to the motivation of faculty to continue taking mini-CEX and some unforeseen circumstances such as attrition of few faculty members and therefore their unavailability for the project, it was observed that it is feasible to conduct mini-CEX without stretching the resources.
Complexity of mini-clinical evaluation exercise conducted
Forty-four (73.3%) clinical encounters were considered of moderate complexity by faculty while the number of low- and high-complexity clinical encounters was 12 (20%) and 4 (6.67%), respectively.
Competencies assessed
The competencies assessed during mini-CEX included interviewing skills 15 (25%), physical examination skills 15 (25%), counseling skills 4 (6.7%), clinical judgment 2 (3.3%), and an overall impression 24 (40%). None of the residents was assessed on professionalism alone [Table 1].{Table 1}
Time taken for observation and feedback
The mean time taken by the assessor for observation was 12.3 min (8–30 min) while the mean feedback time was 4.2 min (3–10 min).
Acceptability of mini-clinical evaluation exercise by faculty
All faculty members found mini-CEX acceptable as a formative assessment tool. Most of them expressed that it can be used to supplement the traditional way of assessment. The feedback from the faculty regarding their experience with mini-CEX is presented in [Table 1].
As is reflected in the table, some of the faculty members reported that mini-CEX was more time-consuming than initially thought. Although all of them agreed that the exercises were easy to carry out and sample wider areas as compared to the traditional IA, some found it difficult to conduct the exercises often. The following are some of the favorable comments in the faculty feedback questionnaire regarding their experience with Mini-CEX:
It is better way to assess students than traditional wayIt is interesting to interact with different students frequently.
However, one faculty member found mini-CEX similar to the case presentation by residents during routine ward and emergency rounds.
Acceptability of mini-clinical evaluation exercise by residents
All residents well-accepted mini-CEX for IA and most of them reported good to high satisfaction with the exercises conducted. A few of them liked the relaxed environment during the exercises, but a few of them found that being supervised by faculty while they were interviewing/examining the patient made them nervous. All were convinced that mini-CEX could be incorporated for IA. Some of them came up with innovative ideas to further enhance the quality of IA using mini-CEX. The feedback from the students is summarized in [Table 2].{Table 2}
In the words of residents:
Mini-CEX is a great opportunity to get assessed by different examiners and know the shortcomings from time to time so that we can improve graduallyThis type of assessment is better than our exam where we are assessed for our learning of 3 years in just 1 day on few casesIt was more relaxed than routine exams, which was inherently different for our set upThe exercises appeared very casualThe exercises should have been more frequent and at the end of 6 months should have been graded by a point-based system as is done in IITs.
Performance of residents
There was consistent improvement in the performance of the residents with the conduction of the exercises, however, we did not observe any significant differences in their performances when different assessors assessed them.
Discussion
We observed that conducting mini-CEX is feasible in the existing setting without stretching the resources, this has also been observed by other authors.[6],[9],[10] We did not had any difficulty in ensuring the workforce, clinical material, and stationary required to carry out the exercises. The biggest hurdle was the motivation of the faculty members for participating in the study. Two of the senior faculty members expressed their inability to participate citing their unavailability due to other professional commitments. This was despite the assurance that mini-CEX was not time-consuming and can seamlessly be incorporated in their existing schedule. Rest of the faculty accepted to participate although there was some difficulty initially to keep their motivational levels up. Once the faculty became comfortable with mini-CEX, the task became progressively easier.
The exercises conducted during this study were not scheduled with respect to time, considering the first exposure of mini-CEX and also to provide faculty with the liberty to complete the exercises at their own comfort. This was found to be a limiting factor in conducting the desired number of exercises in time. We learned that it is better to schedule the exercises at the outset for their smooth conduction. Mini-CEX was well acceptable to the faculty and students alike despite few apprehensions. The primary concern of some faculty members that mini-CEX requires more time was apparent as in few exercises the time taken was nearly 30 min. This reflects inexperience of the faculty members with mini-CEX as they tried to assess multiple skills at the same time and considered it comparable to a directly supervised traditional clinical case presentation. This can be overcome by providing the faculty with a small training on mini-CEX. Although we deliberated them about mini-CEX at the outset, more training in conducting the exercises would have offset this apprehension. In a review on perceptions on WPBA, improper training of assessors has been cited as one of the major shortcomings in acceptance of WPBA.[14] Other authors have also expressed the importance of proper training of assessors.[6],[15],[16],[17] Another concern of faculty regarding difficulty in conducting the exercises often also reflects the low motivation on their part. The time taken for conduction of each exercise was small, so we believe that this apprehension is more a mental block than a real problem. Massie and Ali have also cited this as one of the shortcomings that affect the acceptance of WPBA adversely.[14] In a system, once some form of WPBA is firmly incorporated, and faculty assignments are well laid down, there will be less inertia in conducting the exercises.
The students accepted the tool very well. This was reflected by their high level of satisfaction with the exercises performed. Some of our residents reported anxiety as the teacher was watching them during their performance. This is not surprising as in the present system of assessment; there is usually dissociation between the examiner and the student while the student is actually “performing” on the patient. This has also been observed by other authors [6],[18] except Goel and Singh.[10] With more exposure on 'being observed' and the recognition that it enhances learning, we have sincere hope that this will not disturb the student as also conveyed by Malhotra et al.[7]
Mini-CEX provides a unique opportunity to discuss the weaknesses of the candidate identified at the conclusion of the exercise. This empowers them for consistent improvement in the weak areas so identified. We observed consistent improvement in the performance of our residents over the period when mini-CEX was conducted.
This is the first study on the use of mini-CEX for formative assessment of postgraduate students of surgery form India. The present study is also unique as we could overcome some of the shortcomings reported by other Indian authors on mini-CEX, like the varying status of the assessors,[6] unable to complete the requisite number of exercises,[6],[11] exercises conducted in limited settings,[6],[11] less number of exercises,[9] each candidate examined by only one examiner,[9] and evaluation of a single competency.[11] The findings from this small study may not be generalized and more similar studies need to be conducted to support the evidence generated from this initial work.
Conclusions
Mini-CEX is acceptable to the faculty members and residents as a tool for formative assessment for WPBA. It is feasible to carry out mini-CEX without stretching the existing infrastructure and humankind. Mini-CEX has a potential to supplement or replace the existing system of IA.
Acknowledgment
The authors would like to thank Faculty colleagues and participating students from UCMS, Faculty of CMCL-FAIMER, and CMCL-FAIMER fellows of 2014 batch.
Financial support and sponsorship
Nil.
Conflicts of interest
There are no conflicts of interest.
References
| 1 | Norcini JJ, Blank LL, Arnold GK, Kimball HR. The mini-CEX (clinical evaluation exercise): A preliminary investigation. Ann Intern Med 1995;123:795-9. |
| 2 | Weston PS, Smith CA. The use of mini-CEX in UK foundation training six years following its introduction: Lessons still to be learned and the benefit of formal teaching regarding its utility. Med Teach 2014;36:155-63. |
| 3 | Kogan JR, Bellini LM, Shea JA. Implementation of the mini-CEX to evaluate medical students' clinical skills. Acad Med 2002;77:1156-7. |
| 4 | Pelgrim EA, Kramer AW, Mokkink HG, van den Elsen L, Grol RP, van der Vleuten CP. In-training assessment using direct observation of single-patient encounters: A literature review. Adv Health Sci Educ Theory Pract 2011;16:131-42. |
| 5 | Oliphant R, Drummond R, Jackson A, Ross J, Blackhall V, Ridley-Fink E, et al. The use of mini-CEX in UK foundation training six years following its introduction: Lessons still to be learned and the benefit of formal teaching regarding its utility. Med Teach 2014;36:916. |
| 6 | Singh T, Sharma M. Mini-clinical examination (CEX) as a tool for formative assessment. Natl Med J India 2010;23:100-2. |
| 7 | Malhotra S, Hatala R, Courneya CA. Internal medicine residents' perceptions of the mini-clinical evaluation exercise. Med Teach 2008;30:414-9. |
| 8 | Behre BP. Introduction of mini clinical evaluation exercise (MINI-CEX) as an evaluation tool in psychiatry residency: A comparative study in JNMC, Sawangi, Wardha. JHSE 2014;1:9-12. |
| 9 | Behere R. Introduction of mini-CEX in undergraduate dental education in India. Educ Health (Abingdon) 2014;27:262-8. |
| 10 | Goel A, Singh T. The usefulness of mini clinical evaluation exercise as a learning tool in different pediatric clinical settings. Int J Appl Basic Med Res 2015;5 Suppl 1:S32-4. |
| 11 | Chandra M. Initiating formative assessment of postgraduate students in obstetrics and gynecology. Natl J Integr Res Med 2013;138:476-81. |
| 12 | Fokkema JP, Teunissen PW, Westerman M, van der Lee N, van der Vleuten CP, Scherpbier AJ, et al. Exploration of perceived effects of innovations in postgraduate medical education. Med Educ 2013;47:271-81. |
| 13 | Fokkema J, Teunissen PW. Assessing the assessment of interventions: We're not there yet. Med Educ 2013;47:954-6. |
| 14 | Massie J, Ali JM. Workplace-based assessment: A review of user perceptions and strategies to address the identified shortcomings. Adv Health Sci Educ Theory Pract 2016;21:455-73. |
| 15 | Foley T, Walsh E, Sweeney C, James M, Maher B, O'Flynn S. Training the assessors: A mini-CEX workshop for GPs who assess undergraduate medical students. Educ Prim Care 2015;26:446-7. |
| 16 | Daelmans HE, Mak-van der Vossen MC, Croiset G, Kusurkar RA. What difficulties do faculty members face when conducting workplace-based assessments in undergraduate clerkships? Int J Med Educ 2016;7:19-24. |
| 17 | Liao KC, Pu SJ, Liu MS, Yang CW, Kuo HP. Development and implementation of a mini-clinical evaluation exercise (mini-CEX) program to assess the clinical competencies of internal medicine residents: From faculty development to curriculum evaluation. BMC Med Educ 2013;13:31. |
| 18 | Weller JM, Jones A, Merry AF, Jolly B, Saunders D. Investigation of trainee and specialist reactions to the mini-clinical evaluation exercise in anaesthesia: Implications for implementation. Br J Anaesth 2009;103:524-30. |
|
