Assessing domain based teaching skills of postgraduates after a workshop on effective lesson plan

Introduction: Despite having to spend a substantial amount of time in teaching undergraduates, postgraduate students do not receive any formal instructions on teaching skills. We undertook a project which aimed at imparting basic teaching skills to postgraduates in the form of formulating a good lesson plan and a study to see whether there was an increase in their confidence levels. Three assessors independently assessed the postgraduates’ use of the domains of a lesson plan as a means of data triangulation. Objectives: 1. Train postgraduate students in constructing a good lesson plan. 2. Evaluate the outcomes of this educational program. Material and Methods: The change in confidence levels of postgraduates was assessed through a retro-pre questionnaire and a check list was used by three independent faculty to assess the use of the domains of a lesson plan and to judge the overall effectivity of teaching. Kappa statistics was used to determine the agreement between the three assessors. Results: A significant improvement in confidence levels of the postgraduates in use of the domains of the lesson plan was observed. The assessors found 70-100% use of the domains of a lesson plan which resulted in effective teaching. The assessors also identified some domains which had not been addressed adequately by the postgraduates. Conclusion: Our study demonstrated an improvement in confidence levels of the postgraduates and assessor evaluation confirmed the improvement in their teaching skills .The road ahead would involve retraining the participants in concepts that they found difficult to emulate, include more postgraduates into the program and obtain student perspective on the teaching skills of postgraduates.


Introduction
Postgraduates in their training years have an added responsibility of teaching medical students besides their clinical responsibilities. 1-3 Studies have estimated that postgraduate trainees spend up to 20% of their time on teaching activities regardless of their future plans to continue as medical educators. 1,2 Despite this they do not receive any formal instruction on teaching skills and how to teach effectively. 4,5 It has been shown that despite the lack of good teaching skills, practicing clinicians (and trainee postgraduates) are involved in undergraduate teaching although they have not had any formal training for the same. Some employ the skills they acquired from their own experiences as students, some by observation and others develop their skills through trial and error. [6][7][8][9] Good clinical skills alone are not enough to make a good teacher. The assumption that "the good physician" is synonymous with "the effective teacher" is therefore wrong. Understanding educational theories and how knowledge gets transferred to the learner is important towards acquiring effective teaching skills without which a good physician would not make a good teacher. 7,10 With this in mind we undertook a project which illustrated the use of the domains of a lesson plan as a means of imparting basic teaching skills to our postgraduate trainees. Subsequently we assessed their confidence levels in the use of the domains of a lesson plan before and after the project. Three assessors using a check list for the same domains judged their performance which served as a means of data triangulation.
Objectives 1. Train postgraduate students in constructing a good lesson plan as a means of improving their teaching skills. 2. Evaluate the outcomes of this educational program at Kirkpatrick's level one (reaction) and two (learning).
To achieve the second objective and as a means of evaluating this educational program we aimed to answer the evaluation questions given below: 1. How has this course affected the confidence levels of participants as regards use of the domains of a lesson plan? (Self-assessment by participants -Reaction-Kirkpatrick's level one).
2. How has this course affected the participants use of the domains of a lesson plan as perceived by the assessors (Assessor assessment -Learning -Kirkpatrick's level two).

Material and Methods
This is a prospective interventional study done during a total period of 6 months with a sample size of 25 participants. Ethical approval was obtained and the study was done as a part of the Medical Education Unit activities at Peoples' Education Society (PES) Institute of Medical Sciences, Kuppam, Andhra Pradesh, India. The whole project was explained to the trainee postgraduates' during their orientation program.
25 residents were chosen with almost equal representation from all specialties. It was decided that the other residents would also undergo this program in batches and not all at once so as to ensure the smooth running of the respective departmental activities and clinical work .Each of the participants signed an informed consent for inclusion in the study. (APPENDIX 1) Although postgraduate trainees traditionally conduct demonstrations, tutorials and clinic teaching the number of students in each batch attending a particular demonstration or tutorial comprises about 30-35 students and therefore a lesson plan is as relevant as in a large lecture class.

Plenary sessions
The intervention was in the form of two plenary sessions on effective large group teaching conducted by members of the MEU who had done the Basic Workshop c at the MCI approved centre at Christian Medical College Vellore. One explained the domains or components of an effective lesson plan and the second gave instructions on creating an effective power point presentation. Subsequently a hand out containing the salient points of the two sessions was provided to the participants.
The plenary for the lesson plan was based on explaining the following domains involvedin developing a lesson plan: 1. Structuring a lesson: The postgraduates were explained that structuring a lesson made the lesson more effective 2. How to begin a lesson: (Anticipatory set) -Arousing interest in the topic of study and emphasizing the importance of the topic were effective ways of arousing the interest of the students. Such methods ensured the attention of the audience. 3. How to set objectives: Settings objectives for a particular lesson gave teaching the needed direction .Making the objectives clear enabled the students to understand what they are expected to know or learn. 4. How to plan a lesson: After setting clear objectives, relevant content or subject matter and a logical sequence of presentation were essential components of a lesson plan. 5. How to present a lesson: While presenting the lesson, quoting examples, varying the pace, relating personal experience, using humor made the presentation more effective and appealing to the students. 6. How to ensure student participation. Ensuring student participation is the best way of judging whether knowledge transfer actually takes place. Allowing time for questions, answering questions, asking questions and verbally rewarding correct answers or efforts to answer serve as effective means of student participation. 7. How to use A-V aids: The second plenary dealt with tips on use of A-V Aids particularly the power point presentation. Using visible fonts, adequate contrast and eye friendly colors, avoiding overcrowding the presentation slides and using keywords instead of sentences were explained as important means of an effective power point presentation 8. How to close a lesson: Effective summarization, giving take home messages, giving instructions for the next class and emphasizing must know areas serve as good means to close a lesson.

Practice sessions
The practice sessions were conducted on five consecutive days during the afternoon sessions Each day five residents chose a topic and designed a lesson plan based on the domains provided in the hand out .Each of them then gave his/her "lecture" and suggestions were invited by the peers on improving each domain of the lesson plan. During these sessions content experts were present to authenticate the subject matter in these practice lecture sessions.

Evaluation session
A month later, each participant was asked to conduct a large group teaching. During this lecture he\she was judged by three separate assessors. The assessors used a check list. (APPENDIX III) A content expert was present at each of these sessions to authenticate the educational content of the lecture. For example if it was a physiology postgraduate trainee was giving his lecture, in addition to the three assessors, a faculty from the physiology department was present in order to verify the accuracy of the subject matter. In short the content expert judged the content matter and the three assessors assessed the teaching methodology using a checklist (APPENDIX III).
The check list verified the postgraduates use or non-use of all the components of a lesson plan. The points checked were those marked in italics and bold when describing a lesson plan in the plenary sessions.

Data Collection
Source of data collection 1. Participants\ 25 Postgraduate trainees. 2. Three independent assessors. Tools for data collection 1. Retropre-Questionnare for postgraduate trainees to assess change in confidence levels of using the domains of a lesson plan (APPENDIX II). 2. Check list for assessors to assess the postgraduates' actual use of the domains while teaching and overall teaching skills or effectiveness (APPENDIX III).

Data Analysis
Data analyzed gave the following answers: 1. Change in confidence levels as judged by the participants in each domain. 2. Percentage application of each domain in the lecture as judged by assessors. 3. Agreement between the three assessors using assessor one as reference.
Change in confidence: The confidence evaluation of the participants was done by a proportion test which was a percentage evaluation of the post intervention scores. This particular test has been used here to compare the percentages in categories within specific domains. The overall comparison of the percentages of participants "who understand the concept, can use the concept" etc has been compared within the domains such as "structuring a lesson, how to begin" etc. The p value of each comparison shows that there is a significant increase in the confidence level of the participants Results 1. Proportion test showed that there is a statistically significant increase in the confidence level of the postgraduates in all the specified domains of a lesson plan (Table 1). 2. Researchers found a 70%-100% application of the domains by the participants (Table 2 and Fig. 1) and all found the post project teaching skills of the postgraduates to be effective. There were some domains that had not been addressed by the participants (marked in yellow in Table 2). And these were 'asking questions', 'varying pace' and 'using key words instead of sentences'. So the researchers feel that these areas that should be focused on when retraining the postgraduates.

Fig. 1: Researchers perspective -Use of the different domains of a lesson plan by residents
3. There is a significant agreement (p>0.001) between the assessment done by the three assessors (Table 3.1, 3.2) Table 3: Agreement between the assessors as regards effectiveness of the teaching skills of residents Assessor 1 has been taken as the reference category and Kappa statistics has been used for analysis). Effective was defined as use of 5 out of the 7 domains. Very effective was defined as use of all 7 domains.  Discussion Trainee postgraduates contribute significantly to the quality of undergraduate medical training and it is assumed that by participating in the process, they also improve their own teaching abilities. It is of importance therefore that these postgraduates receive some formal training which aims at improving their teaching skills. [5][6][7][8][9] Our project aimed at imparting teaching skills to the trainee postgraduates and assess their confidence levels after the project. The training given was in the form of a structured lesson plan .Giving structure to the lesson plan encouraged them to think about the various domains of teaching namely arousing interest in the topic, clarifying doubts, encouraging student participation etc. These were concepts that they had not been aware. After the project they expressed (by way of a questionnaire response) an increased level of confidence is addressing the above domains which would result in effective teaching. This was also endorsed by the assessors who carried out the project. Studies which focus on how postgraduates perceive their role as teachers have shown that they enjoyed teaching and felt that there was a need for them to develop their basic teaching skills They also felt that training workshops, constructive evaluation and feedback from senior teachers and experts were needed to improve their teaching abilities. 11 Randomized control trials of "residents or postgraduate trainees as teachers" curriculum have revealed that compared with control resident trainees, the intervention residents expressed greater enthusiasm for teaching, more learner centered and empathic approaches, and a richer understanding of teaching principles and skills. Most intervention residents wanted to continue teaching during and after training. 12 Thus it seems that there is a universal agreement that trainee postgraduates accept their roles as teachers willingly and are receptive to any instructions, workshops, interactive sessions that could improve their teaching skills.
Our study further assessed whether the project had actually resulted in improvement in teaching skills of the postgraduates .Three independent assessors found that the they had made use of the knowledge imparted to them by the trainers and had structured their lesson as elaborated in the lesson plan which constituted a definite improvement in the teaching skills. We could not find any studies which did any such assessment. Our study also found which domains of a lesson plan were difficult to emulate which would mean that more training in such areas is required.
In order to take such studies to further Kirkpatrick's levels namely whether these training of postgraduates actually proves beneficial to the students, studies need to assess the students perspective of improvement in the postgraduates' teaching skills. Further, more studies need to see whether knowledge transfer has been achieved and academic performance of the undergraduates has improved.
Future studies should try to gather the student responses of the quality of teaching before and after training program for postgraduates. The students being the ultimate beneficiaries and the most important stakeholders in any educational project should be able to give their responses as regards an educational program .This would be a good way of data triangulation to judge the effectiveness of a postgraduates' training program.

Conclusion
With continued emphasis on use of Education Technologies for faculty, it is necessary to address the issue of training the postgraduates who are the faculty of tomorrow.
Our study demonstrated the willingness and enthusiasm of trainee postgraduates towards the above observation and a subsequent increase in confidence levels. Giving structure to the lesson improved their confidence in the various domains of the structured lesson. Assessor evaluation helped to identify the concepts that the residents found difficult to emulate. The road ahead would involve including more residents into this program, also including various other teaching techniques, assessing student perspectives and evaluating the actual knowledge transfer and academic performance of students who have been taught by trainee postgraduates.