101 – Introduction of Team Based Learning in Anatomy Course and Student’s Perception of its Usefulness
Trinity School of Medicine
Background: Traditional didactic lectures are an essential portion of most curricula, but this type of activity does not provide engagement of students into the active learning process. We have attempted to implement team based learning (TBL) in our school to develop interpersonal and group interaction skills in our students, and also to prepare them for the lifelong learning and to help them to master the key course concepts.
Methods: The TBL session plans were posted at the students’ portal prior to the beginning of the semester. In the beginning of the semester the students were separated into the groups and were informed about the dates of upcoming session, the amount of the material that should be prepared/learned by a student prior to each session. The pre-reading assignment was given to the students.
A total of 50 students of term 2 were separated into teams of 5-6. After each TBL session the students were completing a survey with 10 questions, where they had to rate the session on the basis of the 5 point Likert-type scale. The responses were anonymous.
Results: after the analysis of the surveys completed by students, it was identified that majority of the students (89%) were satisfied with the TBL session, responding 4 (agree) and 5 (strongly agree). 78% agreed that TBL sessions promoted long term retention of the material. 82% agreed that they have deeper knowledge of the topics covered in TBL sessions.
Conclusion: It was identified that students perceive TBL sessions as a useful method and a successful substitution of a didactic lecture. Due to the success of the conducted TBL sessions we plan to increase their amount in Anatomy course as well as in the other courses.
102 – Using TBL in Calculation-Intensive Courses
Jennifer Mott and Steffen Peuker
Cal Poly San Luis Obispo
In calculation intensive disciplines, such as science, technology, engineering and mathematics, learning objectives include the ability to solve complex problems oftentimes using longer calculations to solve a problem. It can be difficult to follow the 4S format for applications in Team-Based Learning when students are required to solve complex problems. Many calculation heavy problems are difficult to be completed during class time. There is the pressure to be able to solve the problems alone in preparation for exams and students are typically used to only working in teams for laboratory assignments or design projects. Faculty teaching these calculation intensive courses often dismiss even considering implementing TBL because of the aforementioned reasons. How do we create meaningful application assignments that students can complete during class time and still be able to follow the 4S format, specifically, the Significant Problem, Specific Choice and Simultaneous Report aspects?
Some solutions to implement applications in calculation intensive classes based on our experience are:
Significant (but doable) Problem: Break up the problems into smaller problems, assign calculation parts to finish in class, at home as homework or continuing during the next class session.
Specific Choice: The traditional approach is to have students choose the correct answer, but the focus in terms of learning is on the solution process not the numerical answer. Solutions include having students select governing equations they would use to solve the problem, select or create correct schematics to help them solve the problem, report intermediate solutions and solving for the final answer as a homework assignment.
103 – A Modified ‘Michaelsen-Koles’ Peer Evaluation Method in Team-Based Learning
Jody K. Takemoto and Leanne Coyne
California Health Sciences University
In addition to be competent health care professionals, pharmacist work collaboratively on health care teams, manage pharmacy personnel, balance formulary inventory and budgets, and provide patient education in a variety of settings. Because of the diversity in professional responsibilities, it is imperative that student pharmacist acquire and practice approaches to providing competent care and develop a life-long learning mindset to continue to grow personally and professionally. These attributes are so critical to the profession that the Accreditation Council for Pharmacy Education (ACPE) has included them as elements in the Doctor of Pharmacy Standards. In this study we focused on developing student pharmacists’ ability to give and receive feedback. One forum to be able to develop the knowledge, skills, and mindset of giving and receiving feedback are through peer evaluations. Peer evaluations are a component of Team-Based Learning™ (TBL) that is critical to individuals being able to work in groups and eventually form teams. We describe for the first-time implementation of a modified Michaelsen-Koles Peer Evaluation Method with an additional self-evaluation component in an 8-week pharmacy course. Preliminary data suggests that, students developed skills to provide quality feedback and make self-improvements. Generally, students were hard on themselves when commenting on their own opportunities for improvement but were in alignment with their team members on their strengths. Opportunities for refinement of the developed peer evaluation tool are apparent, this provided additional opportunities for student personal and professional development as well as artifacts to present for accreditation.
104 – The Little Things Matter – Making Three Small Changes To My TBL Teaching
Neo Chip Chuan
Background: I have been using Team Based Learning (TBL) since 2016 to teach financial audit to my 2nd year accountancy students. In my most recent semester, I made three small changes to my TBL teaching with the aim of solving three challenges:
- How to allow students to revisit lecture content before the readiness assurance test (RAT)?
- How to identify weaker students within each class?
- How to intervene and help free-riders in the team?
The three changes I had made and the purpose for doing each were:
- Recorded all my face-to-face lectures live so as to allow my students to revisit lecture content anytime anywhere
- Developed a simple and easy to use dashboard for each RAT so as to identify weaker students and tailor my teaching accordingly
- Conducted peer evaluation at the mid-point of the semester so as to identify, intervene and help the free-riders to level up
Results for the three changes were:
- Notwithstanding the availability of the recorded lectures, vast majority of the students still attend my face-2-face lectures. And they now have another option to revisit my lectures anytime anywhere in order to prepare for the RAT.
- From the students’ RAT results, I was able to identify the weaker students for each topic. This helped me to be more targeted in my teaching for each topics.
- Out of 200 students, there were 6 students with self & peer assessment (SPA) score of below 0.80. I was able to intervene early, counsel and motivate 5 of the 6 students such that their final SPA at the end of the semester improved.
Conclusion: As a result of the three small changes made, my teaching using TBL has become more effective for students’ learning and development.
105 – Assessing the impact of team-based learning on maternal and newborn nursing in accelerated Bachelor of Science in Nursing program
St. Luke’s International University, Japan
Background: Team-based learning (TBL) is an effective way of introducing active learning to students. Although TBL yields various favorable learning outcomes, only few studies have supported the effectiveness of team activities in improving the knowledge about maternal and newborn nursing of students in the 2-year accelerated Bachelor of Science in Nursing (ABSN) program. TBL is implemented in the Bachelor of Science in Nursing and ABSN programs for 6 and 3 months, respectively. Therefore, studies on whether TBL learning outcomes are associated with the length of learning periods are limited. In this study, we aimed to evaluate the effects of gaining knowledge and building a team using TBL on ABSN students.
Methods: For 3 months, ABSN students were taught the basic knowledge of maternal and newborn nursing using the TBL method. The students’ final exam scores and team dynamics between two classes of different grades of the ABSN students were assessed. The students took the final exam at the end of the course and evaluated the extent of team dynamics built by their respective teams twice during the course using the visual analog scale (range, 1–10). The study used written questionnaires.
Results: In this study, 59 ABSN students participated who were divided into six groups comprising 4-5 students of each grade. The mean scores of the final exam (range, 0–80) of the two classes of different grades were 64.6 (SD=7.2) and 69.4 (SD=4.6), which reached >80% of the total score. The team building score was higher in the second time than that in the first time; the amount of changes in team dynamics between two questionnaires in both classes were 0.48 (SD=0.69) and 0.68 (SD=1.25).
Conclusion: The 3-month TBL for maternal and newborn nursing was effective in achieving knowledge and building ABSN students team.
106 – TBL: Where do I begin? A Reflection on the first year of using TBL
Discussion of problem: Starting with TBL can be overwhelming particularly at a University or department that is not currently using it at all. Many professors do not have the time or resources to complete the work required to “flip” a traditional class to teach TBL style. This poster or focus group will present common obstacles that occur within the first year of teaching TBL style within a program that does not have faculty members already using it. Most early adopters have to build the plane while flying it.
Proposed approaches and solutions: The following is a list of obstacles, experiences, and lessons learned in the first year of teaching with Team-Based Learning.
Flight school: Getting to know TBL through resources including TBLC, workshops, and books.
Boarding: Introducing and getting buy-in from students new to TBL
Charting a course: Proposing new point break downs. Compromising with faculty who want to teach other ways.
In-flight maintenance: Planning how to implement elements of TBL: using current content, time for planning, use of space and technology.
Crash course correction- Handling student concerns over increased active learning demands and peer review.
Preparation for landing- How to make use of course evaluations and use experiences to share with other faculty.
Recommendations – New adopters of TBL have first-hand experience and insight into the obstacles and opportunities encountered in the use of TBL. A thoughtful reflection on a first-year experience would supply the most relevant data for improvements in fundamentals and innovations for providing mentorship within the TBL community.
Limitations-Individual case study
107 – A longitudinal faculty development program: supporting a culture of teaching in TBL
Annette Burgess, Elie Matar, Brendon Neuen, Greg Fox
University of Sydney School of Medicine
Introduction/background: Recent trends in faculty development demonstrate a shift from short term to long-term programs; formal to informal learning in the workplace; individual to group settings; and from individual support to institutional support. The purpose of this study was to develop and evaluate a one-year Clinical Teaching Fellowship (CTF) program designed to equip early career medical practitioners and basic scientists with necessary skills to facilitate Team-based learning (TBL).
Methods: The CTF program provided formal training, a choice of informal professional development activities, and practical co-teaching opportunities in TBL. Of the 40 registrants, 31 (78%) completed the program. Data were collected via questionnaire and focus group. Quantative data were analysed using descriptive statistics. Framework analysis,using the conceptual framework of Experienced Based Learning, was used to analyse qualitative data.
Results: Participants felt learning was enriched through the combination of training, practical teaching experience alongside senior clinical teachers, the multi-disciplinary context of training and co-teaching in TBLs; and the sense of community. Participants considered the CTF program as relevant to their needs and useful to their career. Competing clinical responsibilities made it difficult to for some participants to attend all training and TBL teaching. Most expressed a desire to continue teaching TBLs in future years, and an interest in remaining connected with the CTF program.
Conclusions: The CTF program provided a longitudinal faculty development framework promoting preparation, practice and development of TBL teaching skills within a supportive environment. In 2019, participant numbers have tripled. Securing institutional support to invest in the growth and development of early career teachers is essential to sustained innovation and excellence in teaching.
108 – Interprofessional Team-based learning in medicine and health: a pilot study
Annette Burgess, Eszter Kalman, Inam Haq, Andrew Leaver, Chris Roberts, Jane Bleasel
Education Office University of Sydney School of Medicine, Sydney Health Professional Education Network, Faculty of Health Sciences University of Sydney
BACKGROUND: Although challenging to integrate within university curricula, evidence suggests interprofessional education (IPE) positively impacts communication and teamwork skills in the workplace. The nature of Team-based learning (TBL) lends itself to IPE, with the capacity to foster a culture of collegiality among health professional students. This study reported on a 2019 single pilot interprofessional TBL session on backpain, involving medical and physiotherapy students. Our aims were to explore factors impacting students’ collaborative work within the TBL process, in order to inform interventions to optimise TBL in the IPE context.
METHODS: In a mixed methods study, 311 students participated in the TBL: 222/277 (80%) Year 1 medical and 89/89 (100%) Year 2 physiotherapy students completed one interprofessional Musculoskeletal Sciences TBL session. A questionnaire, including closed and open-ended items was distributed to students at completion. Data were analysed using descriptive statistics and thematic analysis. Pre-class quiz scores were compared between the groups.
RESULTS: 117/311 (38%) of participants completed the questionnaire. Both medicine and physiotherapy students appreciated the opportunity to learn about the curriculum of another healthcare discipline, and their scope of practice; gain multiple perspectives on a patient case from different disciplines; and recognised the importance of multidisciplinary teams in patient care. Students felt having an interprofessional team of facilitators, providing immediate feedback helped to consolidate learning and promote clinical reasoning. There was no difference between medical and physiotherapy students’ pre-class quiz scores.
CONCLUSION: Our study demonstrated that the small group and task-focused characteristics of TBL provided a student-centred teaching strategy, supporting the achievement of interprofessional learning goals. Students valued their interactions with other students from a different professional degree, opportunities to problem-solve together, and learn different perspectives on a patient case. The individual quiz results demonstrate both groups of students had a comparative level of prior knowledge to be able to work together on class activities.
109 – Nurse’s competencies development by using TBL in a Graduate Studies Course
Viviane Bianca Bella, Maria Mercedes Samperiz Fernandes, Mariana Lucas da Rocha Cunha
Hospital Israelita Albert Einstein
BACKGROUND: Developing core competencies for nurses in specialization courses is challenging, mainly because students have different experiences in managing newborn critical situations. Team Based Learning can assist in the teaching process by stimulating the contribution of each group member to the individual’s learning.
AIM: To assess whether the use of TBL facilitates the learning of postgraduate students in neonatal and pediatric nursing courses, and whether this method enhances core competencies in caring for newborns with jaundice.
METHODS: field research, experimental, with quantitative focus. The sample consisted of 31 post-graduation students, most were women, average age of 32,5 anos; 41.9% already working with neonatology. Data were obtained by applying two instruments after a TBL session about jaundice. One of them was Evaluation of Student Perception of the TBL Method (APA-TBL) and the other to assess the perception of competency development in the research question.
RESULTS: In their perception through TBL session, items with concordance in the four dimensions of APA-TBL were highlighted reinforcing good results based on evidence from previous studies. The five items of the instrument two were distributed in agreement proportions higher than 83.9%. it is noteworthy that the items “I believe that after the TBL will be able to understand the physiology of jaundice in the newborn” and “The TBL helped me to assess the risks and signs of aggravation of neonatal jaundice ”, obtained 96.8% (30). In other words, the students perceived themselves as competent to explain the physiology of jaundice, to assess the risks and signs of worsening of the disease and to care for the newborn in this condition.
CONCLUSIONS: TBL is effective in nursing post-graduation and promotes critical thinking and classroom decision-making. Teamwork improved students performance and subject grasp. It can be incorporated in post-graduation to develop professional practice competencies.
Team Based Learning used to Engage Inclusive Pedagogy in the First Year Art and Design School
Migiwa (Micki) Spiller, Pratt Institute
Background: In an Art and Design institution, first year students consist of a diverse population, and each six-hour studio class (16-20 students) has a wide and disparate range of learning levels that are dependent on factors such as; cultural background, motivation, ability and/or disability, and diverse interests in future majors with different background knowledge to name a few. One of the important learning outcomes of a studio course is the ability to honestly critique works of peers. Differentiated instruction must be built into each class, where student-centered group discussions are key to critique through learning from different perspectives, and using TBL can promote inclusive pedagogy. I discovered through the use of these practices, that shy and international students especially benefited because it created a safe classroom community to practice their verbal communication skills.
Description: Even in Art and Design schools, studio classes tend to follow lecture-based instruction and historically there exists a hierarchy of power in critiques, where the majority of discussion is led by the professor with little dialogue and input from students. This sets up a passive learning situation where students do not often interact, creating silos of learning and prohibits them from seeing different perspectives. The use of different application exercises learned through TBL restructures the old methods of critiques and has helped my students engage in class discussions. They are now much more open to giving and getting critical feedback of one another’s work.
Results: The students reported that my course utilizing TBL has created a more genial working environment where they feel comfortable asking their peers for feedback. End of course surveys have also found that the international students writing and critical assessment of their peers have also improved. Using TBL application exercises have helped students with self efficacy and gave them agency in their own learning process.
This research was conducted with the support of Pratt Institute’s Center for Teaching and Learning through their Faculty Learning Community grant.
The Use of Team Based Learning Principles in Community Based Service Learning Courses
Travis Peterson, Utah State University
BACKGROUND: Research in the Science of Teaching and Learning has shown that the use of high-impact teaching and educational strategies lead to enhanced and higher level learning outcomes. The value of these strategies has been established by the American Association of Colleges and Universities (AAC&U) and the National Survey of Student Engagement. The use of these high-impact activities has not only shown the ability significantly enhance learning, but also influence the professional and personal development of the learner in a positive way. The use of Team Based Learning has also shown great promise to drive deeper student learning and development. The utilization of high-impact strategies and Team Based Learning approaches together may provide an excellent pedagogical modality to enhance high-impact learning for students, especially in community based service learning settings.
DESCRIPTION: This presentation will focus on the intentional utilization of Team Based Learning to enhance the learning outcomes in an undergraduate course designed around the high-impact practices of community based service learning, collaborative project based learning, and writing intensive instruction. Using this approach also provided a structure to move traditional lower-level learning activities from the Boyer model outside of the classroom and bringing the higher-level activities from the Boyer model into the classroom. Thus, providing more time and latitude in the classroom to enhance student learning and program development using team based approaches.
This approach was used in a junior/senior level undergraduate Program Planning and Evaluation course for Community Health and Public Health majors. The objective of the course was for students to learn the processes and ti work in teams to assess the needs of a community, develop a program to meet the needs, implement the program in the community with community partners, and evaluate the community impact and outcomes of the program.
RESULTS: Data regarding student perceptions, satisfaction, learning, professional development, and community impact will be discussed. Samples of the programs produced and community impacts and outcomes achieved will also be discussed.
CONCLUSION: The use of Team Based Learning in a community based experiential learning course provides a course framework and design for enhanced student learning and community impact.
A New and Diverse Interpretation of 4S: Size, Subject, Support,….and Siblings?
Lauren Vicker, St. John Fisher College
In this oral presentation, the traditional TBL 4Ss (significant, same, specific, and simultaneous) are re-interpreted in a new and diverse way. The presentation, led by two professors uniquely connected but with diverse fields of study and different levels of experience with TBL, will explain how institution size and discipline do not matter when implementing Team-Based Learning, and that support, too, can come from diverse areas.
The presenters include an information systems professor from a large public university and a communications professor from a small liberal arts college. Each will demonstrate how their diverse journeys into Team-Based Learning revealed common approaches, outcomes, and lessons learned.
A key feature of this presentation will be the presentation of avenues of support for those who are looking to implement TBL into their curriculum, whether with professional colleagues or as a single faculty member. The presenters will share their strategies and how they adopted TBL to their fields with an emphasis on how the audience can also adapt these ideas to their own fields of study. Attendees will take away new contacts, resources, and practical lessons based on the experiences of the presenters who have tried, failed at times, and succeeded with Team-Based Learning.
Applying TBL to Intermediate Photography
Nick Shepard, Sacramento State University
BACKGROUND: Most studio-based photography courses are taught with a focus on individual training. Traditionally, students must learn a number of techniques to create basic images. In order to understand intermediate-to-advanced methods, students must become competent in a range of techniques quickly. Assessment presents a challenge, especially when it is primarily image-based. I sought to incorporate elements of Team Based Learning (TBL) into the photography curriculum as a way to provide structure and support to students.
DESCRIPTION: In my current semester-long course covering intermediate photographic technique I paired the Readiness Assurance Process (RAP) with traditional individual projects—eg. shooting, editing, and printing assignments. The students participate in a RAP consisting of 10 questions to prepare for technical and conceptual topics. Following this process, we typically engage in a brief demonstration or discussion, followed by supervised lab time where students can apply the concepts to their work. None of these students had previously encountered the RAP. Students will respond to midterm and end-of-semester surveys to describe their experience of the RAP.
RESULTS: Anecdotally, I have encountered better-prepared students, less time presenting technical concepts, and more-structured discussions of theoretical concepts. Students seem to enjoy the discussions during tRAT sessions and generally appear to be more energetic during and immediately after the RAP than were students in courses that did not include the RAP. The technical quality of student work is improved over previous junior courses.
CONCLUSION: Due to the increased engagement shown by students, I intend to integrate the RAP into additional technical courses. Further work is required to integrate TBL more deeply into the curriculum.
TBL Jeopardy™ using White Boards or Turning Point Team Clickers for Creating Engaging Application Exercises and/or Exam Review Sessions
Jennifer Rene Courtney, PharmD | John Cusick, PhD | Eugene Kreys, PharmD, PhD, PCPCS | Suzanne Clark, PharmBS, PhD, California Northstate University College of Pharmacy | California Northstate University College of Medicine | California Northstate University College of Pharmacy | California Northstate University College of Pharmacy
Introduction: Team-based Learning (TBL) and Jeopardy® have traveled parallel paths in recent history. Both were developed in the 1960s & 1970s and both have proven to be durable, such that when their fundamental components are incorporated appropriately, both can be successfully adapted to a range of settings. TBL application exercises are most successful when students are actively engaged. The Jeopardy® platform can be highly engaging and, when correctly delivered, can promote retrieval memory and accountability. Accordingly, it is not surprising that Jeopardy® has been successfully adapted to TBL.
Aims: This presentation will outline how Jeopardy® has been adapted to TBL classes using several approaches, ranging from team white boards to Turning Point® Team Clickers, the latter using methods from Cusick (MedEdPORTAL, 2016).
Methods: White-Board Jeopardy: The economical design and implementation of TBL classes using white boards can employ a free downloadable Jeopardy game and small, erasable white boards for simultaneous reporting of team answers. Questions are mapped to learning objectives as column headings and engagement can be promoted with iRAT/tRAT points incentives.
Turning Point® Team Clickers: The use of team clickers has been developed for a Jeopardy® review game that incorporates a team leaderboard created with the assistance of our University IT. This version of Jeopardy® creates fast, but friendly competition and has been useful when teaching large classrooms of up to 120 students, as the team clickers permit engaging discussions in groups of 5-6 students/team. Individual questions are simultaneously considered by all teams. Scores are continually tabulated and displayed by the team leaderboard.
Assessment of Student Learning and Communication Skills Moving from Lecture to Team Based Learning
Cynthia Standley | Jeremi Smith | John Gage | Lorrie Pena | Paul Kang | Paul Standley, The University of Arizona College of Medicine-Phoenix
At UA COM-P, moving from a traditional, lecture-based scenario to a more active learning TBL paradigm was mandated by the Curriculum Committee in AY 16-17. With funding received from the TBLC, we enrolled students in the Classes of 2020 and 2021 to study how moving from lecture to a combination of independent learning modules (ILM) plus TBL impacts short-term medical knowledge mastery and immediate and long-term gains in interpersonal and communication skills. 64% of the Class of 2020 and 75% of the Class of 2021 consented to enroll in the study for a total research cohort of 112 students. The Class of 2020 has completed their clerkships, while Class of 2021 is currently in the middle of their clerkship experience. As these sessions are non-mandatory, students were grouped based on the percentage of active learning sessions attended and this was then correlated with their interpersonal and communications competency scores in their Year 1 and 2 blocks, as well as primary care clerkships (Family Medicine, Internal Medicine). Using a Kruskal-Wallis test and Spearman’s correlation coefficient, data showed that students who attended > 30% of the active learning sessions were better able to cooperate, collaborate, and communicate with teams (p<0.02) and document and present patient data, use effective nonverbal communication and questioning (p<0.01) in the Cardiovascular-Hematology Block compared to those attending < 10% of these sessions. For the Internal Medicine clerkship, students attending more TBL sessions in Years 1 and 2 received higher scores in using focused listening skills (p<0.02). Students completed an end-of-year survey to evaluate their experience with these TBL sessions. Most students found them beneficial to work through clinical problems, hone critical thinking skills while solving problems, apply previous knowledge to case studies and to compare their thought processes with others’ while gaining insight on different perspectives. However, most students would still rather problem solve on their own using independent study modules. Surprisingly, one student suggested that the most useful part of the TBL sessions was getting the information, not necessarily the peer interactions. We continue to analyze the data as the Class of 2021 finishes their clerkships and will be reporting out the effects of attendance on block exam scores as well as shelf exams and step 1 scores while further analyzing the influence of attendance on the competency scores.
A Qualitative Analysis on the Effectiveness of Peer Feedback in Team-Based Learning (TBL)
Sarah Lerchenfeldt | Suzan ElSayed | Stephen Loftus | Gustavo Patino | David Thomas, Oakland University
Introduction: Peer evaluation is the final practical element of TBL that may be used to promote interpersonal and team skills essential for future success. To our knowledge, there is limited information on medical students’ perceptions of peer feedback, both in terms of its value and how it has affected them as they move forward in their careers. In an effort to improve peer feedback implementation and long-term utility, the goal of this study was to gain a deeper understanding of students’ perceptions about the effectiveness of giving and receiving peer feedback.
Methods: This study utilized an exploratory qualitative research design using focus groups. A total of six focus groups were conducted, each comprised of medical students in their first through fourth years, and medical residents who graduated from the same institution. Discussions were transcribed verbatim and thematic analyses were completed by five independent reviewers. Themes were identified by consensus and any differences were resolved by careful re-reading of the transcripts.
Results: Four key themes and several interconnected subthemes were identified. Subthemes for the first key theme, preparation and training, included the importance of peer feedback instruction and the clarification of what constructive feedback involves. With the second key theme, procedure and implementation, noteworthy subthemes included providing oral versus written peer feedback and the use of self-reflection. Faculty role modeling was a notable subtheme identified under the third theme, evaluation of student feedback. Significant subthemes for the fourth key theme, student considerations, included student maturity and evolution through medical school as well as the stresses of grading, and anxiety about faculty perceptions.
Conclusion: Our analysis raised awareness about several potential areas of concern and difficulty for students in regards to the TBL peer feedback process. Quality improvement initiatives may include the addition of self-reflection and use of oral instead of written feedback.