Summative Peer Evaluation of Teaching: Literature Review and Best Practices
Contents
Part 2: Evidence and Rationale for the List of Best Practices
Part 3: Best Practices for an Effective System of Summative PET
Appendix: Structuring a Preliminary Discussion of Teaching
The Teaching Perspectives Inventory
Also see:
Resources Related to Peer Review of Teaching
CELT PET Report Forum Presentation, Oct. 8, 2009
- PET Report Forum Presentation Text (36 KB PDF)
- PET Report Forum Presentation Slides (99 KB PDF)
Purpose
The goal of this report is to provide faculty members and administrators at Iowa State with a statement of best practices for the Peer Evaluation of Teaching (PET) in cases where it plays a role in personnel decisions such as promotion and the granting of tenure. Its recommendations, listed in part three, are based on a broad survey of scholarly literature, with a focus on empirical studies that provide evidence about the fairness, validity and reliability of various forms of peer assessment of teaching. Our hope is that the practices this document suggests will be broadly adopted by colleges and units in the University that currently require, or plan to require, some form of PET in promotion dossiers or post-tenure reviews. All the guidelines that follow, therefore, are for summative evaluations, in which the evaluators are assumed to be colleagues of equal or greater rank in the same or similar departments and disciplines.
Introduction
According to its mission statement, the primary goals of Iowa State University are to "create, share and apply knowledge."1 Creating knowledge, the statement continues, involves "world-class scholarship in teaching, research and creative endeavors." When it describes teaching in these terms, ISU is following a national trend: the official language of universities once presented teaching and research as separate enterprises, but now often joins the two together. This new discourse presents teaching and research as facets of a broadly-conceived idea of "scholarship," with the understanding that both should be subject to similarly rigorous professional standards, and both should figure in decisions about faculty hiring, tenure, and compensation.2 Advocates of this conceptual shift hope that a widespread acceptance of teaching's status as a scholarly activity will increase its prestige in the academy, encouraging faculty members to approach it with the same intellectual rigor, commitment and ambition that they bring to their research.
Making this new vision of teaching and its place in the professional life of faculty members something more than a catch-phrase is a complicated project that entails profound shifts in institutional culture. If teaching is to become more like research, and to be approached by the university in similar terms, the professional practices linked to developing, presenting and assessing the impact of individual courses will need to change. The ISU mission statement calls for the rewarding and recognition of "excellent teaching." On what basis will judgments of this kind be made? How much bearing should a faculty member's "excellent teaching" have on his or her promotion, and what role should it play in determining a faculty member's long-term status within the institution as a whole?3
The following report, prepared by a subcommittee of the CELT Advisory Board in spring 2009, is concerned with one aspect of this larger task: the transformation of teaching into an activity subject to peer review. It is in three parts, plus an appendix. Part one describes the potential positive impact of a well-designed system of Peer Evaluation of Teaching (PET), and addresses the difficulties the implementation of such a system can pose. Part two presents the empirical basis and rationale for the list of recommendations that appears in part three. The appendix describes two potentially useful starting points for the kind of broad-based discussion of teaching that evaluators should engage in before they begin direct assessment of their peers.
Part 1: The Impact of PET
The Benefits of a Well-Designed System
Most writers in the field agree that for teaching to be truly scholarly, it must become "community property." Traditionally, as education scholar Lee Shulman points out, university teaching has been treated as something "generic, technical," an activity faculty members are obliged to perform, but one that is fundamentally removed from "the disciplinary, inter-disciplinary, or professional community" as a whole. Research, in contrast, is an activity that deeply enmeshes faculty members in this community. The process of peer review is one of the primary means by which research acquires this larger reach; applying the same scrutiny to teaching, Shulman suggests, would do much to increase its credibility as a form of scholarship in its own right.4
More tangibly, studies of classroom learning show the potential of PET, and the culture of scholarship it creates, to improve student outcomes. A 1998 study of 1258 students in seven engineering schools, for instance, found that faculty members who abandoned the conventional lecture paradigm and devoted serious attention to incorporating more innovative techniques into their teaching practice achieved considerably higher scores when their students were asked to assess how much they had learned. This approach to pedagogy, the authors of the study note, is labor intensive, and under a traditional system, which makes research the primary currency of institutional prestige, faculty members have little incentive to put in the effort required.5 A system that used peer review to give classroom innovations the intellectual heft and publicly recognized value of scholarship could provide new incentives, and hence potentially increase overall teaching effectiveness. An article describing the results of a pilot PET program at the University of Nebraska-Lincoln provides some anecdotal support for this observation.6
A well-designed system of Peer Evaluation of Teaching can also give a department valuable tools for making the most of its faculty resources. A 2002 study of 418 introductory psychology students at a large Midwestern Carnegie research institution revealed the crucial importance of the relation between an instructor's teaching style and the nature of the student audience he or she is addressing. In the study, some styles worked better with non-majors and first-year students, while others were more effective with majors and students already familiar with the basics of the discipline being taught. Instructors whose styles led "content-unfamiliar students" to rate their own learning quite highly were often rated lower by "content-familiar" ones, and vice versa.7 By making classroom performance "community property," a department can develop a clear sense of the teaching styles of various faculty members, and use this knowledge to make the most of their abilities.
Finally, on the most practical level, there are certain elements of a teacher's performance that only colleagues in the same or closely-related disciplines can accurately assess. In an important 1980 article, Peter Cohen and Wilbert McKeachie identified ten elements of teaching that colleagues were particularly suited to judge, presented here in Carol-Ann Courtneya's paraphrase: "mastery of course content; course organization; appropriateness of course objectives; appropriateness of instructional materials [?]; appropriateness of evaluative devices (i.e. exams, written assignments); appropriateness of methodology used to teach specific content areas; commitment to teaching and concern for student learning; student achievement based on performance on exams and projects; and support of departmental instructional efforts."8 In all these cases, student evaluations alone are an insufficient indicator of effectiveness; only the informed judgment of disciplinary colleagues can complete the picture.
Two Difficulties: Tradition and Time
One of the key differences between research and teaching in traditional university settings is the way in which they are evaluated. Peer review of research plays an important role in professional life across academe. It is also a central element of graduate training in disciplines that expect university faculty to hold Ph.D.s. The act of conceiving and producing a doctoral thesis is an intensive apprenticeship in the methods of peer review, and a first opportunity to learn how to meet its intellectual demands. The training graduate students receive in teaching has traditionally been far less rigorous. In universities, teaching has been a skill learned "on the job," in a comparatively informal manner, through service as a teaching assistant and perhaps from programs like those offered by organizations like CELT.
This long-standing difference in the way the profession has approached research and teaching creates an important challenge for advocates of the new, broad conception of scholarship. Kremer, in a 1990 study of a peer evaluation system implemented at a "large Midwestern university," found that, among faculty members charged with reviewing the performance of their colleagues, 60% rated themselves as "confident" in their judgments about research, while only 33% expressed similar assurance about their judgments of teaching. In his analysis, Kremer concluded that this confidence gap was largely a product of professional convention: faculty members had long experience with peer review in research, but their training left them much less well-equipped when evaluating teaching in similar terms.9
The only way to address this difficulty is for evaluators to take the time to master new professional skills. This requirement is perhaps the single largest shortcoming of PET: it demands a time commitment from faculty members whose schedules are already heavily loaded. The literature acknowledges this problem, and while there have been one or two attempts to develop strategies that might reduce the time required, the fact that PET entails an additional set of obligations remains impossible to overlook.10 Given this reality, it is likely that the frequency of summative PET in a given department or college will be dictated by institutional requirements, particularly the timetable set for contract renewal, promotion, and post-tenure evaluation reviews.
Part 2: Evidence and Rationale for the List of Best Practices
This section presents the research behind the guidelines that conclude this report. It is in three parts, which reflect the major thematic concerns of the literature on the subject:
- Understanding the distinction between Formative and Summative evaluation
- Guidelines for evaluators
- Guidelines for what is to be evaluated
Formative and Summative Evaluation
This report is concerned with what the philosopher Michael Scriven has called "summative" evaluation: judgments about the results a teacher has achieved. Scriven distinguishes this sort of judgment from "formative" evaluation, which seeks to provide advice in order to help a practitioner improve. 11 Formative evaluations generally occur in the context of a relationship with a mentor, or with an independent expert at an organization like CELT. To be fully effective, formative evaluations should be confidential, and should remain the property of the instructor being evaluated. This allows an instructor the freedom to try new approaches and techniques, without fear of penalty should an experiment fail. Summative evaluations, in contrast, are not confidential, and are usually performed for use in personnel decisions such as contract renewals, promotions, and the granting of teaching awards. Where formative evaluations assess an instructor's teaching on its own terms, summative evaluations add a comparative dimension, placing the individual teacher's performance in explicit relation to the performance of his or her colleagues.12
The importance of this distinction is widely acknowledged in the literature. For many authors, an awareness of it is the necessary foundation for any effective system of PET.13 These two forms of evaluation should be practiced in conjunction with one another: between summative evaluations, an instructor should have the opportunity to use formative evaluations to hone his or her teaching skills. At the same time, however, there is a general consensus in the field that the fairness and effectiveness of each type of evaluation depends on its separation from the other.
Crucially, many authors agree that people who have been formative evaluators of an instructor should not serve as summative evaluators. The institutional framework of ISU already supports this distinction. CELT and its staff exist to provide instructors with ongoing, formative evaluation; the mentors assigned to junior faculty can fulfill a similar function. Summative evaluation, in contrast, should be performed by other faculty members in an instructor's own department, or in another department that is closely related. Where possible, these colleagues should be of higher rank than the faculty member being evaluated.
When designing a system for PET, evaluators should begin the process with a discussion of what effective teaching entails. Two studies have shown that without this kind of preliminary discussion, ratings given by individual evaluators tend to vary quite widely.14 A 2006 study compared faculty ratings of videotaped lectures, one before a broad-based discussion of teaching and one after. Before the discussion, faculty members tended to base their evaluations on the extent to which the teaching style of the instructor on the videotape matched their own, and their individual assessments varied considerably. After the discussion, the interrater reliability improved.15 (For further information about structuring effective pre-evaluation discussions of teaching, see the Appendix.)
The reliability of evaluators is also shaped by the nature of their relationships with the person being evaluated. A 2000 study shows that when teachers are allowed to choose their own evaluators, those evaluators produce assessments that are at considerable variance with assessments from other colleagues or administrators.16 A different sort of problem based on social relationships emerges when evaluators judge colleagues purely on the basis of reputation, in the form of impressions gleaned from "faculty lounge discussions," "debates at department meetings," conversations with students, and "quadrangle discussion."17 Such assessments have been shown to closely track the ratings provided in student evaluations, but since they are not based on any direct experience of a colleague's teaching, their validity is questionable.18
These contrasting scenarios both underscore the need for the designers of PET systems to think carefully about ways to ensure the impartiality and evidence-focus of their evaluations. Studies have tested several techniques that control for the potentially distorting effect of social relationships. The following three strategies have achieved positive results, in the form of increased interrater reliability:
- Having a committee of at least three elected evaluators, who each bring their individual judgments to the table and then, through discussion, mold them into a single report;19
- Establishing a system in which evaluators remain anonymous;20 and
- Selecting evaluators who have received formal training as assessors of teaching.21
Guidelines for What Is To Be Evaluated
To be reliable and valid, an evaluation of teaching should draw on multiple sources of information. In documented cases where a group of evaluators made their judgments "cold," on the basis of a single source of information, like a classroom visit, variation among scores was high.22 A stable consensus among evaluators has been shown to be more likely to emerge in cases where judgments are based on teaching portfolios, collections of documents that include a variety of different types of information.23
Perhaps in response to these findings, colleges and universities across the country have increasingly come to use teaching portfolios as the primary material in summative peer evaluations of teaching.24 In the workbook for its project on the peer review of teaching, which involved forty-two departments at seventeen universities, a committee formed by the AAHEA developed a template for teaching portfolios designed to give their contents "the structure of a scholarly project." This template calls for the inclusion of materials in three categories:
- A description of goals and intentions, in the form of a statement of teaching philosophy and course syllabi;
- Evidence for how those intentions were enacted, which could include evaluations by peers who have observed the faculty member in the classroom and examples of materials developed for courses, like problem sets, examinations, and reading packets; and
- Documentation of results, in the form of end-of-term student evaluation forms and samples of student work.25
In order to make the information included in these portfolios speak as clearly as possible, several authors suggest that the individual components be woven together with narrative commentary. Evaluators can further enrich the context of their assessment by conducting an interview with the faculty member who has assembled the portfolio.26
When used for summative evaluation purposes, teaching portfolios should contain a mixture of mandated material – such as student evaluation forms and peer observation reports – and material chosen by faculty members to reflect their best work. Constructing a portfolio in this way mitigates what the literature shows to be two serious, if countervailing, reliability problems: 1) the well-documented tendency of teachers to over-rate their own performance; and 2) the high interrater variability that often mars assessments of classroom observations.27
Part 3: Best Practices for an Effective System of Summative PET
- Planners should acknowledge the distinctiveness of summative evaluation, and structure their PET system accordingly.
- Summative and formative evaluations should be performed by different people;
- Summative evaluations should include a comparative dimension, assessing a teacher's work in relation to that of his or her colleagues;
- Summative evaluators should not be chosen by the teacher being evaluated, but should instead be elected or appointed;
- Summative evaluators should be colleagues of equal or greater rank in a department or discipline the same as or similar to that of the teacher being evaluated;
- To ensure sufficient reliability, a summative evaluation should be the collaborative product of a committee of at least three evaluators;
- Summative evaluations should occur at prescribed intervals that the evaluee knows in advance, most likely as part of mandatory reviews for contract renewal, review for tenure, and post-tenure reviews.
- Designing a system of summative PET should begin with a discipline-specific discussion of what effective teaching entails, either among the evaluators, or in the unit as a whole. Such a discussion should yield a rubric that the evaluators can use to structure their judgments. (For more on this, see the Appendix.)
- When making their decisions, peer evaluators should draw on multiple sources of information, presented in the form of a teaching portfolio.
- When being used for summative evaluation, a portfolio should include both mandated elements and elements selected by the teacher as examples of his or her best work.
- The portfolio should include the following:
- A statement of teaching philosophy
- Course syllabi
- Reports by peers who have observed the teacher in the classroom or on video
- Examples of course materials (problem sets, exams, reading packets, etc.)
- Samples of student work
- Student evaluation forms
- To be fully effective, summative evaluation should not occur on its own, but should instead alternate with an ongoing program of formative evaluation, provided both by faculty mentors and by members of the CELT staff.
Appendix: Structuring a Preliminary Discussion of Teaching
In research contexts, peer review works because participants both inside and outside a particular field perceive it to be a valid and relatively transparent indicator of quality. Its status depends on its basis in widely-accepted professional norms and habits of mind. For summative PET to function in a similar manner, it must live up to a similar standard. In practice, however, as many studies show, this standard has proved difficult to meet. Multiple raters often assess the same teaching performance in very different ways: where one might be quite impressed, another might find it woefully deficient.28 In a 2002 study of a portfolio-based system of PET, Kathleen Quinlan demonstrated that this variability stemmed from a tendency for peer evaluators to make their own experience the primary basis of their normative judgments about others: colleagues gave high ratings to teaching practices and attitudes that resembled their own, and low ratings to those that differed.29 Another recent study indicates that an awareness of this tendency for evaluators to engage in a "search for self" when assessing their colleagues' classroom performance contributes to a widespread skepticism of the validity of summative PET in general.30
Scholars of peer evaluation, however, have repeatedly shown that beginning an evaluation process with a discussion about teaching in general mitigates both the problem of low interrater reliability and the lack of confidence such variability creates (see part two of this report). A discussion of this type serves to make PET more like peer review of research by constructing a stable, collectively-derived body of general principles against which individual teachers can be assessed. It takes evaluators "outside themselves," in other words, leading them to question and perhaps set aside unreflectively-held assumptions that might otherwise distort their sense of a particular faculty member's effectiveness in the classroom.
Recent scholars in the field have proposed two useful starting points for such a discussion:
- The Teaching Perspectives Inventory
- The Delphi Framework for Teaching Competencies
This appendix presents overviews of each of these, along with a concluding statement about the important role preliminary discussion should play in the design of any system of PET.
The Teaching Perspectives Inventory
On the basis of his findings in a series of international studies conducted between 1992 and 2000, Daniel D. Pratt developed an online instrument for teacher self-assessment which he calls the "Teaching Perspective Inventory" or TPI.31 (It is accessible at www.teachingperspectives.com.) This inventory uses a set of 45 questions to determine the extent to which a given respondent's pedagogical approach is dominated by one of five very different teaching perspectives; each perspective produces equally effective results in the classroom.
A 2007 study tested the validity of this instrument by having faculty members rate a video recording of a lecture twice – once before having completed the TPI, and once after having used their TPI scores as the point of departure for a wide-ranging discussion of teaching. At the end of this process, interrater reliability improved significantly.32 By providing an opportunity for faculty members to reflect on their own approaches to teaching, and to conceptualize several very different but equally effective alternatives, the Teaching Perspectives Inventory made it easier for evaluators to leave their own personal views behind when judging the work of others.
The Delphi Framework for Teaching Competencies
Another useful way of taking evaluators "outside of themselves" is to provide them with a rubric they can use to structure their assessments of their colleagues' teaching. Again, a rubric of this type is most effective when it has been developed in the course of a discipline-specific discussion of teaching practice and desired outcomes.
A 2004 study by a group of Dutch sociologists provides one useful starting place for such a discussion. The study began with 74 university teachers in a representative variety of disciplines, all with at least five years' experience in the classroom. Each member of this group completed a survey in which they rated various aspects of good teaching using a Likert scale; elements that scored low overall were eliminated, while those that scored higher were retained. In a process known as the Delphi technique, the survey was administered repeatedly until the results stabilized, at which point they reflected a broad consensus among all participants.33
The following characteristics of effective teaching emerged from this process, and could potentially serve as the basis for a more specific, discipline-tailored rubric. The list is divided into five categories, each of which represents one aspect of a teacher's responsibilities, broadly conceived:
- Person as teacher
- Is skilled at communicating
- Has a positive attitude towards students
- Exhibits respect for all students
- Expert on content knowledge
- Is capable of using relevant information from specialist literature in his or her own teaching
- Has thorough knowledge of his or her subject
- Has knowledge of new developments in his or her subject
- Facilitator of learning processes
- Places the student at the center when designing educational material
- Is capable of designing activating educational materials
- Is capable of building education in such a way that students gradually learn to learn in a self directed manner
- Is capable of giving feedback
- Places the student at the center of his or her teaching
- Is capable of activating students
- Is capable of assessing students' learning results
- Is capable of re-adjusting his or her practice on the basis of evaluations
- Is capable of designing tests that are appropriate for the desired learning results
- Organizer
- Is capable of cooperating with colleagues
- Is communicative when cooperating with colleagues
- Is capable of contributing to the curriculum
- Scholar/lifelong learner
- Is capable of reflecting on his or her teaching performance
- Is capable of drawing conclusions from reflection on his or her teaching performance
- Is open to innovation
The Key Role of Preliminary Discussion
Whatever specific approach a unit or college chooses, it is crucial that all participants involved in designing a PET system be aware of the central importance of preliminary discussion to the effectiveness of the whole review process. Without an initial period of reflection about what good teaching involves and what specific instructional objectives a given unit wishes to achieve, it is quite possible that the evaluators' conclusions will be unreliable in ways that impair their usefulness. A rubric to structure the evaluators' assessment, designed with the help of tools like the ones described above, is a straightforward and efficient way to provide the solid intellectual foundation any effective system of PET requires.
Notes
1 "Strategic Plan 2005-2010," http://www.public.iastate.edu/~strategicplan/, accessed March 22, 2009.
2 The most influential statement of this new conception of scholarship is Ernest L. Boyer, Scholarship Reconsidered: Priorities of the Professorate (Princeton: Carnegie Foundation for the Advancement of Teaching, 1990).
3 "Strategic Plan 2005-2010."
4 Lee Shulman, "Teaching as Community Property," in Pat Hutchings, ed., From Idea to Prototype: The Peer Review of Teaching, A Project Workbook (Washington, DC: American Association for Higher Education, 1995).
5 Alberto F. Cabrera, et al., "Developing Performance Indicators for Assessing Classroom Teaching Practices and Student Learning: The Case of Engineering," in Research in Higher Education 42:3 (2001): 327-352.
6 Daniel J. Bernstein, et al., "An Examination of the Implementation of Peer Review of Teaching," in New Directions for Teaching and Learning 83 (Fall 2000): 73-86;
7 Dieter J. Schönwetter, et al., "Content Familiarity: Differential Impact of Effective Teaching on Student Achievement Outcomes," in Research in Higher Education 43:6 (2002): 625-655.
8 Peter A. Cohen and Wilbert McKeachie, "The Role of Colleagues in the Evaluation of College Teaching," in Improving College and University Teaching 28 (1980): 147-154; Carol-Ann Courtneya, et al. "Through What Perspective do We Judge the Teaching of Peers," in Teaching and Teacher Education 24 (2008): 69.
9 Kremer, John F. "Construct Validity of Multiple Measures in Teaching, Research, and Service and Reliability of Peer Ratings." In Journal of Educational Psychology, 82:2 (1990): 213-218.
10 For examples, see Raoul A. Arreola, Developing a Comprehensive Faculty Evaluation System, 2nd Ed. (Bolton, MA: Anker, 2000); and Chism.
11 Michael Scriven, "The Methodology of Evaluation," in Robert E. Stake, ed., Curriculum Evaluation (Chicago: Rand McNally, 1967).
12 Nancy Van Note Chism, Peer Review of Teaching: A Sourcebook, 2nd Ed. (Bolton, MA: Anker, 2007).
13 See, e.g., Ronald R. Cavanagh, "Formative and Summative Evaluation in the Faculty Peer Review of Teaching," in Innovative Higher Education 20:4 (1996): 235-240; John A. Centra, Reflective Faculty Evaluation: Enhancing Teaching and Determining Faculty Effectiveness (San Francisco: Jossey Bass, 1993); Chism; Hutchings, ed.; Trav D. Johnson and Katherine E. Ryan, A Comprehensive Approach to the Evaluation of College Teaching, in New Directions for Teaching and Learning 83 (Fall 2000): 109-123.
14 John A. Centra, "Colleagues as Raters of Classroom Instruction," in The Journal of Higher Education 46:3 (1975): 327-337.
15 Courtneya, 2007.
16 John A. Centra, "Evaluating the Teaching Portfolio: A Role for Colleagues," in New Directions for Teaching and Learning 83 (Fall 2000): 87-93.
17 Centra, 1975, quoting J.A. Kulik, "Evaluation of Teaching: Memo to the Faculty" (Ann Arbor: Center for Research on Learning and Teaching, University of Michigan, 1974).
18 Centra, 1975; Kremer; Kenneth A. Feldman, "Instructional Effectiveness of College Teachers as Judged by Teachers Themselves, Current and Former Students, Colleagues, Administrators, and External (Neutral) Observers," in Research in Higher Education 30:2 (1989), 137-194.
19 Lawrence S. Root, "Faculty Evaluation: Reliability of Peer Assessments of Research, Teaching, and Service," in Research in Higher Education 26:1 (1987).
20 Cohen and McKeachie
21 Feldman
22 Centra 1975; Courtneya, et al.; Feldman
23 Centra 2000; Cohen and McKeachie; Root
24 Centra 2000; Chism; Patricia Hutchings, "The Peer Review of Teaching: Progress, Issues and Prospects," in Innovative Higher Education 20:4 (1996): 221-234; Johnson and Ryan; Lisa R. Lattuca and Jennifer M. Domagal-Goldman, "Using Qualitative Methods to Assess Teaching Effectiveness," in New Directions for Institutional Research 136 (Winter 2007): 81-93; and Peter Seldin, The Teaching Portfolio: A Practical Guide to Improved Performance and Promotion/Tenure Decisions, 3rd Ed. (Bolton, MA: Anker, 2004).
25 Hutchings, ed., tab 4.
26 Cavanagh; Centra 1993; Chism
27 Feldman; Centra 2000; Maria Yon et al., "Evidence of Effective Teaching: Perceptions of Peer Reviewers," in College Teaching 50:3 (2002): 104-110.
28 See Centra, 1975; Courtneya et al.
29 Kathleen M. Quinlan, "Inside the Peer Review Process: How Academics Review a Colleague's Teaching Portfolio," in Teaching and Teacher Education 18 (2002): 1035-1049.
30 Maria Yon, Charles Burnap and Gary Cohut, "Evidence of Effective Teaching: Perceptions of Peer Reviewers," in College Teaching 50:3 (2002): 104-110.
31 See Daniel D. Pratt, "Conceptions of Teaching," in Adult Education Quarterly 42:4 (1992): 203-220; Daniel D. Pratt and Associates, Five Perspectives on Teaching in Adult and Higher Education (Malabar, FL: Krieger, 1998); and www.teachingperspectives.com.
32 Courtneya, et al.
33 Dineke E. H. Tiegelaar, Diana H. J. M. Dolmans, Ineke H. A. P. Wolfhagen, Cees P. M. van der Vleuten, "The Development and Validation of a Framework for Teaching Competencies in Higher Education," in Higher Education 48:2 (Sept. 2004): 253-268. The final list of competencies presented above appears on pages 262-263