Responding to student course evaluations

There is a vast literature in education and related fields detailing the problems with student course evaluations. Evaluations have long been a seemingly necessary burden of teaching that yield questionable data for instructional improvements.

I didn't solve these problems, but I tried something different this past term: I responded to my class about the evaluations. While not a practice unique to online learning, I felt that responding provided a good way to close the communication loop in our remote (Zoom platform) online writing-intensive English course, the Literature of Business (1). 

As usual, after the term concluded and I submitted grades, I received my course evaluation results. I always read them with care and interest, and I assure my students of that several times in an effort to encourage them to take the evaluations seriously. This year, however, I added something else: I wrote them a detailed email responding to the evaluations.

I've been thinking about doing this for years, seeing it as a way of affirming for them that I read and take their comments seriously in the quest to improve my teaching. I had paused, though, on following through for various reasons, mainly because I didn't want it to appear that I was getting the last word on the students. This term, I just went for it, and I think--I hope!--I handled it well.

I bcc'ed the whole class, starting with this:

Dear students from ENGL308 The Literature of Business,
I hope the smoke has cleared from the spring term and you are into your summer plans and courses. I wanted to take a moment to respond to the assessments, ideas, and suggestions you provided in your course evaluations for our class. This is something new I am trying so you, the students, can see that I went through your evaluations carefully and thought about them. As I said, I much appreciate your input, and I want you to know that I hear you.

I noted their strong response rate, in this class 13 of 16 active students in the course. 

I then reviewed their responses to the objective, 1-to-5-scale questions. I explained the scale again and went through each question category, starting with questions about the course and moving to questions about their evaluation of their own performance to finally questions about the instructor (me!). I provided a brief summary of the numerical values in each category, trying to remain neutral in this summary (although I did thank them for what were positive responses about my teaching). 

I then summarized, using an occasional quote for illustration, their subjective comments, especially what they found "beneficial" and what needed improving.

In the end of my message, I focused on these improvement areas, writing, "Overall, these responses indicate that you were satisfied with the class and quite satisfied with my instructional efforts. However, over the summer I must think about two primary comments you made about the course": those two main areas were workload and grading.

Of workload, I wrote, "You view the workload as too heavy—and these were thoughtful comments, not complaints." Of grading, while I took the opportunity to describe how my grading of quizzes and informal writing works, I ultimately wrote, "I must continue to find ways to make the grading in my courses more transparent and fairer."

No student responded to this message, but I didn't expect that. I did want them to feel heard and to know that their comments for improving this course do not go into the void. 

Note:

1) The Literature of Business is a writing intensive course I developed. In the course, we read literary texts with business themes and complement them with non-fiction texts from business venues such as The Harvard Business Review. In most assignments, they write as the characters from the literary texts in professional genres, using the non-business texts to support their position. An example assignment is they read Melville's "Bartleby, The Scrivener" and an HBR article "How to Motivate Your Problem People" by Nigel Nicholson. They must write a letter as one of the characters in the story to a lawyer who is asking for statements about whether the boss is somehow to blame for what happens to Bartleby at the end of the story.

Observing and evaluating online writing courses, part 2

Last July, I wrote some thoughts about peer observations and evaluations of online writing courses (OWCs). I promised -- for you and me -- some further examination of this topic. So, last week, at Computers & Writing at St. John Fisher College in Rochester (a well-run and enjoyable conference -- thanks to the organizers and local hosts) my colleague from Kent State Mahli Mechenbier and I presented about conducting such evaluations.

In my talk I was able to show, I think, some progress on my initial articulation of an Elbowesque “movies-of-the- mind” approach to conducting peer evaluations of OWCs. I opened my session by giving the audience members a “task”: “During my session, take some notes about what happens and then imagine you would turn them into an observation letter you would give to me about the session.” I circled back to the "task" at the end. My point was that observations of things like teaching (or conference talks) are always subjective and related to the relative expertise of both evaluator and evaluatee. By understanding that dynamic, we can avoid the judgment-driven process that has done a bizarrely paradoxical thing: oversimplified evaluations while overcomplicating the evaluation process.

The “how” of this “movies-of-the-mind” approach is straightforward, I said: I simply write a letter to the observed faculty member describing what I see and experience in the course. With an OWC, the whole course is laid out in front of an evaluator, so the evaluation can be broader than the customary one-shot onsite visit. I did mention, though, that although I could see most of the course, I still want the teacher to guide me through, showing me only what they think appropriate.

By using this approach, the evaluator avoids reductive rubrics and metrics and instead simply narrates what they see from a colleague's teaching.

Now, politics do emerge in the process. We conduct these evaluations in the context of the academic hierarchy. Mahli emphasized this in her session, pointing out evaluation “issues” that include rank, power disparities between faculty, lack of a true peer relationship, being unknown to an evaluator, and that the whole process can be viewed as a “waste of time.”

As she does in a lot of her work, Mahli focused on contingent faculty. Especially because so many contingent faculty are asked to teach online, the issue of how such faculty are evaluated and by whom is a big topic, she said. Your campus might use Quality Matters or even (however unlikely) have developed its own approach/methodology of evaluation, but who has the expertise to conduct an evaluation -- especially if all of the online teachers are contingents but the institution requires tenured folks to evaluate!

I ended my talk with a big zoom-out, as the reductive evaluations we do in our classes continue up the ladder until we're creating stupid, standardized test-based metrics of entire schools. Such evaluations have had devastating impact to communities.

We discussed these issues in the direct context of the C&W crowd, and we're going to focus on the administrative side of OWC evaluations in Council of Writing Program Administrators conference in Raleigh in a month and a half.

Observing and evaluating online writing courses

Conducting a course observation/evaluation of an online writing course (OWC), whether peer-to-peer or administrator-to-teacher, presents various challenges. Teaching evaluations in general, or, perhaps it would be more accurate to say the process of conducting teaching observations, can be vexed. You can run into issues ranging from the academic hierarchy to establishing fair criteria

These problems can become more complicated online, especially with OWCs. Evaluators, often administrators, may lack the expertise to "judge" online learning. We have known this about distance learning for a while; Benigno and Trentin said in 2000, “Given the special characteristics of online courses, their quantitative/qualitative evaluation calls for the adoption of specific procedures to assess both the learning process and the participant performance” (1). Our OWI Committee's Position Statement took this problem on, and in a number of Effective Practices we encourage observations specific to OWI:

• Effective Practice 7.11: Online writing teachers should be evaluated/assessed by a peer or supervisor who has similar training and equal or superior abilities/experience in writing instruction generally and OWI particularly.
• Effective Practice 7.12: OWI teacher assessment should occur in the setting and modalities that the teacher uses in the online writing course.
• Effective Practice 7.13: OWI teacher assessment should be engaged as rigorously as—and not more rigorously than—it would be in a similar traditional onsite course.

Quality Matters has a well-regarded review process. But – and I need to talk with my QM certified colleagues and friends more – my understanding is this process and rubric can be challenging to apply to OWCs (although some have made the effort [2]).

Here are some guiding ideas for me when I've observed an OWC or the online component of a hybrid:

"Movie” of the course. First off, my underlying teacher observation strategy, in any modality, is to write a "movie" of the course (I kinda took this from Peter Elbow's "movies of the mind" approach to writing critique  [3]). My letter is just that: Here is what I saw. Ultimately, I’m not the assessor. Someone up the ladder is. I think the idea of a judgment-based evaluation is a problem, so I just say what I see and provide scant editorial commentary. The resulting narrative can be judged by others, and that will help us to avoid stupid teaching observation rubrics that include things like what people are wearing when they teach.

A guided visit. I prefer that the instructor leads me through the course, even if we do that over the phone. Most instructors have been open to providing me with full access, but I'd rather not just wander through unchaperoned. Onsite, the instructor has some control of what I see, and I believe the same should apply online.

The importance of time online. I try to think about the time that both the students and instructor are putting into the work that week. This opens up tricky questions about time and learning/teaching. How much time should someone put into a class? That's tough, and the Carnegie credit is under fire, but once I observed an online course in which the only thing going on one week was that a paper was due. I talked to the instructor about it in terms of an onsite course: What would you be doing all week, sitting in the classroom waiting for students to drop off papers? What would you do with the class time? I try to look at the work of a few students and attempt to get a sense of how long I think it's taking them to read, post, review, etc. Time and space are different online, especially in an asynchronous writing environment.

What does the student see? I try to view the class design and structure from the role of a student, but I don’t obsess over those things. In short, I don’t need pretty, but I do look for function and usability.

There's more to be said about this topic. I have long believed that OWI offers great opportunity for writing instruction. In similar ways, OWCs may offer new ways to observe and understand teaching. In a recent conversation with my colleague Michael Moore from DePaul, we discussed how the “integrity” of observation can be strengthened in an online course, for a simple reason: The whole course is right in front of you. Rather than pretend that a one-shot visit to a classroom one day is good enough for us to say something general about a teacher, we instead can see a broader view of the course and that person's teaching.

Course observations raise good issues that, as I mentioned, I'm going to investigate more over the next few months.

Notes

1) V. Benigno and G. Trentin. The evaluation of online courses. Journal of Computer Assisted Learning. Volume 16, Issue 3, pages 259–270, September 2000.
2) Harkness, S., Soodjinda, D., Hamilton, M., & Bolig, R. (2011, November).  Assessment of a pilot online writing program using the QM Rubric.  Paper presented at the 3rd Annual Quality Matters Conference, Baltimore, MD.
3) In Writing Without Teachers.