Responding to student course evaluations

There is a vast literature in education and related fields detailing the problems with student course evaluations. Evaluations have long been a seemingly necessary burden of teaching that yield questionable data for instructional improvements.

I didn't solve these problems, but I tried something different this past term: I responded to my class about the evaluations. While not a practice unique to online learning, I felt that responding provided a good way to close the communication loop in our remote (Zoom platform) online writing-intensive English course, the Literature of Business (1). 

As usual, after the term concluded and I submitted grades, I received my course evaluation results. I always read them with care and interest, and I assure my students of that several times in an effort to encourage them to take the evaluations seriously. This year, however, I added something else: I wrote them a detailed email responding to the evaluations.

I've been thinking about doing this for years, seeing it as a way of affirming for them that I read and take their comments seriously in the quest to improve my teaching. I had paused, though, on following through for various reasons, mainly because I didn't want it to appear that I was getting the last word on the students. This term, I just went for it, and I think--I hope!--I handled it well.

I bcc'ed the whole class, starting with this:

Dear students from ENGL308 The Literature of Business,
I hope the smoke has cleared from the spring term and you are into your summer plans and courses. I wanted to take a moment to respond to the assessments, ideas, and suggestions you provided in your course evaluations for our class. This is something new I am trying so you, the students, can see that I went through your evaluations carefully and thought about them. As I said, I much appreciate your input, and I want you to know that I hear you.

I noted their strong response rate, in this class 13 of 16 active students in the course. 

I then reviewed their responses to the objective, 1-to-5-scale questions. I explained the scale again and went through each question category, starting with questions about the course and moving to questions about their evaluation of their own performance to finally questions about the instructor (me!). I provided a brief summary of the numerical values in each category, trying to remain neutral in this summary (although I did thank them for what were positive responses about my teaching). 

I then summarized, using an occasional quote for illustration, their subjective comments, especially what they found "beneficial" and what needed improving.

In the end of my message, I focused on these improvement areas, writing, "Overall, these responses indicate that you were satisfied with the class and quite satisfied with my instructional efforts. However, over the summer I must think about two primary comments you made about the course": those two main areas were workload and grading.

Of workload, I wrote, "You view the workload as too heavy—and these were thoughtful comments, not complaints." Of grading, while I took the opportunity to describe how my grading of quizzes and informal writing works, I ultimately wrote, "I must continue to find ways to make the grading in my courses more transparent and fairer."

No student responded to this message, but I didn't expect that. I did want them to feel heard and to know that their comments for improving this course do not go into the void. 

Note:

1) The Literature of Business is a writing intensive course I developed. In the course, we read literary texts with business themes and complement them with non-fiction texts from business venues such as The Harvard Business Review. In most assignments, they write as the characters from the literary texts in professional genres, using the non-business texts to support their position. An example assignment is they read Melville's "Bartleby, The Scrivener" and an HBR article "How to Motivate Your Problem People" by Nigel Nicholson. They must write a letter as one of the characters in the story to a lawyer who is asking for statements about whether the boss is somehow to blame for what happens to Bartleby at the end of the story.

Observing and evaluating online writing courses

Conducting a course observation/evaluation of an online writing course (OWC), whether peer-to-peer or administrator-to-teacher, presents various challenges. Teaching evaluations in general, or, perhaps it would be more accurate to say the process of conducting teaching observations, can be vexed. You can run into issues ranging from the academic hierarchy to establishing fair criteria

These problems can become more complicated online, especially with OWCs. Evaluators, often administrators, may lack the expertise to "judge" online learning. We have known this about distance learning for a while; Benigno and Trentin said in 2000, “Given the special characteristics of online courses, their quantitative/qualitative evaluation calls for the adoption of specific procedures to assess both the learning process and the participant performance” (1). Our OWI Committee's Position Statement took this problem on, and in a number of Effective Practices we encourage observations specific to OWI:

• Effective Practice 7.11: Online writing teachers should be evaluated/assessed by a peer or supervisor who has similar training and equal or superior abilities/experience in writing instruction generally and OWI particularly.
• Effective Practice 7.12: OWI teacher assessment should occur in the setting and modalities that the teacher uses in the online writing course.
• Effective Practice 7.13: OWI teacher assessment should be engaged as rigorously as—and not more rigorously than—it would be in a similar traditional onsite course.

Quality Matters has a well-regarded review process. But – and I need to talk with my QM certified colleagues and friends more – my understanding is this process and rubric can be challenging to apply to OWCs (although some have made the effort [2]).

Here are some guiding ideas for me when I've observed an OWC or the online component of a hybrid:

"Movie” of the course. First off, my underlying teacher observation strategy, in any modality, is to write a "movie" of the course (I kinda took this from Peter Elbow's "movies of the mind" approach to writing critique  [3]). My letter is just that: Here is what I saw. Ultimately, I’m not the assessor. Someone up the ladder is. I think the idea of a judgment-based evaluation is a problem, so I just say what I see and provide scant editorial commentary. The resulting narrative can be judged by others, and that will help us to avoid stupid teaching observation rubrics that include things like what people are wearing when they teach.

A guided visit. I prefer that the instructor leads me through the course, even if we do that over the phone. Most instructors have been open to providing me with full access, but I'd rather not just wander through unchaperoned. Onsite, the instructor has some control of what I see, and I believe the same should apply online.

The importance of time online. I try to think about the time that both the students and instructor are putting into the work that week. This opens up tricky questions about time and learning/teaching. How much time should someone put into a class? That's tough, and the Carnegie credit is under fire, but once I observed an online course in which the only thing going on one week was that a paper was due. I talked to the instructor about it in terms of an onsite course: What would you be doing all week, sitting in the classroom waiting for students to drop off papers? What would you do with the class time? I try to look at the work of a few students and attempt to get a sense of how long I think it's taking them to read, post, review, etc. Time and space are different online, especially in an asynchronous writing environment.

What does the student see? I try to view the class design and structure from the role of a student, but I don’t obsess over those things. In short, I don’t need pretty, but I do look for function and usability.

There's more to be said about this topic. I have long believed that OWI offers great opportunity for writing instruction. In similar ways, OWCs may offer new ways to observe and understand teaching. In a recent conversation with my colleague Michael Moore from DePaul, we discussed how the “integrity” of observation can be strengthened in an online course, for a simple reason: The whole course is right in front of you. Rather than pretend that a one-shot visit to a classroom one day is good enough for us to say something general about a teacher, we instead can see a broader view of the course and that person's teaching.

Course observations raise good issues that, as I mentioned, I'm going to investigate more over the next few months.

Notes

1) V. Benigno and G. Trentin. The evaluation of online courses. Journal of Computer Assisted Learning. Volume 16, Issue 3, pages 259–270, September 2000.
2) Harkness, S., Soodjinda, D., Hamilton, M., & Bolig, R. (2011, November).  Assessment of a pilot online writing program using the QM Rubric.  Paper presented at the 3rd Annual Quality Matters Conference, Baltimore, MD.
3) In Writing Without Teachers.

OWI: Opportunities in medium, genre, volume

I have another blogging life, and this month those lives weirdly intersect, as I'm building here on my post for my blog Virtual Children (which I've been keeping on the Website When Falls the Coliseum, for several years). I posted there about another blog article written by a friend of mine on the Website 11trees; his smart post describes how much he writes in a work day, what he called just an average writing day for a "knowledge worker." In a one-day diary, he calculates he wrote about 2,500 words, saying, "We write more words every day than many college or high school students write in an entire term." 

His post is so smart for many reasons, particularly because it spurs great conversations about genre, medium, and writing volume. In his log, the writer describes all kinds of writing he did during his "writing day." At 5 am he's emailing; he works on PowerPoints and text messages; he drafts notes and other materials in/through word processing programs and different collaboration software. He's composing collaboratively with people around the world, planning, drafting, and putting on finishing touches.

If you teach writing, you have probably thought a lot about how and how much people write. I think the generation we’re educating now interacts with each other through writing more than any generation ever

But our teaching to prepare students for their writing lives often doesn’t account for the rhetorical complexity, the varied genres, and the sheer volume of the writing they will do. In fact, in many cases, writing instruction looks pretty similar to the way it looked many decades ago.

I remember in my earliest moments of interest in OWI that I felt that digital instructional settings opened new avenues for teaching and learning writing. In OWCs, students compose in multi-audience environments that are close to writing they will do in their professional lives -- and these environments are simply inherent parts of the course. Also, it was evident that electronic writing spaces allow teachers to assign lots of informal and exploratory writing, providing many opportunities for students to engage in meta-reflection about their composing processes. 

To me, ed tech tools are not exclusive to online instruction -- I mean, you can use digital tools with any type of course -- but the digital environment of OWI does seem to lead teachers to explore tech tools, so they will find new ways to interact with and even assess student writing that make the whole process more productive. For instance, 11trees is a great example of a tool that makes it easier for teachers to respond to student writing.

As I’ve described before, a brief analysis of my own 10-week, quarter-system classes showed  students were writing some 15,000 words through message boards and electronic journals. I was proud when I found that out and still am; I feel they had a challenging writing experience. But if you think about that number in comparison with the 2,500 daily words of a "knowledge worker,” you realize I might need to significantly up the workload yet if I'm going to responsibly teach writing that prepares students for what they’ll face at the next stage.

By the way, I'm not advocating a kind of mindless kowtowing to the “real world.” I don't want us to dispense of genres/modes like the exploratory essay or writing to learn. But the limitations of "school writing" get exposed in productive ways in online writing courses. The only game in town need not be "papers" that don't resemble the short, high-stakes writing that they do. We can put them in posting and sharing situations that allow us to better teach them what they will do -- and are doing -- in other parts of their lives.

This is a major component of the promise I see of OWI. Perhaps we need to do more in our research to understand not just the kinds but the amount of writing that take place in various professions.

Rubric for evaluating message board posts

"But how do I evaluate message board posts?"

That's a question I'm often asked when talking with colleagues about using the flexible, user-friendly technology of message boards in their courses. For some time, I've been refining a rubric for discussion posts. Below, I share that rubric.

It's unlikely you'll grade every post with the actual rubric, since students will write so many message board posts each term. The idea is to share this with students so the evaluation of this key course component is transparent, opening up ways of having a dialogue about course assessment.

A few quick notes about the rubric. It has 12 categories. The bottom eight will look mostly familiar to you. They are based on writing areas like purpose, organization, evidence, and mechanics, and each has five levels of performance, correlating with a grade. But the rubric is specifically for asynchronous, dialogic writing, and the criteria reflect this. For instance, the "Audience and context" criterion is geared toward how the poster/post interacts with others; that helps me encourage conversation. "Originality/creativity" is because I overtly want to encourage students to embrace risk and creativity in their writing.

The first four categories look somewhat different. These are binary categories, for things that I view as either "it's there or it ain't." They help me define things like timeliness and communicate my interest in ethics.

Communicate. The rubric, as I said, is designed to help me communicate with students, providing us all with a way to share an understanding of expectations in this environment.

General criteria

Gross mistakes. Either "There are no gross mistakes in the post" or "There are excessive errors or some other gross mistakes that cause readers to doubt the effort/expertise of the document’s creator; these posts score no better than 60%"

Ethics. Either "The post appears ethical, including its use of evidence" or "Some aspects of the post are unethical; for example, it uses inappropriate evidence (including plagiarism); recommends morally questionable practices; attacks others; or uses deceitful data; these posts may fail"

Timeliness. Either "The post is on time" or "Late post; late posts lose 20% of their grade for each week they are late"

Length: Either "Post length is appropriate; longer posts are welcomed" or "Post is too short; short posts lose 20% of their grade; excessively short posts count only as “peeps”"

Writing criteria

Purpose/Main idea/Focus
A: Clear main idea that raises an excellent, focused point
B: Clear main idea that raises an interesting point
C: Loses focus; main idea is questionable
D: Why did the writer create this post?; main idea seems uninteresting and perhaps even unreasonable
F: Main idea is unfocused and unclear; off topic

Organization
A: Excellent organization; clear topic sentences; transitions between ideas are handled well
B: Organized effectively but could be refined/ tightened a bit (better topic sentences, transitions, etc.)
C: Adequately organized; needs better division between ideas.Note: One- paragraph primary posts receive a maximum of B
D: Disorganized; little coherent structure; confusing
F: Completely disorganized

Evidence
A: Relevant, correctly cited evidence; quotes from posts
B: Relevant evidence; some citing issues
C: Adequate evidence; citations are wrong
D: Evidence not used or used incorrectly
F: No evidence

Audience and context
A: Clearly connected to other posts and conversations
B: Relevant to other posts and conversations
C: Not clearly connected to other posts; unclear if writer read other posts; repetitive; does not build the conversation much
D: Highly repetitive; not connected to other posts and conversations; may attack other posters; does not build the conversation at all
F: Irrelevant to conversation; makes no effort to connect with audience; flaming

Style
A: Highly engaging, distinct writing style; solid sentences and word choice; may take productive risks
B: Solid writing style; writer could have written more clearly/more concisely
C: Some of the writing is awkward and clumsy; weak word choice or unsophisticated sentence structure
D: Writing is awkward, repetitive, and/or wordy; the writing was not engaging
F: Writing style is inadequate for a college-level assignment

Grammar and mechanics
A: Few errors, if any; writer shows considerable mastery of the language
B: Some grammatical/ mechanical errors, but they do not interfere with the reader’s understanding of post
C: Numerous errors that interfered with the reader’s understanding of the post
D: Many errors that made the post difficult to understand; reader questions writer’s credibility and skill
F: Post is filled with errors; reader doubts writer’s competency

Originality/ creativity
A: Highly original or creative; may take a productive writing risk
B: Some originality or creativity, but doesn’t push the limits
C: No real originality
D: Repeats other ideas or posts’ reader may not have read other posts
F: Blatantly copies other posts

Understanding of course material
A: Clearly demonstrates understanding of course material
B: Good understanding of course material
C: Some of the course material details are incorrect
D: Low understanding of course material
F: Poor understanding of course material; some errors