Tuesday, May 31, 2016

Observing and evaluating online writing courses, part 2

Last July, I wrote some thoughts about peer observations and evaluations of online writing courses (OWCs). I promised -- for you and me -- some further examination of this topic. So, last week, at Computers & Writing at St. John Fisher College in Rochester (a well-run and enjoyable conference -- thanks to the organizers and local hosts) my colleague from Kent State Mahli Mechenbier and I presented about conducting such evaluations.

In my talk I was able to show, I think, some progress on my initial articulation of an Elbowesque “movies-of-the- mind” approach to conducting peer evaluations of OWCs. I opened my session by giving the audience members a “task”: “During my session, take some notes about what happens and then imagine you would turn them into an observation letter you would give to me about the session.” I circled back to the "task" at the end. My point was that observations of things like teaching (or conference talks) are always subjective and related to the relative expertise of both evaluator and evaluatee. By understanding that dynamic, we can avoid the judgment-driven process that has done a bizarrely paradoxical thing: oversimplified evaluations while overcomplicating the evaluation process.

The “how” of this “movies-of-the-mind” approach is straightforward, I said: I simply write a letter to the observed faculty member describing what I see and experience in the course. With an OWC, the whole course is laid out in front of an evaluator, so the evaluation can be broader than the customary one-shot onsite visit. I did mention, though, that although I could see most of the course, I still want the teacher to guide me through, showing me only what they think appropriate.

By using this approach, the evaluator avoids reductive rubrics and metrics and instead simply narrates what they see from a colleague's teaching.

Now, politics do emerge in the process. We conduct these evaluations in the context of the academic hierarchy. Mahli emphasized this in her session, pointing out evaluation “issues” that include rank, power disparities between faculty, lack of a true peer relationship, being unknown to an evaluator, and that the whole process can be viewed as a “waste of time.”

As she does in a lot of her work, Mahli focused on contingent faculty. Especially because so many contingent faculty are asked to teach online, the issue of how such faculty are evaluated and by whom is a big topic, she said. Your campus might use Quality Matters or even (however unlikely) have developed its own approach/methodology of evaluation, but who has the expertise to conduct an evaluation -- especially if all of the online teachers are contingents but the institution requires tenured folks to evaluate!

I ended my talk with a big zoom-out, as the reductive evaluations we do in our classes continue up the ladder until we're creating stupid, standardized test-based metrics of entire schools. Such evaluations have had devastating impact to communities.

We discussed these issues in the direct context of the C&W crowd, and we're going to focus on the administrative side of OWC evaluations in Council of Writing Program Administrators conference in Raleigh in a month and a half.

Labels: , ,

Monday, March 28, 2016

Multilingual students as online course moderators

I use student moderators in my online and hybrid courses, as I described here a few years ago. Through time, I think I have improved at defining moderation for my students. I now send them a message like this:

Thank you for signing up to moderate next week. Moderating is not difficult, but it does take some extra work for the moderators that week. I think many of our moderators for the most part have done a good job this term, so you know. 

As moderator, please do NOT start a new thread. Simply post early in the week in one of the existing threads letting everyone know that you will be moderating that thread. Note that you each only need to work in one of the threads in this week’s forum.

You can moderate the conversation in any way you like, but please do the following:
  • As I said, write a short post letting everyone know that you'll be moderating the conversation on your thread. Do this early in the week.
  • Keep up with what's being written.
  • Try to move the conversation forward with questions and new ideas. Basically, your goal is to build the conversation on the thread.
  • **At the end of the week, please post a message summarizing the activity on your thread and your participation on it as moderator.**
Please note that moderating is a 15-point grade and that your moderator posts are separate from your other graded Discussion posts this week. You still must fulfill the other posting requirements for the week.

Again, if you have any questions, please let me know.

As you can see, moderating is a graded assignment. Many student moderators do a superb job, including in their end-of-week wrap-up of their thread. (Some moderators in the first-year writing course I just taught did such a great job that I felt almost intrusive and unnecessary getting involved with their threads.)

Moderating provides teaching advantages, including leveraging teaching time, but, more importantly, it provides many learning opportunities. I wonder if it might particularly help multilingual/ESL students.

There is some debate about the effectiveness of online environments for multilingual writing students. Recently, a multilingual student in my course was straightforward and articulate in describing how the moderating experience helped her in the course and with her writing. In her end-of-week summary (which I quote as is here), she wrote that “as a non-native speaker student, I should take every chance to read, as a way to improve my thinking in English. But in the previous weeks, I failed to force myself to read all the posts because some of them are really long and hard to understand. But as a moderater, it's different. You have to read all the posts and understand them, thun you can ask questions to encourage the discussion going on.”

In this brief metacommentary, the student emphasizes the careful reading and writing she had to do as moderator. She, to me, makes it clear that moderating created a productive pressure for her.

The dialogic exchange of the discussion board and the broadening of audience to include fellow students seem to be what is key here: The inherent written conversation may put productive pressure on students that helps them, as this student says, not just with reading and writing but with thinking through -- rhetorically -- their communicative role in the community of our class.

This is one story, of course, but it may help us think specifically about how moderating -- and similar assignments/roles -- encourages our multilingual learners.

Labels: , , , ,

Friday, January 29, 2016

OWI Committee’s activities at CCCC in Houston

This year at CCCC in Houston, our Committee for Effective Practices in Online Writing Instruction will be in the final stage of a three-year plan following the publication of our Position Statement. During the past two 4Cs, we've chosen a theme/focus for our many conference activities. Year one was Institution Matters. Year two was Faculty Matters. This year we will be focusing on Student Matters.
It's important and appropriate that we're concluding with student-focused concerns about OWI. Our committee and its members have several projects underway to help sharpen the field's understanding about the student experience in OWI so we can move the overall OWI research agenda forward. A big piece is our recently piloted survey about student OWI experiences, which I wrote about a few months ago in this space.
If you're headed to Houston and interested in OWI, we have a number of activities planned for CCCC that we welcome you to take part in:
  • On Wednesday, April 6 from 9:00 am to 12:30 pm, we will be offering the half-day workshop, “Taking Action with Student Retention and Success: An OWI 'Student Matters' Workshop.”
  • On Thursday, April 7 from 3:15 to 4:30, our committee will offer a panel, “CCCC Committee for Effective Practices in Online Writing: Student Matters”(I'm part of this panel).
  • Friday night, we’ll be having our Special Interest Group (SIG) meeting from 6:30 to 7:30
We’ll also be having our closed committee meeting on Friday morning.

The Student Matters theme will run throughout the conference for us. All these student-focused activities and the accompanying dialogue with our colleagues should help us considerably with the next major tasks our committee faces, including revising the Position Statement.

Labels: , , ,

Monday, November 30, 2015

Breaking the ice in your online writing course

The first week of an OWC presents teachers with some big opportunities that they can capitalize on with a good icebreaker.

An icebreaker/introduction activity can help in any course, but it's particularly valuable in the writing-focused OWC.

For one, icebreakers can be integral to constructing the author-reader relationships that drive the course; I think students need to learn quickly who the audience members (including me) are who will read them over the next few months.

Icebreakers provide students with an early low-stakes opportunity to develop their voices. Who are they in this online course? Even simple things like how they sign off on this initial post can help create the personalities the rest of the class will "see" and interact with.

Icebreakers also give teachers opportunities to develop voice and to model message board behavior. More on that in a moment.

Of course, in a broader sense, an icebreaker can help establish course community. In our committee’s A Position Statement of Principles and Example Effective Practices for Online Writing Instruction, Effective Practice 11.2 states: “OWC teachers should develop course community early by employing ‘icebreakers’ and other activities that make use of the LMS and that engage student writing.” Students can learn about each others' interests and make person-to-person connections while learning the course interface.

I use various icebreakers. Here is one that I think has provided good first-week results:
SUBJ: Who are you?
Dear members of English 102,
Make sure you read and follow the directions below carefully.
I'd like to get to know you a bit and, perhaps more importantly, for you to get to know each other. So, could you please let us know a few things about yourself?:
Tell us who you are and where you're from.
-What is your major? What are you interested in pursuing as a career?
-Describe one thing--and it can be anything--that you have that helps demonstrate an important aspect of you. Think of it this way: if we were in a face-to-face classroom and I asked you to bring something that demonstrates an important aspect of you, what would you bring and why?
-What are some topics that you like to debate or that you have strong feelings about? (Note: You don't even have to reveal which side you're on; just tell us what the topic/subject is.)

Please provide your email address, and sign this post the way you would like us to address you (i.e., do you go by a nickname?).
Looking forward to hearing about you,
Prof. Warnock

Some online teachers might resist the idea of going retro and asking students what they would bring to an onsite class, but this simple request has worked well. In their responses to each other, I’m struck by how many of them build connections that they sustain throughout the term. Once a student talked about loving elephants and attached a picture of an elephant figurine. Surprise, surprise, another student also expressed a love of elephants. Friends for life!

I require and grade the icebreaker post and a secondary response post (1), so they also get a sense of how the course grading works (it's all very low-stakes).

I do work hard during week 1, as I respond to every student’s icebreaker post. I think it’s important to establish connection from the go with them. I also, in this textual environment, want to establish myself as a real person and model how to engage on discussions.

So I write to them. You would bring your Philadelphia Eagles jersey? I share your pain. Live in Blackwood, South Jersey? I'm from near there. Bring in your Led Zeppelin IV vinyl? That's the first album I ever bought ($2 at the Berlin Auction). Are you an engineering major? That’s how I started out. Bring in pictures from your trip to India? I’d love to go there some day.

You no doubt could come up with cooler ideas (2), but any activity that gives students a chance to write about themselves helps them establish themselves in week one of an OWC – and allows you to do the same.

1) I discussed my grading approach a few years ago in "Rubric for evaluating message board posts."
2) A simple search for "icebreakers" will provide you with many good suggestions.

Labels: , ,

Tuesday, September 29, 2015

Learning about the student experience in OWI

What are students doing and experiencing in online writing courses (OWCs)? How do they feel about learning in their OWCs? What do we really know about this particular learning context/environment? Our Committee for Effective Practices in Online Writing Instruction (OWI) (I'm co-chair) is eager to learn more and is engaged in several research projects and initiatives to answer questions like these.

Since the publication of our committee's Position Statement of Principles and Example Effective Practices for OWI, our group has structured its efforts along the lines of a three-year plan: year one was “Institutional Matters”; year two was “Faculty Matters”; this year is “Student Matters.”

For me, I have been curious about the fact that many teachers might not have a good anchor for thinking about our OWI students' experiences for a simple reason: Many teachers have themselves never taken an online course. So, simply put, in onsite instruction, while teachers can draw on their own personal histories when developing their pedagogy, online, they’re often adrift because they don’t have the background to know what a course “looks” and "feels" like to an online learner.

Onsite, a teacher might remember what it was like to sit in the back of the class or receive praise or work next to an annoying (or charming) partner or sit in a room with a hangover – uh, I mean head cold. But if you haven’t taken an online course, it could understandably be difficult to envision a student’s experience with discussion boards or the pressure of taking a timed quiz in a dorm room or how it feels to speak up using a microphone and Webcam in an online synchronous classroom. What are those activities like?

Our committee wants to help fill in the gaps in this knowledge, especially clarifying students’ experiences in terms of access and disability. Our primary initiative is the OWI Student Survey. Several of my colleagues have worked hard to create a series of questions to help us understand what is happening for students in OWCs. This fall, we’re piloting the survey, but if any of you are interested in sharing it with your students, please reach out to me.

The survey process and results will be part of our workshop and panel presentation at next year's CCCC in Houston. At CCCC, we'll also be exploring several other aspects of students and OWI.

All of this work dovetails with a project I’m engaged in right now with one of my own students – I’m going to hold off on disclosing exactly what we’re doing for another month or so, until we see where the project will ultimately lead, but we're basically working on co-authoring an extended narrative of an online course as seen through the eyes of not just the teacher, but the student. This is another effort to provide a view of how the online writing instructional environment looks -- through student eyes.

Labels: , , ,

Friday, July 31, 2015

Observing and evaluating online writing courses

Conducting a course observation/evaluation of an online writing course (OWC), whether peer-to-peer or administrator-to-teacher, presents various challenges. Teaching evaluations in general, or, perhaps it would be more accurate to say the process of conducting teaching observations, can be vexed. You can run into issues ranging from the academic hierarchy to establishing fair criteria

These problems can become more complicated online, especially with OWCs. Evaluators, often administrators, may lack the expertise to "judge" online learning. We have known this about distance learning for a while; Benigno and Trentin said in 2000, “Given the special characteristics of online courses, their quantitative/qualitative evaluation calls for the adoption of specific procedures to assess both the learning process and the participant performance” (1). Our OWI Committee's Position Statement took this problem on, and in a number of Effective Practices we encourage observations specific to OWI:
  • Effective Practice 7.11: Online writing teachers should be evaluated/assessed by a peer or supervisor who has similar training and equal or superior abilities/experience in writing instruction generally and OWI particularly.
  • Effective Practice 7.12: OWI teacher assessment should occur in the setting and modalities that the teacher uses in the online writing course.
  • Effective Practice 7.13: OWI teacher assessment should be engaged as rigorously as—and not more rigorously than—it would be in a similar traditional onsite course.
Quality Matters has a well-regarded review process. But – and I need to talk with my QM certified colleagues and friends more – my understanding is this process and rubric can be challenging to apply to OWCs (although some have made the effort [2]).

Here are some guiding ideas for me when I've observed an OWC or the online component of a hybrid:

"Movie” of the course. First off, my underlying teacher observation strategy, in any modality, is to write a "movie" of the course (I kinda took this from Peter Elbow's "movies of the mind" approach to writing critique  [3]). My letter is just that: Here is what I saw. Ultimately, I’m not the assessor. Someone up the ladder is. I think the idea of a judgment-based evaluation is a problem, so I just say what I see and provide scant editorial commentary. The resulting narrative can be judged by others, and that will help us to avoid stupid teaching observation rubrics that include things like what people are wearing when they teach.

A guided visit. I prefer that the instructor leads me through the course, even if we do that over the phone. Most instructors have been open to providing me with full access, but I'd rather not just wander through unchaperoned. Onsite, the instructor has some control of what I see, and I believe the same should apply online.

The importance of time online. I try to think about the time that both the students and instructor are putting into the work that week. This opens up tricky questions about time and learning/teaching. How much time should someone put into a class? That's tough, and the Carnegie credit is under fire, but once I observed an online course in which the only thing going on one week was that a paper was due. I talked to the instructor about it in terms of an onsite course: What would you be doing all week, sitting in the classroom waiting for students to drop off papers? What would you do with the class time? I try to look at the work of a few students and attempt to get a sense of how long I think it's taking them to read, post, review, etc. Time and space are different online, especially in an asynchronous writing environment.

What does the student see? I try to view the class design and structure from the role of a student, but I don’t obsess over those things. In short, I don’t need pretty, but I do look for function and usability.

There's more to be said about this topic. I have long believed that OWI offers great opportunity for writing instruction. In similar ways, OWCs may offer new ways to observe and understand teaching. In a recent conversation with my colleague Michael Moore from DePaul, we discussed how the “integrity” of observation can be strengthened in an online course, for a simple reason: The whole course is right in front of you. Rather than pretend that a one-shot visit to a classroom one day is good enough for us to say something general about a teacher, we instead can see a broader view of the course and that person's teaching.

Course observations raise good issues that, as I mentioned, I'm going to investigate more over the next few months.


1) V. Benigno and G. Trentin. The evaluation of online courses. Journal of Computer Assisted Learning. Volume 16, Issue 3, pages 259–270, September 2000.
2) Harkness, S., Soodjinda, D., Hamilton, M., & Bolig, R. (2011, November).  Assessment of a pilot online writing program using the QM Rubric.  Paper presented at the 3rd Annual Quality Matters Conference, Baltimore, MD.
3) In Writing Without Teachers.

Labels: , , , , , , ,

Sunday, May 31, 2015

The Provoker

An aspect of teaching an OWC that I really enjoy is taking on the role of what I call “The Provoker,” a contradictory anti-voice, in the class written discussions.

I’ve discussed this before, but coming off my winter online persuasive writing course, I wanted to provide more detail about a practice while pedagogically fun is I believe also a key part of my students’ writing education in my courses.

The Provoker is a rhetorically edgy, devil’s advocate-type voice in the class discussions. I post using an alias; for some years, I’ve used the alter-ego “Dr. Logoetho.” I always include a few Provoker threads in an OWC. Sometimes I, Prof. Warnock, introduce our “guest” and facilitate his posts; other times I log into the CMS using a guest access so posts don’t actually have my named attached to them. I do make it clear to students that we’re playing a rhetorical game here (only once was a student confused about this). So it’s me – but it’s not.

Dr. Logoetho takes extreme stances:

Dear students in English 102:
I’m tired of hearing everyone complain about the cost of college. Considering how much people benefit earnings-wise over the course of their lifetime based upon the degree they have earned (see http://www.acinet.org/acinet/finaidadvisor/earnings.asp?nodeid=21), I argue that college should cost MORE money than it does now. I would ask if you agree, but how could you not?: I make a very reasonable and logical argument.
Yep, that’s what I think,
Dr. Logoetho

He writes with more than a little cheek, taunting the students: 

Dear students in Prof. Warnock’s English 102 course,
Kind of nice to see you again. I have something to say this week about Wikipedia. I think that Wikipedia is a completely useless site except as entertainment. No one should ever use it for anything of importance or for real research. Wikis are an unreliable way to build true information. I think the positives of Wikipedia are grossly exaggerated and are only promoted by people who have something to gain from Wikipedia or who don’t understand the informational value of the Web!

There. If you want to take me on, at least show a little argument savvy – Prof. Warnock gave you access to some materials this week, I think – and use some evidence.
I doubt you're up to the task,
Dr. Logoetho

With me, the official Professor, out of the picture, I find that students write with verve and passion while also composing solid, interesting arguments. They use evidence. They have to avoid logical fallacies, particularly ad hominem, when dealing with an often disrespectful interlocutor. They have to think through written, and sometimes emotional, argumentation. And they often work together, building off each other to take down Dr. Logoetho.

Especially in a persuasive writing-type class, I want students to have smart, authentic arguments, but a course can be a difficult place for that. Provoker threads allow students to write and argue without worrying about offending classmates or dealing with the authority-laden quagmire of “debating the teacher.” Term after term, I feel Provoker threads bring out some of their best writing.
It’s enjoyable for me too. I’m competitive and debate them head-on. I even get to bust on myself, old “Prof. Warnock.” Here’s Dr. Logoetho replying to their rebuttals to his Wikipedia “argument”; you can see that, throughout, I cite them directly (pseudonyms below) to show the power of their arguments:

SUBJ: Dr. Logoetho mad. Very mad
Dear students of Prof. Warnock,
You all think you’re so smart. Blair asks, “I only want to know how you can decide what source is valid compared to another source.” I’ll tell you: Peer review! Editors! Experts! Let the experts be the gatekeepers. Let’s get scholarly. Pete writes, “Who is to say that the information on this web page is not scholarly as any other?” Well, just about everyone, Pete! Read Praxis about the definition of a scholarly source. As Wendy points out (but in a nicer way than me; everyone is nicer than me), any buffoon can be a Wikipedia editor (in fact, even poor Prof. Warnock is a Wikipedia editor). Then Janet says, “Scholarly articles and sources are questioned all the time,” – name one time this happened! Let’s stop all this nonsense![...]

From the above, as you might imagine, the students usually “defeat” Dr. Logoetho by week’s end. He slinks away, and Prof. Warnock pops in and congratulates them. It’s fun. And I think it’s also good writing instruction.

Labels: , , , , , ,