Observing and evaluating online writing courses, part 2

Last July, I wrote some thoughts about peer observations and evaluations of online writing courses (OWCs). I promised -- for you and me -- some further examination of this topic. So, last week, at Computers & Writing at St. John Fisher College in Rochester (a well-run and enjoyable conference -- thanks to the organizers and local hosts) my colleague from Kent State Mahli Mechenbier and I presented about conducting such evaluations.

In my talk I was able to show, I think, some progress on my initial articulation of an Elbowesque “movies-of-the- mind” approach to conducting peer evaluations of OWCs. I opened my session by giving the audience members a “task”: “During my session, take some notes about what happens and then imagine you would turn them into an observation letter you would give to me about the session.” I circled back to the "task" at the end. My point was that observations of things like teaching (or conference talks) are always subjective and related to the relative expertise of both evaluator and evaluatee. By understanding that dynamic, we can avoid the judgment-driven process that has done a bizarrely paradoxical thing: oversimplified evaluations while overcomplicating the evaluation process.

The “how” of this “movies-of-the-mind” approach is straightforward, I said: I simply write a letter to the observed faculty member describing what I see and experience in the course. With an OWC, the whole course is laid out in front of an evaluator, so the evaluation can be broader than the customary one-shot onsite visit. I did mention, though, that although I could see most of the course, I still want the teacher to guide me through, showing me only what they think appropriate.

By using this approach, the evaluator avoids reductive rubrics and metrics and instead simply narrates what they see from a colleague's teaching.

Now, politics do emerge in the process. We conduct these evaluations in the context of the academic hierarchy. Mahli emphasized this in her session, pointing out evaluation “issues” that include rank, power disparities between faculty, lack of a true peer relationship, being unknown to an evaluator, and that the whole process can be viewed as a “waste of time.”

As she does in a lot of her work, Mahli focused on contingent faculty. Especially because so many contingent faculty are asked to teach online, the issue of how such faculty are evaluated and by whom is a big topic, she said. Your campus might use Quality Matters or even (however unlikely) have developed its own approach/methodology of evaluation, but who has the expertise to conduct an evaluation -- especially if all of the online teachers are contingents but the institution requires tenured folks to evaluate!

I ended my talk with a big zoom-out, as the reductive evaluations we do in our classes continue up the ladder until we're creating stupid, standardized test-based metrics of entire schools. Such evaluations have had devastating impact to communities.

We discussed these issues in the direct context of the C&W crowd, and we're going to focus on the administrative side of OWC evaluations in Council of Writing Program Administrators conference in Raleigh in a month and a half.

The Provoker

An aspect of teaching an OWC that I really enjoy is taking on the role of what I call “The Provoker,” a contradictory anti-voice, in the class written discussions.

I’ve discussed this before, but coming off my winter online persuasive writing course, I wanted to provide more detail about a practice while pedagogically fun is I believe also a key part of my students’ writing education in my courses.

The Provoker is a rhetorically edgy, devil’s advocate-type voice in the class discussions. I post using an alias; for some years, I’ve used the alter-ego “Dr. Logoetho.” I always include a few Provoker threads in an OWC. Sometimes I, Prof. Warnock, introduce our “guest” and facilitate his posts; other times I log into the CMS using a guest access so posts don’t actually have my named attached to them. I do make it clear to students that we’re playing a rhetorical game here (only once was a student confused about this). So it’s me – but it’s not.

Dr. Logoetho takes extreme stances:

Dear students in English 102:

I’m tired of hearing everyone complain about the cost of college. Considering how much people benefit earnings-wise over the course of their lifetime based upon the degree they have earned (see http://www.acinet.org/acinet/finaidadvisor/earnings.asp?nodeid=21), I argue that college should cost MORE money than it does now. I would ask if you agree, but how could you not?: I make a very reasonable and logical argument.

Yep, that’s what I think,
Dr. Logoetho

He writes with more than a little cheek, taunting the students: 

Dear students in Prof. Warnock’s English 102 course,

Kind of nice to see you again. I have something to say this week about Wikipedia. I think that Wikipedia is a completely useless site except as entertainment. No one should ever use it for anything of importance or for real research. Wikis are an unreliable way to build true information. I think the positives of Wikipedia are grossly exaggerated and are only promoted by people who have something to gain from Wikipedia or who don’t understand the informational value of the Web!

There. If you want to take me on, at least show a little argument savvy – Prof. Warnock gave you access to some materials this week, I think – and use some evidence.

I doubt you're up to the task,

Dr. Logoetho

With me, the official Professor, out of the picture, I find that students write with verve and passion while also composing solid, interesting arguments. They use evidence. They have to avoid logical fallacies, particularly ad hominem, when dealing with an often disrespectful interlocutor. They have to think through written, and sometimes emotional, argumentation. And they often work together, building off each other to take down Dr. Logoetho.

Especially in a persuasive writing-type class, I want students to have smart, authentic arguments, but a course can be a difficult place for that. Provoker threads allow students to write and argue without worrying about offending classmates or dealing with the authority-laden quagmire of “debating the teacher.” Term after term, I feel Provoker threads bring out some of their best writing.
It’s enjoyable for me too. I’m competitive and debate them head-on. I even get to bust on myself, old “Prof. Warnock.” Here’s Dr. Logoetho replying to their rebuttals to his Wikipedia “argument”; you can see that, throughout, I cite them directly (pseudonyms below) to show the power of their arguments:

SUBJ: Dr. Logoetho mad. Very mad

Dear students of Prof. Warnock,

You all think you’re so smart. Blair asks, “I only want to know how you can decide what source is valid compared to another source.” I’ll tell you: Peer review! Editors! Experts! Let the experts be the gatekeepers. Let’s get scholarly. Pete writes, “Who is to say that the information on this web page is not scholarly as any other?” Well, just about everyone, Pete! Read Praxis about the definition of a scholarly source. As Wendy points out (but in a nicer way than me; everyone is nicer than me), any buffoon can be a Wikipedia editor (in fact, even poor Prof. Warnock is a Wikipedia editor). Then Janet says, “Scholarly articles and sources are questioned all the time,” – name one time this happened! Let’s stop all this nonsense![...]

From the above, as you might imagine, the students usually “defeat” Dr. Logoetho by week’s end. He slinks away, and Prof. Warnock pops in and congratulates them. It’s fun. And I think it’s also good writing instruction.

In an OWC, reading is an inherent part of the experience

What do your students read in their writing courses, especially their first-year courses? And why are they reading those things?

As a teacher, I always assign a lot of readings in my courses. In fact, my process of designing a course usually starts with the readings; then I construct the assignments and then move on to the rest of the course.

For a variety of reasons, I’ve started wondering about this approach of using lists of prescribed readings in my courses. Particularly in an online writing course (OWC), is this really what I need to do?

This year, at Drexel we hired Irvin Peckham to direct our First-Year Writing Program. Irvin has brought some different approaches to teaching writing. He's focused on student attitudes toward writing, for one thing, and he also has a scaled-down approach to syllabi, which includes sparer required writings in these first-year courses. I’m oversimplifying his thoughts about this – check out his blog Personal Writing in the Classroom -- but he has led me to consider that in a well-run writing course, students’ earnest reading of each others' texts is plenty of great reading work for the term.

Yep. And this is particularly so in an OWC. When I looked closely at things this way, it struck me that students were doing a considerable amount of reading in my OWC courses. I’ve focused on the amounts of writing they are doing, but because so much of that writing is open to all course participants, they also read a great deal. My colleague Beth Hewett makes this argument in her new book Reading to Learn and Writing to Teach: Literacy Strategies for Online Writing Instruction. She points out that good OWI practices include a strong course culture of reading and re-reading.

Research has backed up that the reading in OWCs is quite different compared to onsite first-year writing courses; June Griffin and Deborah Minter compared the reading in fully online vs. onsite writing courses and found that the reading load online was more than 2.75 times greater. (1)

Let’s not get into the “do they read it all?” question right now, because the assumption that students ever experience everything in any class modality is flawed. (As I always say, in my many experiences observing onsite classes, I’m struck by students who aren't paying a lick of attention even though there’s some stranger in a tie sitting directly behind them.) My point here is that the literacy experience in an OWC is intense and rigorous by its very nature. Students in the course will engage in a lot of engaged reading on discussion boards and through peer reviews, reading that often turns directly into writing. They are casting their eyes over many, many words in powerfully engaged and often transactional ways.

Of course, there’s still room for those texts that I like to assign, but in an OWC, you don’t have to overburden them (or yourself) with structured reading. Really, I should have known. Writing courses are different from other courses because their syllabi emerge through the organic development of the students’ writing within them. Online, this is even more so.

Note:

1) Griffin, June & Deborah Minter. (2013). The rise of the online writing classroom. College Composition and Communication 65(1), 140-61.

Peer review in OWI: Slowing it down

I’ve written here before (but how time flies since 2008 and even 2012!) about how peer review and OWI intersect so well. I keep returning to this topic, because, for so many of us who teach writing, peer review is the thing: a central part of writing instruction.

Simply put, going online accentuates and highlights that centrality. If you teach an online writing course (OWC) asynchronously, the whole class becomes a peer review. It becomes a workshop, sometimes a cauldron, in which students' ideas come under close examination. Students can continually push each other through a textual dialogue that they, even they, the children of the electronic age they are, are unaccustomed to.

In a way that some may find paradoxical to critics of digitality, an OWC works so well in this way because it is a uniquely slowed-down course experience. Students get to think. They interact with each other in a naturally contemplative environment, one that by its nature does not reward the quickest hand or the sharpest tongue.

For many teachers in any medium, that’s what good peer review is all about. It's not about shuffling papers around a room and asking students to give each other brief comments (leading, too often, to "All you need to is __ and you'll get an A!"). It's about providing students with time, even a few structured minutes, to delve into each others' thoughts made manifest through writing.

I haven't taken video of students working on written discussions in my courses (although, at CCCC 2014 I saw a fascinating presentation by Patricia Portanova of the University of New Hampshire; she used screencasts to investigate how writers are affected by distraction). But it would be interesting to compare students' cognitive and perhaps physical behaviors when responding to a colleague's post to the behaviors when they provide face-to-face responses in a classroom. How much thinking is going on in each situation?

Online, during the routine written conversations they have, students critique each other in detailed, complex ways. They push each other to think and elucidate ideas. They often are complimentary in authentic ways to writing that moves and impresses them. Without the confines of the class space, they can slow down and really think about the piece of writing.

As I've mentioned before, this dynamic can become transparent in an asynchronous OWC. If you are building a dialogue, then all postings become a kind of peer review: Here’s what I have to say about the writing you put before me. Based on my response, you can tell if I thought it was clear, if I thought it was powerful, where I thought it might need more elaboration or even evidence.

My students push each other hundreds of times in this way in the course. I believe it tightens their writing in ways I could never do alone.

OWI: Opportunities in medium, genre, volume

I have another blogging life, and this month those lives weirdly intersect, as I'm building here on my post for my blog Virtual Children (which I've been keeping on the Website When Falls the Coliseum, for several years). I posted there about another blog article written by a friend of mine on the Website 11trees; his smart post describes how much he writes in a work day, what he called just an average writing day for a "knowledge worker." In a one-day diary, he calculates he wrote about 2,500 words, saying, "We write more words every day than many college or high school students write in an entire term." 

His post is so smart for many reasons, particularly because it spurs great conversations about genre, medium, and writing volume. In his log, the writer describes all kinds of writing he did during his "writing day." At 5 am he's emailing; he works on PowerPoints and text messages; he drafts notes and other materials in/through word processing programs and different collaboration software. He's composing collaboratively with people around the world, planning, drafting, and putting on finishing touches.

If you teach writing, you have probably thought a lot about how and how much people write. I think the generation we’re educating now interacts with each other through writing more than any generation ever

But our teaching to prepare students for their writing lives often doesn’t account for the rhetorical complexity, the varied genres, and the sheer volume of the writing they will do. In fact, in many cases, writing instruction looks pretty similar to the way it looked many decades ago.

I remember in my earliest moments of interest in OWI that I felt that digital instructional settings opened new avenues for teaching and learning writing. In OWCs, students compose in multi-audience environments that are close to writing they will do in their professional lives -- and these environments are simply inherent parts of the course. Also, it was evident that electronic writing spaces allow teachers to assign lots of informal and exploratory writing, providing many opportunities for students to engage in meta-reflection about their composing processes. 

To me, ed tech tools are not exclusive to online instruction -- I mean, you can use digital tools with any type of course -- but the digital environment of OWI does seem to lead teachers to explore tech tools, so they will find new ways to interact with and even assess student writing that make the whole process more productive. For instance, 11trees is a great example of a tool that makes it easier for teachers to respond to student writing.

As I’ve described before, a brief analysis of my own 10-week, quarter-system classes showed  students were writing some 15,000 words through message boards and electronic journals. I was proud when I found that out and still am; I feel they had a challenging writing experience. But if you think about that number in comparison with the 2,500 daily words of a "knowledge worker,” you realize I might need to significantly up the workload yet if I'm going to responsibly teach writing that prepares students for what they’ll face at the next stage.

By the way, I'm not advocating a kind of mindless kowtowing to the “real world.” I don't want us to dispense of genres/modes like the exploratory essay or writing to learn. But the limitations of "school writing" get exposed in productive ways in online writing courses. The only game in town need not be "papers" that don't resemble the short, high-stakes writing that they do. We can put them in posting and sharing situations that allow us to better teach them what they will do -- and are doing -- in other parts of their lives.

This is a major component of the promise I see of OWI. Perhaps we need to do more in our research to understand not just the kinds but the amount of writing that take place in various professions.

Online peer review writing groups

In talking with teachers about online writing courses (OWCs), I find myself circling back to some core teaching ideas, and one is that OWI appeals so strongly to me because the OWC becomes, especially when asynchronous forums are at its center, an ongoing peer review. And I sure do believe in peer review.

In OWCs, students are consistently reading, thinking about, and responding to each other’s writing. They are working with each other as writers. It’s serious work, non-surface work. In fact, an oft-encountered problem with “regular” peer review is that students who are not confident or are unsure how to respond turn into grammar guards/editors. You rarely see message board conversations that devolve into that, and when you do, it often looks like flaming.

Some of what makes peer review work so well online is simple: It’s easy to bring students together and share documents digitally. But using easy-to-facilitate groups can become an even more powerful part of the OWC experience. I often conduct my group online peer reviews similarly to how I would other peer reviews in terms of review instructions, but there are two major logistical differences: 1) reviewers are responsible for all drafts in their group, and 2) comments are not just additive; they must be cumulative. In other words, say Ariana posts her document and Alexa reviews it. Drew cannot just come along and review Ariana’s draft without also accounting for Alexa’s comments. The reviewer's writing challenge becomes more complex, and the reviewee must read and account for multiple, perhaps contradictory, comments. They can synthesize such comments and consult me – or the writing center or someone else.

Technology-wise, groups are easy. (I use Blackboard Learn in my courses and find, despite the vitriol often directed at Blackboard, that it mostly does everything I need it to do. That being said, I do think its Group function is not very good, but my understanding, based on a Bb Learn “listening session” I attended recently, is that Bb will soon improve this functionality.) In terms of group creation and maintenance, you can simply let the CMS randomly create the groups for you, or you can purposefully build groups based on various criteria, distributing students in ways you think make sense. Four is a good number of students in a group.

Exchanging documents is easy on message boards. They can simply use attachments. Or they could use Google Docs or something similar. They can respond back to drafts in a post within the group and/or make comments using Track Changes or other annotation software. 

You can also choose whether students remain in the same group all term or whether you reshuffle when a new assignment arises. There are good arguments for both.

The online peer review group is basically a writer's circle, and perhaps students who have a good experience in class will find ways to set up similar circles for their own writing projects.

Addressing MOOCs with the CCCC OWI Position Statement

Last time, I posted about MOOCs (Massive Open Online Courses), suggesting some good ideas/practices that might emanate from massive courses. Now, I'm looking briefly at how writing MOOCs can be viewed in the context of A Position Statement of Principles and Example Effective Practices for Online Writing Instruction (OWI), published last year by our CCCC Committee for Effective Practices in Online Writing Instruction (again, I'm co-chair of this hard-working group).

Many people in higher ed, of course, are up in arms over MOOCs. Many good reasons exist not to like these courses, but I think fear and helplessness driving many criticisms: People rightly feel helpless in the face of this tide of depersonalized instruction. Teachers and whole institutions are scrambling. However, as I’ve said, some of education’s resistance to MOOCs is hypocritical because we’re complaining about depersonalized learning experiences even though for decades we've herded students into 500-seat lecture halls and allowed our writing courses to be taught by people without offices who make about $150/week per course (if that).

The principles and practices in the OWI Position Statement can help writing teachers and WPAs by providing evidence and conversational scaffolding. The Position Statement, by design, can help potential writing MOOC teachers think through these courses; it might help some argue against MOOCs. Briefly, here are a few specific principles and how they might operate in this context:

OWI Principle 2: "An online writing course should focus on writing and not on technology orientation or teaching students how to use learning and other technologies." This means that for MOOC students, who are, by the definition of "open," not necessarily part of an institution, the learning platform has to be really easy to use. Students can’t spend their term figuring out how to manage the technology – there will be little teacher or IT support for them.

OWI Principle 5: "Online writing teachers should retain reasonable control over their own content and/or techniques for conveying, teaching, and assessing their students’ writing in their OWCs." Principle 5 addresses MOOCs in several ways, but especially with this: With 80,000 (or even 800) students, teachers have no control over assessing their students’ writing.

OWI Principle 6: "Alternative, self-paced, or experimental OWI models should be subject to the same principles of pedagogical soundness, teacher/designer preparation, and oversight detailed in this document." A MOOC can't result in the cutting of pedagogical corners (for those feeling institutional pressure...).

OWI Principle 13: "OWI students should be provided support components through online/digital media as a primary resource; they should have access to onsite support components as a secondary set of resources." A MOOC problem is that if you like MOOCs and work from that paradigm, you end up with support services that might replicate the same lack of institutional contact. If a 40,000 to 1 ratio is good enough for instruction, why not counseling, advising, and registrar ratios? Since the C stands for "course," institutions will have to make sure students understand what they get with "Open."

OWI Principle 9: "OWCs should be capped responsibly at 20 students per course with 15 being a preferable number." Those seeking a line-in-the-sand principle could find it here. Remember, C in MOOC is for "course." According to Principle 9, with 40,000 students, it can't be a "course." But maybe it can be a MOOE, with E for “experience.”

We designed these principles to help educators with OWI, including new approaches to teaching writing online. MOOCs are only one example. Massive courses aren’t good or bad, but teachers have to believe in a certain type of teaching to make them work. The Position Statement can help support or at least think through the kind of teaching that Massive Open Online Course represents.

Writing MOOCs -- what are they good for?

If you follow higher education even a little, you know about MOOCs: Massive Open Online Courses. These courses – a look at early MOOCs is here – have been promoted for all sorts of reasons. I’m not against MOOCs, but, like others who are wondering if this trend will run its course, I don’t want to get too far ahead of ourselves.

First off, these courses represent, by and large, technologically-facilitated ways to deliver lots of content to lots of people. And why not? Schools have been looking for ways to streamline content delivery for a long time, including the well-entrenched giant lecture hall experience. Shifts to distribution-of-knowledge have always been viewed with skepticism. I mean, the book, writing itself, was once seen as a threat to traditional oral knowledge distribution systems, yet we now live in a time when sitting alone reading a book is a mark of erudition. Things change.

But a MOOC to learn statistics or history or poetry or game theory is one thing. Bring on that slick, video content! Learning the skill of writing in such an environment is different. Several institutions are offering writing MOOCs for thousands of students. Some are being run by quality, experienced writing faculty. I had personal experience with one of these courses, and while I remain underwhelmed by these efforts, I believe MOOC-like thinking might yet drive some ideas for writing instruction:

1) Improving the online writing group. If I'm working on a high-stakes writing project and you are working on a high-stakes writing project, we could get together, even by email, and work together quite productively to help one another, yes? MOOCs might help us develop new tech platforms and logistics for facilitating such interactions. Maybe, even better, we will break through with new approaches, pedagogies, and philosophies about peer review.


2) Parallel/"piggyback" writing experiences. I'm putting "experiences" in Italics for a reason -- I'm not thinking of courses. MOOCs are helping us think deeply about how students are educated. Many disciplines lament their students' writing, yet how to fix it within a curriculum? Adding courses is not an option in many cases. Students at Drexel, for instance, are already faced with a tight curriculum. But what if you created an online writing experience -- again, not a course -- that students could encounter in parallel with their other courses? Students could "unlock" progressive components while working on documents and writing skills integral to their success in their particular program. This wouldn't be Massive. It wouldn't be Open. But it could provide students with a structured way to help students write. I have been working with Drexel's School of Ed on exploring something like this for graduate education programs.

3) Rethinking gateways and competencies. Drexel's College of Nursing and Health Professions has been using what some have been calling, somewhat unfortunately, "mini-MOOCs." (Forget the mini-massive contradiction; the idea is good.) Students can enroll in a small cohort for a kind of free gateway course. I'm oversimplifying, but if students succeed in the course and then enroll in the program, they can then be awarded credit for the experience. Lots of students want equivalency credits for writing, and I think this is a potential way to address this problem, which I found nettlesome during my time as Drexel's Director of First-Year Writing. If students can enroll in an open course, they can demonstrate proficiency in ways that could later be counted/certified.

4) Providing highly focused writing instructional experiences. I don't think much of a composition course let alone an entire writing curriculum as a MOOC, but what about running an ongoing four-week course in creating job documents (e.g., cover letter, resume, thank you letter -- very focused)? This course could use the power of peer review productively (and perhaps its development could even be backed/supported by an online resume company). You could design similar courses for highly focused writing contexts.

5) Assessment of student learning. We still have a lot to learn about how teaching is connected with student improvement in writing. Some folks in composition are looking at this very question, based on the early writing MOOCs. The MOOC delivery model may open new avenues of inquiry for us.

Many ideas and technologies are fueling "massification." We, writing teachers, should be working to develop ways of using these concepts and tools, because, if we don't, without a doubt, ideas not of our own will be imposed upon us.

Why OWI?

I’ve had many opportunities over the past year to talk in depth with colleagues from around the country about teaching writing online, and it's made me realize that there’s a good question for the online writing teacher to return to: Why are you teaching writing in these digital environments?

What is driving your teaching of writing online? Are you doing it because you want to? Because you have to? Is this a step in your professional development? Curiosity? Finances?

While self-reflection itself is a good thing, this is not just an effort to know thyself. The answers to these questions can inform what we do in the classroom – and they may help us to uncover in ourselves what I think of as an under appreciated aspect of teaching: teacher satisfaction and, well, happiness. (And our institutions should help us; see OWI Principle 12: "Institutions should foster teacher satisfaction in online writing courses as rigorously as they do for student and programmatic success.")

These questions about our own motivation can also lead to similar queries about programmatic OWI initiatives. In the midst of the mad rush to, ah, I have to say it, explore MOOCs, schools themselves might engage in some productive conversations about origins and goals. Why have schools and their programs explored OWI? What is to be gained for their students? What is to be gained for their faculty? How might OWI shift programmatic and institutional identity?

I believe that understanding who you are as a teacher comes first in your migration to teaching writing online, and part of that understanding is your motivation. Do you love writing? Do you love working closely with students? Are you are seeking the fame and fortune associated with first-year writing instruction? Schools can grapple with the same type of issues for their programs.

As we work through the details first of learning to teach online and then refining our pedagogy, it is worth pausing and asking that core question: Why am I doing this? The answer, carefully considered, perhaps on a slow summer day, will I believe help us do what we do more effectively for ourselves and our students.