Red flags and white text: A plagiarism checker problem

For catching AI-based plagiarism, Turnitin and other checkers are limited but have their uses. A main issue with these checkers is flawed false positives (e.g., see this review by my colleague Bill Vargo on his blog), but I'm also always concerned with their tendency to default position the student as culprit.

I encountered an interesting situation last term with Turnitin.

In an online first-year course, my students used Jason Ohler's Taming the Beast (a really great text that's more relevant now than ever, BTW) to critically evaluate an everyday technology. They wrote white papers and then op-eds about their technology, and the class culminated (thanks to my colleague Dan Driscoll for the inspiration for this assignment progression) with a third project in which students created a social media campaign about their technology:

You will translate/convert work from your white paper and op-ed into a brief social media campaign (typically, a series or collection of posts). Think about your intended audience, your goal purpose with this campaign, and how to use multimodal strategies to communicate information quickly and efficiently; you may also link your messages to other texts and conversations. Your campaign—again, a collection of posts or the like—should express critical views of technology that you have developed throughout the term; use concepts such as revenge effects, as expressed by Ohler and Tenner, and problem posing, especially this “transformer”: “While most people believe __, a closer look reveals __.” I do hope you have some fun with this—and keep in mind that you’ve been practicing short form writing all term on the class Discussions.

When I began evaluating their final submissions, I was momentarily deflated when the first one had a red flag, stating, in accusatory language: "Hidden text: Attempts to throw off similarity detection." I sought further explanation and clicked to read the following:

Further explanation: Integrity insights to review as a priority. 6727 suspect characters.

What is hidden text?

An attempt to hinder similarity detection by exploiting exclusion mechanisms or artificially inflating the word count. Text is blended into the white background of a document to make it invisible.

But this was one of the top students, and I had met with her several times, so I doubted she cheated on her final project. As soon as I looked at it, I saw the "problem." This assignment was in an unconventional genre, social media posts, so in replicating posts, the student frequently used white text on dark backgrounds. There was no hiding going on at all.

So much for innocent before proven guilty.

I do appreciate Turnitin. As I've said many times, I mainly use it because of its three-minute easy-to-use audio response feature (I've long advocated for such response; see Note below), but it also does help with out-and-out cheaters. However, it has significant limitations, and as this example showed, it often encourages us to assume the worst--and sometimes that may happen when our students are providing their creative best.

Note:

1) I've written several pieces including "Cutting Keystrokes, Improving Communications: Response Technologies for Writing Instruction." California English 15, no. 1 (2009) and “Responding to Student Writing with Audio-Visual Feedback.” Writing and the iGeneration: Composition in the Computer-Mediated Classroom. Eds. Terry Carter and Maria A. Clayton. Southlake, TX: Fountainhead Press, 2008. 201-27.

A mix of approaches in this term's asynch OWC

I'm again teaching an asynchronous first-year online writing course (OWC) this term, and in an environment of increasing AI apps in which teachers not only worry about academic integrity but have more fundamental concerns about what it means to teach writing, I have found myself working from established practices while still refining them to account for the fast-paced changes we're facing.

I'm engaged in a variety of practices to try to maintain a writing-heavy class environment, some of which I've written about individually before:

• I'm again using the "textual brew" of unusual texts and discussion prompts.
• I'm trying to strengthen the relationship between the discussion posts and the major course projects.
• Speaking of discussions, I'm using them in a tried-and-true way to see a great deal of informal student writing: It's mid week 4, and in a class of 20 we're already over 550 posts (including my contributions).
• Drawing on the fundamental nature of low-stakes, informal writing, I'm having students in mini  assignments--right on the discussions!--analyze and revise their own posts; e.g., revising out instances of the verb "to be" in a post they choose.
  
• I've included the stoplight approach to AI that I mentioned last time.
• I'm asking students to include a one-page metacommentary about their use of AI in conjunction with that stoplight approach.
  
• Working with several of my colleagues in a mini teaching circle, I've assigned a fantastic book that helps students take a critical look at technology, Jason Ohler's Taming the Beast.

How's it all going? Talk to me after the quarter is over. However, I'm enjoying it: The challenge is there. These practices have created a productive pressure on my pedagogical approaches.

"Stoplight" approach to AI use in writing assignments

Like you, my Inbox is bombarded with messages about AI. I try to keep up, to stay especially focused on the writing instruction-centric ones, but it seems that only an AI-based triage tool can help me with the deluge...

Teachers do keep productively discussing the challenges of student use of AI, especially in writing instruction. This term, I am team-teaching an Honors seminar Contrasts in AI: Now & Future, part of Drexel's overall Honors College theme this year: Black & White. The class has been a great experience, particularly working with my colleague, a computer and electrical engineering professor.

Our course is about AI, and in addition to many informal assignments such as reading notes and quizzes, we also have three short papers/projects. We talked about this a lot in the summer, and we eventually developed a "stoplight" approach that we thought not only fit well with our course subject matter but also helped us address the challenges of requiring short papers in this current AI educational environment. Lightly edited from the syllabus, here is what we asked of students:

You will write three short, 500- to 1,000-word papers/projects in the course, due Weeks 4, 8, and 11. We’ll discuss each in class when it is assigned, but in general you’ll follow these guidelines:

1. For the theme/thesis of each paper/project, you will develop a contrasting perspective/view of some aspect(s) of AI, building on the course materials and conversations.
2. Each paper/project will be different in its use of AI, based on a “stoplight approach”:
• Project one is RED: You cannot use any AI, including grammar checkers like Grammarly. It may be impossible to avoid some built-in tools in Word and Google Docs.
• Project two is YELLOW: You can use some components of AI to help you develop your ideas and deliver them. You might think of writing on a kind of scale of complexity or a continuum, starting with the word, working through the sentence, then paragraph, then overall concept/purpose. (We missed a few steps in between for sure.) For “Yellow,” please do use AI bots/apps in your project, but only do so up to the paragraph “level.” We ask that you still not use AI for invention, conceptual, or organizational matters, but you should feel free to “partner” with ChatGPT, Copilot, etc. to help you at the sentence- or word-level, perhaps in particular with grammar and mechanics.
• Project three is GREEN: You can freely use AI as a partner to help you write your paper. Note that you cannot just submit the raw text generated by an AI app/tool, but you can certainly go through an evolution of prompts in presenting your final paper.
3. For each paper, you will include at the end a one-page metacommentary about how the use of or the constraints on the use of AI affected your development of your ideas. There are alternative ways you could present this metacommentary that we’ll discuss in class. Note this is different from the AI statement described in the Course Policies below.

So far, we've received the red and yellow papers/projects. This class of bright students have written strong, thought-provoking projects, but, boy, has the hidden gold been in the metacommentaries. 

In these end reflections, students have commented in considerable depth about the constraints of the red and yellow categories, how they navigated what those categories have meant to their writing process, and in some cases how they hadn't realized how much they had become reliant on AI tools in their writing. (Note the metacommentaries are part of the grade, a deliberate move we made so as to reward their thoughtful work.)

I want to reiterate that our class is about AI: Issues surrounding AI are part of the course subject matter. That certainly helped lead us to developing a "medium is the message"-type approach to the assignments.

However, including these detailed metacommentary components has helped us with our writing pedagogy. While plagiarism and authenticity weren't primary targets, the metacommentaries have encouraged authenticity while also providing them with a platform to discuss their own writing and thinking processes, in line with decades of composition work about the value of such writing.

As instructors, we are eager to receive paper/project three next week, not only to see their final thoughts about AI contrasts but to read how the open green category differed for them in their use of AI in their writing.

GSOLE Principles and Tenets revision presentation at CCCC 25 in Baltimore

Since its founding in 2016, the Global Society of Online Literacy Educators (GSOLE) has continued to grow and evolve, expanding its membership, initiatives, and offerings.

A key example of this evolution has been a recent effort by the GSOLE leadership (you can see current leadership here) to revise GSOLE's Online Literacy Instruction (OLI) Principles and Tenets, which were approved and published by the GSOLE Executive Board on June 13, 2019. Somehow, over five years have passed since our group developed these initial guidelines for OLI.

In education generally and literacy/writing specifically, much has changed since June 2019, and many of those changes, especially pandemic-related online learning and now the impact of AI, need to be examined in terms of the way we approach OLI. 

I'm part of this revision effort, and a hardworking group mainly broke up the work into subcommittees to evaluate how the Principles and Tenets might change. My subcommittee focused on AI.

We just learned of the acceptance of our proposal for CCCC 25 in Baltimore to present both the results of this work as well as the process we took to get there. We'll be eager to share the way that what we believe were fundamentally sound--they really are!--initial Principles and Tenets have adapted in the face of major changes in the landscape of OLI.

Faced with disruption, my students were flexible, adaptable, nimble

[I adapted this from my post on the site When Falls the Coliseum.]

Drexel was one of the universities to experience a protest this past academic year, as toward the end of our fast-paced spring '24 term and overall academic year, protesters camped in the center of campus for nearly a week.

The university went on lockdown when the encampment started on a Saturday night, gradually resuming normal operations during a five-day period. For two days, many classes were online, including mine.

In my class, studenting-wise, the kids did alright.

I know this pandemic "generation" has been through a lot, but collective worries that they have fallen to pieces and will not be able to cope or function in the "real world" (whatever that is) were certainly challenged by my students' quick and in most ways expert transition to a Zoom-based classroom.

The university was functioning hour-by-hour, so the entire Drexel community would only receive information about the next day's operations in the evening. We all had to be ready to shift gears.

I don't know if they were happy about it, but every student showed up for our synchronous online class on Tuesday, and they were on time and ready to go. All but one had cameras on, and they worked well both in the whole group class and the breakout rooms, interacting with me and their classmates.

We shared a few winky in-jokes about online learning, but in terms of the outcomes for that day's lessons, we stayed on track--I did my best, but it was largely thanks to their attention and efforts.

Few would likely choose to resume learning in a solely digital modality, but if need be, they showed they could pull it off.

These students are not withering little snowflakes, fluttering about, buffeted helplessly by forces they cannot/will not resist, melting when the temperature of their environment clicks up a notch. I think, as I did during much of the pandemic, that they're warriors. I keep finding that they adjust and adapt when called upon, and they showed that again last month.

Drexel acted with relative speed in opening its campus and beginning onsite instruction again. We finished the quarter. Graduations for both colleges and the university as a whole came and went.

I watched students walk in front of me across the stage at the College of Arts and Sciences graduation, and, you know, I felt not only proud that I had a chance to work with many of them but confident that they'll be the ones in charge some day.

Messy final drafts and metacomments

Like many teachers, I've been thinking about how writing studies faculty can help ourselves and our colleagues by drawing on our expertise to keep up with some of the challenges AI poses in writing instructional environments.

One teaching strategy I have been using is discussion-board dialogue about final student projects , as I described in "AI apps and student writing worries?: An old, reliable OWT practice can help"and "Generative AI and integrity concerns: More discussion board support."

That was a mixed bag. I used this teaching strategy in two very different courses. In one class it worked great. In another class, not so much.

In the latter class, I think the lukewarm success was partly because the course assignments didn't lend themselves to conversation. This course, The Literature of Business, has assignments that, while students can make them their own, does create somewhat similar results.

So I lived and learned. In recognition of those problems, I added another layer. I started also using metacomments: simply put, I ask students to write about their own writing as part of the assignment. This is an old composition practice, and in fact Julianne Ross-Kleinmann and Wess Trabelsi recently described a version of it in a smart T.H.E Journal piece "Teaching and Learning, Cheating, and Assessment in the Age of AI":

Another way to change teachers' method of assessing papers is to ask students to defend their work. This means having a conversation with them about their paper, either in person or when they upload it to the learning management system. Why did they make this argument or include that fact? What is the crux of their position? If they can't answer these kinds of questions, maybe they shouldn't receive a good grade on the paper — even if it is an excellent piece of work on its own. (emphasis mine)

Metacomments push students to talk about their practice and to defend their choices. In asking for these comments at the paragraph level, I am seeking granular analysis about their rhetorical choices.

My primary goal is constructive pedagogy, but if we're worried about authenticity and plagiarism, these comments head off a lot of clear, wholesale "borrowing": even if generative AI (or their roommate!) wrote a paper for them, they would still in many cases have to think through what each paragraph means in the context of the assignment.

If they tried to get AI, they still have to cut-and-paste individual paragraphs and generate the purpose--note, not the summary--of that paragraph in the context of the entire assignment. They're going to have to do some work in most cases.

My students have produced strong metacomment work. At times the comments are good. Sometimes they are fantastic. And occasionally they are better than the writing project itself.

However, I had to shake myself out of normalcy and embrace a new mindset when reading student writing. While in writing studies we often talk about embracing process, we traditionally still want something clean handed to us. Indeed, I think most teacher-readers are enamored of the "clean draft": neat, tidy, finished. I get why, but if we use metacomments the way I'm suggesting, we're asking for a "messy final draft," one in which students accompany each paragraph with comments about what they want to achieve; such writing will be intrusive. 

I don't require students to be fancy in formatting their metacomments. Footnotes or endnotes are ideal, but I'm fine with inserted bracketed or boldfaced comments after each paragraph: Messy, but functional for this purpose!

Ultimately, I'm moving away from the cleanly "typed" paper toward a product that vividly shows the often messy excellence of student thinking.

AI and the Philadelphia Eagles' greatest center ever

Occasionally I get to cross my two little corners of the web. Here is a version of a piece I wrote on my blog on the site When Falls the Coliseum about the farewell speech of Philadelphia Eagle center Jason Kelce. It was a powerful, emotional speech, and a strong piece of writing and rhetoric.

The delivery was not just quintessential Kelce, but it was very much something of the writers of his time and generation: Dressed in a sleeveless workout shirt and sandals, he read the speech from his phone; he had perhaps even composed it on that phone--writers regularly compose on phones now.

For a book club at Drexel, I happen to be reading The Meaningful Writing Project by Michele Eodice, Anne Ellen Geller, and Neal Lerner, in which they describe their research project to discover how students report the qualities of meaningful writing experiences, how those experiences are connected to learning, and how what faculty offering such projects conclude about meaningful writing, finding this:

meaningful writing projects offer students opportunities for agency; for engagement with instructors, peers, and materials; and for learning that connects to previous experiences and passions and to future aspirations and identities. (4)

Watching Kelce, I thought and felt a lot of things from many perspectives. In my role as a teacher, I thought: "No way AI wrote that speech."

Why? Because I was struck by how authentic Kelce's speech was. It was him in content, style, and voice.

I thought how proud I am when I receive authentic writing from my students. And it made me think again of the big challenge--a challenge now renewed because of generative AI (GAI)--facing us writing teachers to create writing assignments that are not plagiarism proof so much as plagiarism discouraging because they bring out authentic, meaningful writing from our students.

I know it's not simple for us as teachers to create such writing assignments. After all, we're not going to be able to reproduce easily contexts like Kelce's, where someone emotionally bids goodbye to an activity of passion after nearly a decade and a half.

But I believe GAI productively pressures teachers to create more meaningful writing environments for our students. Assignments with authentic roots will have a greater likelihood of inspiring such responses.

There's considerable and rightful concern in composition about GAI. Will GAI engines grow in sophistication to the point that we won't need people for that very human task, writing? Maybe, but as I watched Kelce, I thought, "I don't think so"--or at least, "We've got a ways to go before that happens."

I won't start boasting that I can create plagiarism-proof assignments: Students will take up the challenge and prove me wrong. But listening to Jason Kelce's farewell from football, I realized that if I keep striving to develop assignments that mean something to them, students will return something meaningful for me and, more importantly, for themselves.