Process Narration Videos and authentic student writing

Just like everybody else in writing classrooms nowadays, I'm dealing with AI. I like to think I'm mostly working with it, although, my sanguine teaching nature aside, I'm for sure at times working against it.

In several classes, I've overtly invited students to write about and sometimes even for the machine, as in my recent Language Puzzles and Word Games: Issues in Modern Grammar course, which includes this writing project choice (among many): "Using a prompt similar to an assignment from one of your courses, compare the text created by a generative AI chatbot/natural language generator with your own writing specifically in terms of grammar and usage."

In the Fall, I got to talking to an exceptional student, Veronica Medlock, about the drafting process in her writing. After that conversation, she quickly created and sent me a short YouTube video of the drafting changes while she composed, accompanying the video with a voice narration of the process.

It was fascinating.

We're trying to take this show on the road and have sent in a conference proposal about how such "Process Narration Videos," as we're calling them, can help promote and support authenticity in student writing assignments. Students can compose and submit these easy-to-create videos, which are built on the draft stage "mapping/tracking" that is a fundamental component of writing composition tools like Word and Google Docs, with accompanying voice narration.

Why is this so great? Of course, the English and writing teaching and learning literature has been overflowing with discussions about the problems of teaching writing in the--dare I call it?--age of AI. Teaching publications are dedicating focused space to AI (1) and instructors nearly every day receive invitations to professional development opportunities or information about new ed tech tools. Language and literacy teachers at all levels (and, of course, in many, if not all, other academic disciplines, e.g., math and programming) are dealing with the challenges of AI in teaching with a range of attitudes, approaches, tools, and philosophies. 

Some of my colleagues feel the authenticity problem is so dire that they are abandoning process-driven writing approaches and are instead returning to blue books or other non-digital, offline writing approaches. Those who have not given up have taken various often time- and energy-consuming stances and approaches: Requiring increasing amounts of informal writing, carefully scrutinizing the drafting process, leaning heavily on AI checkers in a kind of technological arms race, etc.

But as part of an overall writing pedagogy, Process Narration Videos provide a foundationally sound, pedagogically driven solution to authenticity issues in writing instruction. They reinforce good writing pedagogy, as they are grounded in writing process and ask students to reflect on their writing. They also help students see different stages of their writing, incorporate metathinking, consider rhetorical choices, and embrace informal writing, which of course is often embedded in strong writing processes.  

If our proposal gets accepted, Veronica and I will discuss the compositional background of this approach while focusing on the practical, hands-on way that it can be applied in classrooms across educational levels and curricula.

An important point is that students are often as frustrated by the prospects of other students using AI as their teachers are. So we're aiming to hit a nice balance: While Process Narration Videos can serve a dutiful--and responsible--academic process in discouraging AI-connected plagiarism, they are also simply good teaching and learning practice.

Note:

1) The topic of the most recent issue of CCC [77.1] was, as the Editors’ Introduction stated, “A Dappled, Undisciplined Response to Generative AI."

As AI becomes increasingly [S]ubtle

Lately, it's not often I refer back to my 2002 dissertation, "Subtle Technology and Student Writers: The Ambient Influence of Technological Myths on Communicators," but in the diss I had a core idea that's relevance might be re-emerging, a concept I described as Subtle T(echnology); I'll define it here very briefly as "a theoretical transformation digital technology might undergo that leads to complete dependence on these machines." The transformation occurs "when five interdependent traits of digital technology are fully realized: ubiquity, transparency, dependency, interconnectedness, and insubstantiality" (1).

With the explosion of AI use, I think Subtle T provides a valuable lens for viewing this technology, which I could argue has been integrating itself into our activities in ways that meet the traits above.

Of course, I was always a Writing Studies/Comp Rhet researcher and practitioner, so in the diss I described how Subtle T functioned in writing instructional scenarios.

Recently, I had a situation in a course in which I thought a student might have plagiarized language in parts of a project. I didn't think this student had malevolent intent, but some of the writing seemed inauthentic when compared with their other writing, particularly informal writing, in the course.

My course AI policy encouraged productive use of LLMs, so I was more interested in a constructive dialogue than a wrist-slapping--or worse.

When we met, I pointed out passages I felt were inauthentic, and I said while I didn't think it was material deliberately clipped from other sites or sources, it appeared to be AI-generated text. 

We had a good conversation, but the student was nonplussed as I talked about using AI and when eventually asked straight-up if they used AI tools. They told me, equally straight-up, no. The student said while they had indeed used material they found in Google searches to help develop the project, they were "just using Google," not AI.

Well, of course, if you are "just using Google" right now, the first "hit" is an "AI Overview," marked with a "sparkle icon." The student was surprised by this and told me, and after our discussion I believed it, that they didn't think of the results of Google searches as AI, despite the "sparkle icon."

This to me was another good example of how we'll have to keep up in our courses with AI. Many teachers are growing accustomed to handling the results of prompt-based AI use: We feel confident that students who use prompts and submit the subsequent text from LLMs are overtly plagiarizing. But embedded tools?--we'll have to sharpen definitions. Grammarly is one thing. AI built into Google searches is another. 

I almost feel that we might have to broaden our perspective and be careful not only of students using AI but of AI using them: Their common behaviors may be co-opted in ways that are ubiquitous, transparent, dependent, interconnected, and insubstantial. They will notice less and less.

Maybe it's time for me to renew my efforts to get that diss published. 

Note:

1) From page 1 of the diss, which I completed at Temple University. 

Red flags and white text: A plagiarism checker problem

For catching AI-based plagiarism, Turnitin and other checkers are limited but have their uses. A main issue with these checkers is flawed false positives (e.g., see this review by my colleague Bill Vargo on his blog), but I'm also always concerned with their tendency to default position the student as culprit.

I encountered an interesting situation last term with Turnitin.

In an online first-year course, my students used Jason Ohler's Taming the Beast (a really great text that's more relevant now than ever, BTW) to critically evaluate an everyday technology. They wrote white papers and then op-eds about their technology, and the class culminated (thanks to my colleague Dan Driscoll for the inspiration for this assignment progression) with a third project in which students created a social media campaign about their technology:

You will translate/convert work from your white paper and op-ed into a brief social media campaign (typically, a series or collection of posts). Think about your intended audience, your goal purpose with this campaign, and how to use multimodal strategies to communicate information quickly and efficiently; you may also link your messages to other texts and conversations. Your campaign—again, a collection of posts or the like—should express critical views of technology that you have developed throughout the term; use concepts such as revenge effects, as expressed by Ohler and Tenner, and problem posing, especially this “transformer”: “While most people believe __, a closer look reveals __.” I do hope you have some fun with this—and keep in mind that you’ve been practicing short form writing all term on the class Discussions.

When I began evaluating their final submissions, I was momentarily deflated when the first one had a red flag, stating, in accusatory language: "Hidden text: Attempts to throw off similarity detection." I sought further explanation and clicked to read the following:

Further explanation: Integrity insights to review as a priority. 6727 suspect characters.

What is hidden text?

An attempt to hinder similarity detection by exploiting exclusion mechanisms or artificially inflating the word count. Text is blended into the white background of a document to make it invisible.

But this was one of the top students, and I had met with her several times, so I doubted she cheated on her final project. As soon as I looked at it, I saw the "problem." This assignment was in an unconventional genre, social media posts, so in replicating posts, the student frequently used white text on dark backgrounds. There was no hiding going on at all.

So much for innocent before proven guilty.

I do appreciate Turnitin. As I've said many times, I mainly use it because of its three-minute easy-to-use audio response feature (I've long advocated for such response; see Note below), but it also does help with out-and-out cheaters. However, it has significant limitations, and as this example showed, it often encourages us to assume the worst--and sometimes that may happen when our students are providing their creative best.

Note:

1) I've written several pieces including "Cutting Keystrokes, Improving Communications: Response Technologies for Writing Instruction." California English 15, no. 1 (2009) and “Responding to Student Writing with Audio-Visual Feedback.” Writing and the iGeneration: Composition in the Computer-Mediated Classroom. Eds. Terry Carter and Maria A. Clayton. Southlake, TX: Fountainhead Press, 2008. 201-27.

Messy final drafts and metacomments

Like many teachers, I've been thinking about how writing studies faculty can help ourselves and our colleagues by drawing on our expertise to keep up with some of the challenges AI poses in writing instructional environments.

One teaching strategy I have been using is discussion-board dialogue about final student projects , as I described in "AI apps and student writing worries?: An old, reliable OWT practice can help"and "Generative AI and integrity concerns: More discussion board support."

That was a mixed bag. I used this teaching strategy in two very different courses. In one class it worked great. In another class, not so much.

In the latter class, I think the lukewarm success was partly because the course assignments didn't lend themselves to conversation. This course, The Literature of Business, has assignments that, while students can make them their own, does create somewhat similar results.

So I lived and learned. In recognition of those problems, I added another layer. I started also using metacomments: simply put, I ask students to write about their own writing as part of the assignment. This is an old composition practice, and in fact Julianne Ross-Kleinmann and Wess Trabelsi recently described a version of it in a smart T.H.E Journal piece "Teaching and Learning, Cheating, and Assessment in the Age of AI":

Another way to change teachers' method of assessing papers is to ask students to defend their work. This means having a conversation with them about their paper, either in person or when they upload it to the learning management system. Why did they make this argument or include that fact? What is the crux of their position? If they can't answer these kinds of questions, maybe they shouldn't receive a good grade on the paper — even if it is an excellent piece of work on its own. (emphasis mine)

Metacomments push students to talk about their practice and to defend their choices. In asking for these comments at the paragraph level, I am seeking granular analysis about their rhetorical choices.

My primary goal is constructive pedagogy, but if we're worried about authenticity and plagiarism, these comments head off a lot of clear, wholesale "borrowing": even if generative AI (or their roommate!) wrote a paper for them, they would still in many cases have to think through what each paragraph means in the context of the assignment.

If they tried to get AI, they still have to cut-and-paste individual paragraphs and generate the purpose--note, not the summary--of that paragraph in the context of the entire assignment. They're going to have to do some work in most cases.

My students have produced strong metacomment work. At times the comments are good. Sometimes they are fantastic. And occasionally they are better than the writing project itself.

However, I had to shake myself out of normalcy and embrace a new mindset when reading student writing. While in writing studies we often talk about embracing process, we traditionally still want something clean handed to us. Indeed, I think most teacher-readers are enamored of the "clean draft": neat, tidy, finished. I get why, but if we use metacomments the way I'm suggesting, we're asking for a "messy final draft," one in which students accompany each paragraph with comments about what they want to achieve; such writing will be intrusive. 

I don't require students to be fancy in formatting their metacomments. Footnotes or endnotes are ideal, but I'm fine with inserted bracketed or boldfaced comments after each paragraph: Messy, but functional for this purpose!

Ultimately, I'm moving away from the cleanly "typed" paper toward a product that vividly shows the often messy excellence of student thinking.

Integrity and AI

At Drexel I'm a member of a recently assembled committee called the Provost's Office Working Group on Policy for Academic Integrity with Artificial Intelligence. Like many (most? all?) colleges and universities, we're trying to create academic integrity guidelines (or guardrails) for our faculty and students.

One aspect of our working group's efforts that I like is that in the background of our statement, the initial item describes the value of AI in education. We're not treating these tools first and foremost like a problem: Instead, we are treating them like tools vast potential, but potential we must understand within a sphere of sensible, ethical guidelines.

This is crucial, because in line with our local work, statements and guiding principles are being posted everywhere. Here are a few samples.

Even early on, people were developing language, such as the course policy wording by Ethan and Lilach Mollick in “Why All Our Classes Suddenly Became AI Classes” (1). They provide sample course policies:

I expect you to use AI (e.g., ChatGPT and image generation tools) in this class. In fact, some assignments will require it. Learning to use AI is an emerging skill and I provide tutorials on how to use them. I am happy to meet and help you with these tools during office hours or after class.

Be aware of the limits of ChatGPT, such as the following:

• If you provide minimum-effort prompts, you will get low-quality results. You will need to refine your prompts in order to get good outcomes. This will take work.
• Don’t trust anything it says. If it gives you a number or fact, assume it is wrong unless you either know the answer or can check with another source. You will be responsible for any errors or omissions provided by the tool. It works best for topics you understand.
• AI is a tool, but one that you need to acknowledge using. Please include a paragraph at the end of any assignment that uses AI explaining what you used the AI for and what prompts you used to get the results. Failure to do so is in violation of academic honesty policies. Be thoughtful about when this tool is useful. Don’t use it if it isn’t appropriate for the case or circumstance.”

The February 2023 Institute for Learning and Teaching at Colorado State describes a "multiple possibilities" approach to syllabus statements with AI, including examples from syllabi:

• The Prohibitive Statement states that any AI "use on graded work/ work for credit will be considered a violation of the academic misconduct policy." The Prohibitive Statement is appropriate "for a class in which the course outcomes have been compromised by use of the technology," such as a writing course.
• The Use-With-Permission Statement "makes it clear that only certain uses will be acceptable" to the instructor.
• The Abdication Statement includes an example that states, "From his point forward, I will assume that all written work has been co-authored or entirely written by ChatGPT." In this case, instructors will grade such writing "normally" and the "grade will be a reflection of your ability to harness these new technologies as you prepare for your future in a workforce that will increasingly require your proficiency with AI-assisted work."

Notre Dame's May 2023 statement is short, strongly worded, and straightforward: "When students use generative AI (such as ChatGPT) to replace the rigorous demands of and personal engagement with their coursework, it runs counter to the educational mission of the University and undermines the heart of education itself."

Director of the University of Missouri's Office of Academic Integrity Ben Trachtenberg took an FAQ-type approach to a March 2023 statement, answering questions such as

• May Students Use ChatGPT and Similar Tools for their Academic Work?
• Can Professors Tell if Students Use ChatGPT?
• What’s the Bottom Line?

Answering the last question, he indicates students should seek to uphold Mizzou's values as described in the Mizzou Honor Pledge: "If you think your instructor would object to your using ChatGPT (or a similar tool) in a certain way, you should not do it. If you are unsure, you should ask your instructor first."

Interestingly, the resource site for Royal Military College of Canada (RMC)/Canadian Forces College (CFC) focuses on syllabus policies about the low quality of detection tools: "Until formal direction is received from the Department of National Defence, it is suggested that, for privacy, ethical, and security reasons, no student, faculty, or staff should be required or expected to create or pay for personal accounts in commercial generative systems such as ChatGPT."

In Listserv conversations I have followed within Drexel and beyond, many programs and individual faculty have tried to define the way they will judge AI use in their own classrooms. In addition, members of the academic community are increasingly concerned that authors will submit AI-composed manuscripts. 

As usual, it's a good idea to ask colleagues what they are doing: Our teaching communities often contain deep wisdom, and we should continue to take collaborative, local approaches to creating sensible policies about the use of AI/ML. The material above represents just a few of these efforts.

Notes:

1) From February 9, 2023 Harvard Business Publishing Education.

Trying to keep up with natural language processing tools

Turnitin responded quickly. Faced with natural language processing tools, it deployed Turnitin's AI writing detection model. One day recently I opened Turnitin through my CMS, and there was the AI detector, ready to go.

This is likely a necessary response by the long-standing plagiarism guardian. "AI" has zoomed past cut-and-paste, and tools like Turnitin are trying to keep up.

As I've said on several occasions, I use Turnitin not so much for its plagiarism-catching talents but because it is integrated into Blackboard and has an easy-to-use commenting interface that includes the ability for a teacher to create a three-minute voice response. (I wish more teachers would use voice commenting features...)

We know its AI checker is not going to be perfect, but I had to relay this brief anecdote about just how far away it appears we are.

A sharp student in my class Language Puzzles and Word Games: Issues in Modern Grammar just submitted her second short paper.

Ah, relief: The AI checker came up 0%. (I expected nothing less, of course!)

A problem, though: Her project, in line with what we're doing in the course, was a comparison of a paper she wrote in another class and how ChatGPT would have responded to a similar prompt. In her paper, she used giant chunks of ChatGPT-generated text to illustrate grammar, usage, and style differences in her work from the natural language generator.

0%. Zero. Zilch. Nada. The AI detector didn't pick up any of it.

(The text she used was not bound by quotation marks, FYI.)

No one is expecting perfection, especially at this stage (and Turnitin includes serious qualifiers about the effectiveness of its tool), but we should remain very aware that digital watchdogs likely won't catch digital text. We have to do it ourselves.

We're in the early stages of a major shift in teaching, and we're collectively engaged with on-the-fly decisions about how we're adjusting to that change. I trust my students, but teachers will invite authorial misrepresentation if we don't develop pedagogies on the front end instead of relying on cross-our-fingers detection after the papers are in.

VEXT: An application for idea generation and academic integrity

VEXT is an application designed for use by administrators, teachers, and students (and others) to change approaches to academic integrity as well as to help see source and idea relationships in documents. According to its website, "VEXT offers an ethical approach to idea generation, data visualization, and plagiarism checking."

On the student side, VEXT provides a visually attractive way of approaching source usage and idea relationships. From the faculty/admin site, it offers ways to manage information from a cohort and deal with academic integrity.

At its "heart" is what VEXT calls a "paper fingerprint" (see the site), a rectangular, strip-like image consisting of thin, vertical bars that helps students visualize data so they can recognize "themes, patterns, and sentiment" in their writing: This "code bar" provides, essentially, a map of their ideas and how they are using sources in their writing.

VEXT, in some ways, was born of frustration with contemporary plagiarism software paradigms, which can place faculty and students in adversarial roles. VEXT's site urges users to "Evolve from Plagiarism Detection to Data Driven Insights," and one of the blog posts on the VEXT site declares that "The old paradigm of plagiarism detection is dead."

The app promises to offer a different approach. Using machine learning algorithms, it instead helps users visualize "the DNA' of your papers and pedagogy."

For faculty, the app has a dashboard (1) to help spot data trends in assignments while also allowing for "crowdsourced plagiarism detection"--seeing those trends in certain contexts. For administration, there is a broader dashboard that helps administrators make decisions and see analytics to determine the effectiveness of curriculum across a swath of instructors; this includes an open, searchable database so administrators can access knowledge being created across an institution.

Note:
1) I have been interacting with the company since the summer, and they are fast at work improving the back-end dashboard tools.