When utilized well, ChatGPT and other LLM (Large Language Model) AI engines can be an effective teacher’s assistant and help provide students with detailed, actionable feedback. While it can’t (or at least shouldn’t) replace the teacher’s role in creating feedback, it can help improve that feedback. ChatGPT can handle lower-level feedback, freeing up a teacher’s valuable time to focus on higher-impact tasks. 11trees has already written a deep dive into the strengths and weaknesses of generative AI for feedback along with some tips for how to use it effectively. In this post, we’re going to show how tools like ChatGPT can be effectively paired with Annotate PRO’s (AP) College Edition Library to create more effective feedback on academic writing.
In our example, we’ll provide feedback on a 6.5-page essay from a student who drew parallels between the main characters in two popular movies. The example essay and feedback can be seen here as a Google Doc. For those unused to this sort of work, imagine grading even a moderate-sized class of 20 students’ essays. At an average of 6 pages, you’d have 120 pages worth of reading to do, before ever providing feedback. This is where ChatGPT’s ability to quickly process large chunks of information can be an effective teaching assistant. It serves as a filter to help you know where and with whom to dedicate more time to giving feedback.
We gave ChatGPT access to the essay and this prompt (we did not provide the essay assignment prompt):
ChatGPT Prompt
“Imagine you are a high school English teacher. Provide up to five bullet points of areas that need improvement such as argument, proofreading, style, etc. Provide a specific example in the paper for each bullet point. Here is the paper:”
Result (Click to Expand)
1. Argument Structure and Clarity: The argument you are making about the transformation of Garrigan’s character throughout the movie needs to be made more clearly, especially in the introduction and conclusion. The analogy comparing Garrigan to John McClane from Die Hard is interesting, but it could be more clearly defined and directly tied to your thesis.
Example: “In The Last King of Scotland we meet a character by the name of Nicholas Garrigan… Throughout the movie, he still succumbs to the corrupting influences of power, fame, and sex (Hemanth).” This section is unclear about your argument. Are you arguing that Garrigan’s susceptibility to these influences shows a lack of character or personal development, or are you arguing that these experiences lead to character growth and increased strength of will?
2. Cohesion of Arguments: While the comparison of Garrigan and McClane can serve as a useful device, it is necessary to consistently tie these comparisons back to the main argument.
Example: “Opposite of this is John McClane whose non-arrogant, confident demeanor in this role would illuminate these flaws in Garrigan.” While this comparison does highlight differences between the two characters, it doesn’t clearly relate back to your argument about Garrigan’s transformation or how these differences actually inform his development.
3. Citation and Use of Evidence: Ensure all evidence used from the film or external sources is correctly cited and directly supports your argument.
Example: “However, although Garrigan has the ear and determination of a leader, throughout the movie he still succumbs to the corrupting influences of power, fame and sex (Hemanth).” It’s unclear what evidence Hemanth provides that supports this point. Making this clear would strengthen your argument.
4. Sentence Structure and Syntax: Aim to write in a more concise, straightforward manner. Avoid overly long sentences and ensure clarity of meaning in each sentence.
Example: “This is evident in the beginning of the film during the depiction of his school and family life.” This sentence could be restructured for clarity, e.g., “Garrigan’s sheltered upbringing is evident from the depictions of his school and family life early in the film.”
5. Consistent and Formal Tone: The essay’s tone is generally formal, but there are a few instances where it becomes colloquial or casual, which can detract from the seriousness of your argument.
Example: “A nice touch would be that sarcastic smirk McClane gives to his enemies in Die Hard showing that they do not stand a chance and McClane does not take them seriously.” Phrases like “a nice touch” might be seen as less formal and could potentially undermine your academic argument. Consider rewording this sentence to maintain a consistent formal tone.
ChatGPT’s feedback attempt provides the teacher with a starting point, a heat map of areas the student needs help with. Do you agree with all of them? Points four and five are subjective and would come down to teacher discretion if they’re helpful or necessary. The main takeaway from ChatGPT’s review is that the student struggles to make a well-supported argument. To provide the student with actionable learnings, however, the teacher needs to provide more specific feedback. Which is where AP and the College Edition Library can be helpful. AP and its Libraries are to substance what Grammarly is to mechanics.
College Edition
One of the main areas covered by the College Edition (CE) Library is Argument. The Argument group, inside the CE Library, has over forty comments that address fallacies, wording, and specific praise for where students might have done well. Take one point that ChatGPT gave: “It’s unclear what evidence Hemanth provides that supports this point. Making this clear would strengthen your argument.” That’s a great starting point, but the student would benefit much more with help on how to strengthen their argument. Take for example the “Unsupported” Comment from CE:
You have made an unsupported claim here. You either need to find evidence to back it up, or else take it out. See this link for more information.
It gives guidance and links to an academic resource that will help the student develop a key skill. Combine the CE comment with the point ChatGPT made and the student now knows what they need to work on and, importantly, how to improve.
The process we’re summarizing here isn’t streamlined. It requires multiple windows, clicks, and juggling of different solutions. It doesn’t scale for 20 papers, nevermind 100. We aren’t comfortable handing over our student’s IP to murky tech companies. Remember, if you aren’t paying for software you aren’t the customer. You’re the product.
At 11trees, we are exploring these sorts of combinations. We believe we can streamline the workflow for teachers, assuming full transparency about use of data, and create value for educators and students. For instance, AP+ users who use Canvas Speedgrader can access the (optional) Comment History feature to review previous feedback provided to specific students. This isn’t the student’s work, it’s the comments provided (across assignments and potentially multiple courses or terms) by the teacher to the student. If that feedback was shared with a Large Language Model with a prompt to synthesize the most important lessons, we could quickly create personalized development plans for our students.
We see lots of opportunity but also many questions. LLMs like ChatGPT aren’t very good at knowing their own limitations. They can gloss over gaps or even sometimes invent citations or evidence. Their responses can be generic and, well, robotic. Students need personalized, detailed feedback. Not generalities. But we’re excited to explore and welcome the dialog with our users over responsible, value-added applications of LLM/AI technology.
AP is free for individual educators
Create as many Libraries as you’d like, save zillions of clicks with our fantastic toolbar for Canvas, Bb, Moodle, Schoology, Teams, Brightspace, Google Docs, and Microsoft Word…
And if you use Google Docs, Canvas, or Microsoft Word or Blackboard? Our toolbars and optimization to work with these solutions may just make you weep with joy.
In Google Docs and Canvas SpeedGrader Annotate can (optionally) log ALL of the feedback you create, whether you search your libraries or not. In Canvas SpeedGrader the Annotate PRO top toolbar even presents you with the current student’s avatar and a history of feedback provided to them.