When eighth-grader Timothy asks ChatGPT to flat out write his persuasive essay about school dress codes, one might assume he reveals that he has learned to see writing as performance for teachers rather than communication with an audience.
Have Timothy try the same prompt in several LLMs, compile all of the AI responses, assemble them into a portfolio, and then ask him to offer an oral report answering the following:
- What were the common themes found in all of the AI-based responses? Which LLM took the most creative path to developing the argument? Cite some comparative examples.
- How well did each LLM cite its sources? Did one LLM perform better than the others? How were they the same? How were they different?
- Which LLM demonstrated the best fluency in written language based on using definitive statements and rhetorical strategies? Did any LLMs "beat around the bush" and avoid making conclusive remarks?
- Which LLM produced the most readable essay? What elements of structure and language choices made it more readable than the others?
- Which LLM produced the strongest argument? On what basis do you make this assessment?
- As a user of an AI system, how could you have produced the output more efficiently? How could you have guided the AI system to produce stronger results or a better output format?
- How might this exercise, as an AI-based process, be applied to other situations where humans need reliable information from diverse points of view?
This is an example of an AI Augmented Learner operating in the metaphor of an orchestra conductor, creating meaning from applying scholarly skills in combination with acumen in operating AI systems as an agent (not a client). Read more: https://stevecovello.substack.com/p/what-is-scholarship-in-the-age-of
Great idea!! Maybe a little too time consuming considering it's just one assignment, but I personally would definitely think about going down this path. Not in a bunch of machines but maybe a few. Might not make the cut but in medias res but worth a thought. I hope we get some more takers, Steve. I really like your impulse here.
As the teacher, Timothy's choice might make me consider a few things
First, how is Timothy viewing the assignment; what is his own purpose in submitting the essay? Was he just wanting to get through a required task that is not important to him with the least effort? Was he perhaps accepting the assignment to be a demonstration of how he would solve the communication problem of persuading some audience to his own perspective on school dress rules, or was his purpose aligned at all with the teaching/assessment purposes of the assignment?
I might also stop to think about the relationship of the writing act to the writer's purpose. Does the purpose justify the means? Does it not matter how the writing is accomplished, if the writing serves the writer's purpose?
And because I'm in the thinking mood, I might revisit my purpose in designing the assignment as an educator and whether that purpose is properly communicated to the student.
I believe all this thinking would prepare me well for a deep discussion with Timothy on his choice.
What about grading the prompts and if they are made increasingly more sophisticated ?
I do think prompting skills are going to need careful expert assessment. As writing improves I expect prompting to improve.
Have Timothy try the same prompt in several LLMs, compile all of the AI responses, assemble them into a portfolio, and then ask him to offer an oral report answering the following:
- What were the common themes found in all of the AI-based responses? Which LLM took the most creative path to developing the argument? Cite some comparative examples.
- How well did each LLM cite its sources? Did one LLM perform better than the others? How were they the same? How were they different?
- Which LLM demonstrated the best fluency in written language based on using definitive statements and rhetorical strategies? Did any LLMs "beat around the bush" and avoid making conclusive remarks?
- Which LLM produced the most readable essay? What elements of structure and language choices made it more readable than the others?
- Which LLM produced the strongest argument? On what basis do you make this assessment?
- As a user of an AI system, how could you have produced the output more efficiently? How could you have guided the AI system to produce stronger results or a better output format?
- How might this exercise, as an AI-based process, be applied to other situations where humans need reliable information from diverse points of view?
This is an example of an AI Augmented Learner operating in the metaphor of an orchestra conductor, creating meaning from applying scholarly skills in combination with acumen in operating AI systems as an agent (not a client). Read more: https://stevecovello.substack.com/p/what-is-scholarship-in-the-age-of
Great idea!! Maybe a little too time consuming considering it's just one assignment, but I personally would definitely think about going down this path. Not in a bunch of machines but maybe a few. Might not make the cut but in medias res but worth a thought. I hope we get some more takers, Steve. I really like your impulse here.
As the teacher, Timothy's choice might make me consider a few things
First, how is Timothy viewing the assignment; what is his own purpose in submitting the essay? Was he just wanting to get through a required task that is not important to him with the least effort? Was he perhaps accepting the assignment to be a demonstration of how he would solve the communication problem of persuading some audience to his own perspective on school dress rules, or was his purpose aligned at all with the teaching/assessment purposes of the assignment?
I might also stop to think about the relationship of the writing act to the writer's purpose. Does the purpose justify the means? Does it not matter how the writing is accomplished, if the writing serves the writer's purpose?
And because I'm in the thinking mood, I might revisit my purpose in designing the assignment as an educator and whether that purpose is properly communicated to the student.
I believe all this thinking would prepare me well for a deep discussion with Timothy on his choice.