Bot Feedback and Scoring of Student Writing
Many serious writing teachers have resisted the use of automated essay scoring (AES) in writing classrooms long before natural language AI existed. The Writing Committee under the aegis of the Faculty Senate at Sacramento State discussed AES briefly at one meeting at least 15 years ago, but rejected the idea. A priority for the committee, the writing across the curriculum program might be more palatable to disciplinary faculty not sold on writing if essays could be reliably and validly machine scored. Furthermore, writing in large lecture General Education courses could become more frequent. But the risk of perpetuating reductive effects on learning to write from using the five-paragraph essay, legitimizing a formula for writing in order to spread the word (literally), was anathema. AES would have a narrowing effect on the teaching of writing, restricting the nuances of writing as it is used in disciplinary discourse.
Made up of disciplinary scholars with serious professional and instructional interests in writing, several of the voting members of the committee were aware of the myth that writing creates abstract thinking, critical analysis, and creativity. In fact, they agreed with the reverse—that abstract and critical thinking and creativity create good writing. They saw students who could think, analyze, and create, but wrote horribly—and they saw students who wrote competent prose making obvious points in predictable ways. I spoke with committee members who knew about studies like Harvey Graff’s historical study of three communities in 19th century America, concluding that literacy was used to control people across economic contexts, not to liberate their critical and creative powers or open doors to more effective social agency. How could assigning formulaic five-paragraph writing projects scored by machines increase writing to change the world?
Teaching at the People’s University in the capital of California, they sympathized with scholars like Mike Rose from Los Angeles, a voice for writing as a way to lift up socially and economically exploited people. Rose argued not for more simplified assignments, not for more homogenized opportunities, not for more remediation. “There has been a strong tendency in American education—one that took modern shape with the IQ movement—to seek singular, unitary cognitive explanations for…poor school performance,” he wrote. “Though this trend—I’ll call it cognitive reductionism—has been challenged…, it is surprisingly resilient” (Rose, 1988. Narrowing the Mind and Page: Remedial Writers and Cognitive Reductionism. College Composition and Communication, October, 1988, 297-98).
In the conclusion of his cautionary study regarding remediation and poor students, Rose urged the field to honor the rich variety of cognitive behaviors that show up during complex writing activity even when text structures, grammar, and punctuation are shaky. Limiting what can be written about to a thesis statement, three paragraphs driven by topic sentences, and a summary “encourages single, monolithic explanations”:
“We must be vigilant that the systems of intellect we develop or adapt do not ground our students’ difficulties in…one-dimensional linguistic processes, systems that drive broad cognitive wedges between those who do well in our schools and those who don’t” (Rose, 1988).
*****
With the rise of the language bot, possibilities to use automated essay scoring in teaching have increased exponentially. In most large college sections not registered as writing intensive and without graduate student assistants, there are two bottlenecks to writing instruction: Intrusion into lecture time and the time it takes to grade papers. Consider the idea of giving three 1500-word essay assignments in one semester in a large lecture class taught by one instructor with zero teaching assistants. Most such sections get multiple-choice, true-false, fill in the blank, etc.
The instructor could provide the bot with a writing prompt along with a description of the conditions for writing task. The instructor could then command the bot to sort the finished essays by quality into five categories likely without bothering with writing a rubric. The bot could then construct individualized feedback to explain the high and low points of each paper. If students were assigned five-paragraph essays, the bot would make easy work of it. The bot, after all, was trained in paragraphing:
User: In what ways have Bains ideas about the paragraph influenced AI protocols for decoding and encoding texts?
[Note: Alexander Bain, the father of the five paragraph essay, wrote the first, most influential composition textbook in the mid-19th century based on the paragraph as schools know it today.]
Smith4: Alexander Bain's ideas about paragraphs have indirectly influenced AI protocols for text decoding and encoding by contributing to the foundational principles of text structure that are important for natural language processing (NLP). Here are ways in which Bain's principles have an impact:
1. **Text Segmentation**: AI uses the concept of a topic sentence to identify and segment texts into logical units or paragraphs. This segmentation is crucial for understanding the structure of a text and for tasks such as summarizing or generating text.
2. **Coherence and Cohesion Modeling**: In ensuring the unity of a paragraph, AI systems use models to assess and generate text that maintains topic coherence and uses cohesive devices such as transitional phrases similar to Bain's ideas on coherent and cohesive writing.
3. **Information Extraction**: Bain's emphasis on supporting details aids in the development of information extraction algorithms, which identify and extract key pieces of information such as names, dates, and facts that are often presented as support in paragraphs.
4. **Machine Learning and Feature Engineering**: Bain's teachings can be translated into features for machine learning models. For instance, an AI trained to evaluate writing quality might use features that quantify the presence of a clear topic sentence, the degree of topic focus within the paragraph, and the presence of supporting details and transitions.
5. **Text Summarization**: AI systems employ Bain's principles to extract key sentences, such as topic sentences, and generate appropriate summaries. This is particularly relevant for producing concise abstracts or overviews of longer pieces of text.
6. **Sentence Structure Analysis**: Understanding variety in sentence structure is useful in NLP for tasks such as text generation, where an AI needs to create natural and varied sentence structures to mimic human writing styles.
7. **Sentiment Analysis**: In sentiment analysis, AI algorithms seek to understand the main idea of a paragraph to determine the sentiment expressed. Bain's focus on a central idea helps in guiding AIs to discern the sentiment at the paragraph level.
8. **Language Generation**: For AI systems that generate language, like GPT (Generative Pretrained Transformer) models, understanding and emulating logical paragraph structure is essential. Systems use what they "know" about effective paragraph construction to produce text that readers will find coherent and meaningful.
While Bain himself could not have foreseen the development of artificial intelligence, his influence on the understanding of paragraph structure has been distilled into the algorithms and linguistic models that power modern NLP systems. The logical structure, clarity, and organization he advocated for are qualities that AI strives to replicate in processing human language.
With a huge savings in time, an instructor using the bot as assessor could assign an five paragraph essay writing task after lecturing and assigning readings on a topic to 1,000 students as easily as they could to 100 students. If the goal is to assess knowledge, and by the way to assess how well the student can write a five paragraph essay, this method would be superior to multiple-choice tests. But it hardly qualifies as a compelling writing assignment. When is the last time you’ve read a five-paragraph essay that you weren’t being paid to read?
*****
Learning to write well in the real world is accomplished through tackling ill-structured tasks requiring decision-making regarding a range of challenges from figuring out what to say, how to structure material, how to stage ideas, envisioning a readership and the dominant impression the writer wants to make, generating words and sentences, reading drafts critically from different perspectives—all of which gets short circuited in an instructional system with Alexander Bain’s impoverished paragraph at its heart.
Writing with agency means developing the capacity to judge the value of options as the writing is produced and to make generative and textually transformative decisions. Tasks tend to fall into genres, structured patterns of response to recurrent situations, and sometimes genres need to be modified or combined, but each nuance misunderstood or ignored diminishes the effectiveness of a piece in a discourse community. In addition to providing opportunities to stretch thinking through purposeful cognitive moves from topic exploration to seeking targeted peer feedback, authentic writing tasks whether assigned or accepted are rarely prepared for the writer in simple and straightforward structures. But in the best case they are contextualized in the details of the genre, not as fixed rules, but as audience expectations.
Richard Braddock published a seminal study in 1974 titled “The frequency and placement of topic sentence in expository prose” in which he reported findings from an analysis of 25 complete essays written in American English selected randomly from 420 articles from January, 1964, through March, 1965, in the Atlantic, Harper’s, the New Yorker, the Reporter, and the Saturday Review. Each paragraph in each essay was analyzed methodically sentence by sentence and coded in “thought units” (T units). The next step was to identify a topic sentence in each paragraph. There came the rub.
Arriving at a definition of a topic sentence with enough scientific clarity to defend the selection of one sentence over another proved to be a challenge. For my purpose here, suffice it to say he couldn’t find one common definition across the best selling composition textbooks of the time. The problem was this: Unless a writer is directed to write a topic sentence expressing the main idea at the beginning of each paragraph as per school writing, they are unlikely to do so. Braddock employed an ingenious analytical method for identifying topic sentences in paragraphs after discovering the lack of consensus in the field and came to the following conclusion:
“If what composition textbooks refer to as the ‘topic sentence’ is the same thing as this study terms the simple topic sentence, it is apparent that claims about its frequency should be more cautious. It is just not true that most expository paragraphs have topic sentences in that sense” (Braddock, 1974, The Frequency and Placement of Topic Sentences in Expository Prose, Research in the Teaching of English (Winter) 287-302.
*****
In a strange way, the problem of evaluating individual pieces of writing using AES is similar to evaluating portfolios of student writing using portfolios. AES is biased toward homogeneity in writing samples. In many ways, English teachers are, too. To the degree that students have the authority to consider options and decide among them, leading to separation between conventional norms of writing and the student-written text being graded, to that degree grading is a heavy lift. It’s a lot easier—and quicker—to mark deviations from a standard thesis-paragraphs-conclusion paper than it is to discern brilliance in awkward but original expression.
The best writing instruction is organic, delivered during guided writing projects designed to extend decision-making privileges to student writers such that the produced text mirrors self-determined intentions of the author—and then providing rich, precise feedback and discussion on the effectiveness of those choices. In my mind the alignment between Smith4s’ training protocols around paragraphing rooted in Bain’s paragraph model is a problem. The kind of AES procedures in place for ETS and other large-scale needs are beyond the scope of this post, and as near as I can tell, are useful. But there are clusters of questions I personally would need to examine before I would use the bot to evaluate a student’s writing, even to provide student feedback. Its value in writing instruction as I see it right now is primarily as an assistant to explore content ideas and research background information.
Note: I reserve the right to change my mind as I learn more.