Teacher as Reader: Pamela Moss's 1992 Portfolio Assessment Framework for the AI Writing Era
Pamela Moss, an influential voice in discussions about validity in assessment, raised the issue of consequential validity in standardized testing in the 1990s. It isn’t enough for a test to claim construct validity, i.e., the test measures what it says it measures and nothing else. In addition, each use of test scores as the basis for action must also demonstrate consequential validity, i.e., its use must have a positive effect1.
The problem with multiple-choice standardized tests may or may not include poor construct validity. For example, such designs do not measure natural reading comprehension and so is invalid. We could argue this point. But we can’t stop there, according to Moss. Because these tests narrow the curriculum and hamstring teachers, they are not consequentially valid and ought not be used.
*****
In 1992 Moss advocated for a more dialectical approach to validity, one that continuously examines the interplay between evidence, values, and consequences2. At a moment when AI threatens to upend traditional writing assessment, a little-known framework from 1992 offers surprising guidance.
When Moss argued that inter-reader reliability was the wrong metric for portfolio assessment, she couldn't have imagined ChatGPT. In other words, she argued that two independent readers of a single portfolio may or may not agree and likely won’t. But that’s beside the point. The classroom teacher-reader in Moss’s paradigm is the single best reader of their own students’ portfolios. Her insistence that teachers read student writing developmentally rather than evaluatively provides exactly the principles we need to assess writing in an AI-integrated classroom.
*****
Moss challenged fundamental assumptions about writing assessment. The classroom teacher, she argued, was not a scorer but a witness to development, not a judge but a collaborator in meaning-making. High inter-reader reliability, that holy grail of direct assessment, actually missed the point entirely. The most reliable reader was the one who had witnessed the journey.
Consider how Kathryn Howard, one of Moss's teacher-collaborators, read her student Barry's portfolio. In his first personal monologue, Howard saw beyond surface scatter to notice a writer struggling to control his text.
By January, in his Lord of the Flies essay, she tracked emerging command of transitions and evidence use. In his Tell-Tale Heart piece, she noted his growing awareness of audience. By his final Animal Farm essay, she celebrated not just his sophisticated theme but his command of revision, moving whole sections, reshaping arguments, developing ideas through multiple drafts.
*****
This developmental reading (non-evaluative) becomes even more crucial in an AI era. When students can generate proficient prose, we need teachers who can read beyond surface polish to track genuine growth. Moss's framework, with its emphasis on vision, development, craft, response, and sense of writer, offers a calibrated lens for this new challenge.
Where Moss originally defined vision as "the reader's reconstruction of the writer's semantic intent," today's teachers must distinguish between AI-generated and student-developed semantic intent. They need to track the emergence of genuine questions in student bot prompts, critical challenges to bot responses, to notice when students move beyond AI suggestions to develop original ideas, and to recognize real intellectual risk-taking rather than clever prompt engineering.
The development of ideas and growth in expression takes on new complexity in digital spaces. Traditional development traced how ideas connected and arguments built. Today's teachers must also read for student decision-making about when to use AI, watching how they accept or reject AI suggestions, tracking authentic development through multiple drafts, and distinguishing productive struggle from AI dependence.
Craft in the digital writing workshop requires expanded attention. Beyond traditional focus on language, form, and style, teachers now must track intentional choices about when to use AI. They watch for voice development in imagery and details, looking for personal expression rather than lackluster compliance. They learn to distinguish original figurative language from AI-generated metaphors, personal resonance from prompted description.
Teacher-as-Reader response, too, must evolve. Traditional teacher responses focus on coherence, evidence, arrangement, and the like. Today's teacher-readers must notice authenticity, distinguish genuine from artificial engagement, and recognize real thinking beneath digital glitz and pablum. Teachers must become skilled at seeing where personal connection is wholly personal despite defensible programmed response.
The sense of writer as digital collaborator presents new challenges for teacher-readers. Where they once tracked writer self-awareness, teacher-readers must now read for understanding of the nature of AI collaboration. They must notice how students reflect on tool use decisions, watching for the development of authentic voice. This metacognitive dimension becomes crucial as students learn to navigate writing with AI support.
*****
Moss’s framework expanded offers crucial guidance for schools struggling with AI integration. Instead of focusing on detecting AI use or preventing AI "cheating," we can focus on developing students' capacity to use AI strategically while maintaining authentic voice and genuine intellectual growth.
The implications extend beyond individual classrooms to how schools might develop portfolio systems, support teacher development, and maintain meaningful accountability while embracing technological change.
Most importantly, this approach maintains focus on what matters most: students developing as writers and thinkers. When teachers read developmentally rather than evaluatively, they can guide students toward strategic tool use while nurturing authentic voice.
The challenge of AI in writing instruction isn't going away. But neither is the need for students to develop genuine writing capacity. Moss's framework, reimagined for our digital era, offers principles for maintaining what matters in writing instruction while embracing new tools. The teacher as developmental reader, rather than AI detector or evaluative scorer, becomes key to navigating this new seascape.
https://www.academia.edu/121454676/The_Role_of_Consequences_in_validity_Theory
Moss, Pamela A.; Beck, Jamie Sue; Ebbs, Catherine; Matson, Barbara; Muchmore, James; Steele, Dorothy; Taylor, Caroline; Herter, Roberta (1992). "Portfolios, Accountability, and an Interpretive Approach to Validity." Educational Measurement: Issues and Practice 11(3): 12-21. <http://hdl.handle.net/2027.42/72001>