Learning to Learn from Learning at Scale: Warm, Cool, Hard

There was a golden moment in American education — a genuine awakening to the need to change schools to meet a changing world. There was an appreciation for the wisdom of seeing local schools as communities of teachers and learners with great potential to humanize and more thoroughly convince young people of their remarkable capacity to learn.

Before those of us who lived through it could catch our breath, however, it was gone, and what replaced it brought us to an unfortunate space in which public schools are sailing in troubled waters. Looking back at this moment, young educators might take away a sense of possibility — a reminder that schools have briefly, genuinely been organized around something better than a test score.

The 1980s Reforms: Wave I

In 1986, the Carnegie Task Force on Teaching as a Profession released a seminal report titled A Nation Prepared: Teachers for the 21st Century, which is unavailable online except in bits and pieces. The report was a direct response to the earlier report A Nation at Risk (1983). In Risk, teachers were portrayed in my opinion as factory workers in need of executive, top-down, state curriculum directives.

If the economy required workers who could think, learn continuously, and solve unfamiliar problems, however, teachers needed to be professionals themselves capable of developing those capacities, not technicians delivering a scripted curriculum. Nation Prepared directly led to the creation of the National Board for Professional Teaching Standards and gave the restructuring movement much of its intellectual vocabulary.

The economy Risk described no longer needed mindless compliance from workers. It needed judgment, workers who could read complicated material, evaluate and construct arguments, apply quantitative reasoning to problems they’ve never seen before.

What employers were already finding, and would increasingly find, is that graduates trained on routine tasks couldn’t do any of this. The skills required weren’t routine. They were the skills of pattern recognition, of creative problem-finding, of learning continuously as the terrain shifted.

The implication for higher-level teaching went unstated but was structural: you cannot develop in students what you do not practice yourself. A teacher who can help learners figure out what they need to know to do what they need and intend to do, where to find it, and how to make meaning from it, that teacher is already working as a reflective practitioner. The ethical weight of practice lands not on coverage and testing, not on compliance, but on whether the learner leaves better equipped to think.

Prior to Nation Prepared, teacher certification was handled entirely by individual state governments through bureaucratic licensing procedures, which varied wildly in quality and mostly failed to reflect the high-level expertise required for professional practice. The most immediate and literal influence of Nation Prepared was that it explicitly called for the establishment of a National Board for Professional Teaching Standards.

Nation Prepared envisioned teachers as autonomous professionals and “shapers of reform.” The certification process it called for was designed to be deeply reflective and evidence-based, built not around minimum competency requirements but around rigorous benchmarks defining what accomplished teachers should know and be able to do. It required teachers to submit portfolios of student work and video recordings of their classroom instruction — evidence that they could make complex pedagogical decisions independently and demonstrate their impact on student learning.

California’s school restructuring efforts during the 1990s materialized in the context of Nation Prepared under Senate Bill 1274 (SB 1274), officially known as the Demonstration of Restructuring in Public Education program. The historical link between Nation Prepared and the emergence of SB 1274 in 1990 represents a profound ideological shift often viewed as the force that changed from Wave 1 to Wave 2 of educational reform.

Post-1980s Reform: Wave II

California provided massive state grants amounting to roughly $150 million over its lifespan to schools to restructure their curriculum, assessment, and governance, provided their innovations were firmly anchored in evaluating authentic student classroom work rather than relying on evidence from standardized testing to inform their professional judgments.

This expectation pushed schools to collaboratively design “more authentic measures of student learning,” fulfilling the overarching goal of SB 1274: to empower teachers with the professional autonomy to redesign their schools from the bottom up, judging their success by the tangible, complex work their students produced.

My middle school was awarded a sizable grant from the California Center for School Restructuring (CCSR). Having served as the committee’s scribe to prepare our proposal under the direction of Bill Giachino, a gifted middle school principal and dear friend whom we have lost, I drafted the proposal in which the school made a long-term commitment to “examining student work” collaboratively and in public.

My dissertation study would never have been possible without SB 1274 funding. As one of several action elements of our proposal, we would design, implement, and evaluate the impact of a portfolio assessment system without referring to standardized test data — an experiment with three English teachers teaching portfolio classrooms and four English teachers serving as raters at the conclusion of each trimester. Interested readers can find my book published by NCTE on ERIC.

The mechanism at the center of the SB 1274 examining work was the Tuning Protocol, developed by Joseph P. McDonald and David Allen as staffers at the Coalition of Essential Schools. The portfolio project was a small resource allocation; the bulk of funding went toward facilitating and practicing the Tuning Protocol on the campus and at state-designated sites where groups of teachers from different SB 1274 projects demonstrated their protocol work.

Five high schools in the Coalition’s Exhibitions Project needed a structured way to receive feedback on their developing student assessment systems — exhibitions, portfolios, performance tasks. The Coalition was Ted Sizer’s network, built on the principles of Horace’s Compromise (1984), and the assessment problem it faced was precisely the one Sizer had identified.

If you were going to ask students to demonstrate competence through genuine performance rather than standardized testing, you need a professional community capable of judging that performance together, consistently, without either dissolving into polite approval or destroying one another with punitive evaluation.

McDonald and Allen’s protocol answered that need with a sequenced structure. Educators brought samples of student work — photocopies, video clips, performances — and a focusing question. After a period of silent examination, participants offered feedback in a specific order: warm observations first, acknowledging what was alive in the work; cool observations second, often framed as probing questions; and then hard questions driven by criteria against which the work was being evaluated. The presenter listened in silence during feedback, then reflected aloud.

The protocol had its trial run in 1992 and traveled quickly. The Center for Restructuring adopted and adapted it for California’s restructuring sites, scaling it from the assessment of individual student work to the examination of whole-school reform questions.

The Emergence of Teacher Learning Communities

The SB 1274 grants funded structures. What they could not buy was culture. Martin Krovetz, writing in The School Community Journal in 1993, made this distinction the center of his contribution to the restructuring movement.

Drawing on a fifteen-school network in four California counties — three of them operating under SB 1274 funding, five carrying planning grants — Krovetz documented what the legislation’s architects had assumed but not said: that restructuring without re-culturing changed little regarding student learning.

Opening his 1993 article with Roland Barth’s formulation, Krovetz grounded his argument in a precondition:

“The relationships among adults in schools are the basis, the precondition, the sin qua non that allow, energize, and sustain all other attempts at school improvement. Unless adults talk with one another, observe one another, and help one another, very little will change.”

Structural changes — new governance arrangements, altered schedules, interdisciplinary teams — generated motion, not transformation, when the adults in the building remained what Krovetz described as compartmentalized and engaged in parallel play, competing rather than collaborating.

The connective tissue SB 1274 required was a collegial learning community: teachers supporting each other in developing more authentic measures of student learning. Krovetz described the key elements of such communities as choice, support, stimulation, trust, and respect for the professional judgment of peers.

His Principal Support Network — fifteen principals meeting monthly, talking openly and confidentially, helping each other think through strategies for leading change — was one instantiation. The Professional Development Support Teams in Santa Cruz, where teachers chose to work in peer coaching groups in lieu of formal evaluation, were another.

What the SB 1274 sites that developed impressive approaches had in common was not grant size. It was whether the adults had built the kind of trust that made looking at student work together — publicly, critically — not only possible, but productive.

What Hard Questions Reveal

Hard questions are humbling, the main reason the Tuning Protocol became such an important restructuring tool. If teachers don’t regularly look at student work from their colleague’s practice, a core resource is kept off the table. It can be hard for teachers to let outsiders look at their students’ work through a critical lens. If teachers look at student work just to celebrate or to pick up some cool ideas for new or revised assignments, the real questions are never addressed.

Hard questions are challenging for teachers and learners alike. In the portfolio project during an interview, a seventh-grader named Clarissa described what happened when she stopped treating the portfolio assessment criteria as a checklist and started treating them as genuine descriptions of what writers do. One criterion had to do with experimentation; effective writers try ideas out experimentally and learn from them.

“In the second trimester, I learned about experimenting. In the diary I made, I used experimenting as a way to achieve criteria. But in this trimester, though, I used experimenting as a way to help me write. Not to experiment just to experiment. I used experimentation as an answer to when I got stuck in writing poetry.”

The shift Clarissa names — from performing criteria to inhabiting them — is precisely the shift that the hard-question mindset is designed to produce. She went further, describing her revision process for a single poem:

“I think I have written 3 different versions of that poem. Trying to use the right words, the right form. I think what pushed me into doing these was when I letted someone read it, and they didn’t get the meaning. I know that I knew what it meant, but the reader didn’t. That was important.”

Clarissa had become, in the language of the rubric, a student who “writes like a reader.” The criterion was important to all three of the portfolio teachers, and the examination teachers were prepared to ask this hard question at grading time. Naming it by agreement for teachers and as a criterion for self-assessment for students made it a target worth thinking about, strategizing, worth pursuing.

Another student, Richard, began the year angry that his first-trimester portfolio had earned a B instead of an A. By year’s end he had revised his understanding of what grades were for:

“After the first trimester I thought this portfolio project was a bad idea, because I didn’t get the grade I wanted. I cared more about my grade than what I was learning, which I won’t be doing any more.”

What changed was Richard’s relationship to the criteria and what he cared about. He stopped reading them as potential obstacles to getting the grade and started reading them as descriptions of something he might actually learn to do.

Not every student made that shift. Some remained at least partly in what one researcher called the “achievement club” — students who aligned their behavior with the criteria in form rather than in substance: “All of my work has been built around the criteria sheet, and I have used the criteria sheet to make revisions to my work.”This category of student met the criteria without being changed much by them.

Having interviewed Richard, I had a clear and empathetic understanding of his position. After all, nothing had changed in his other classes. This experiment would be over. He promised himself that he would not forget this insight. In the end, it was about what he learned, not what his teacher decided to enter on his report card.

A student named Sophie, reflecting near the end of the year on why portfolio students had outperformed their control peers on a direct reading assessment, located the explanation not in the portfolio mechanism itself but in what the criteria had done to the culture of the classroom:

“Probably the public criteria made a difference. I don’t think the traditional classrooms have the criteria, and since we did our work based on those criteria, I think that it made us grow a lot more than the traditional classes would because they didn’t have that criteria. I mean, you have, like, a goal and you want to meet that goal.”

Her classmate Jackson connected the criteria directly to his perception of what the teacher felt compelled to do: the teacher “wanted to prepare us to the point where we could get a good grade because if she hadn’t prepared us, students in the class wouldn’t be getting A’s or B’s.” Public criteria, in other words, changed students’ perception of teachers — not just of grades and assessment.

When criteria are public, shared, and seriously applied, they change what and how teachers teach, what and how students attempt in their work, and what teachers and learners alike understand the work to be for.

The Grading Handoff: How Does One Teach without Grades?

Before the first scoring session, the portfolio teachers and examination teachers met to discuss the upcoming process, and the conversation quickly turned anxious. Who would answer parent phone calls about grades? What would happen if the examination committee’s judgment differed sharply from the portfolio teacher’s own assessment of a student’s work?

One teacher admitted that her reputation as a strong English teacher felt at risk. Another worried that students who had not yet produced sufficient evidence would pay a price for her own inexperience with the system.

What held the arrangement together was the shared rubric and a formal agreement to abide by the committee’s decisions and consult in good faith when disagreements arose. The portfolio teachers had agreed, in principle, in advance, to accept the grades the committee issued.

That agreement was not easy. It required teachers to trust that criteria, which they had a hand in writing and vetting, stated clearly enough and applied consistently enough could substitute for the intimate knowledge a teacher accumulates about her students over months of daily contact.

In most cases, that trust was warranted by a statistical correlation. The Pearson r between grades the portfolio teachers would have assigned and grades the examination committee actually issued ranged from 0.86 to 0.95 across three semesters.

When discrepancies arose that couldn’t be resolved by the examination jury, the teacher-of-record served as a validation scorer and the two parties discussed the portfolio before a final grade was reported. The process was not frictionless. But it worked because everyone involved had agreed, before a single portfolio was scored, that the criteria had to match the evidence in the eyes of the jury who could consult the teacher — not just the teacher, not just the committee, and not the relationship between them.

The End of the Experiment

The SB 1274 Demonstration of Restructuring in Public Education program reached its statutory sunset at the end of the 1997–98 academic year. The California Center for School Restructuring closed its doors, released its final summative reports on the 144 participating demonstration schools, and the specific infrastructure for evaluating authentic student work through the Tuning Protocol was left to individual schools and districts to sustain without state support.

The program was not terminated prematurely. It simply expired. But its funding and philosophy were not renewed, and the political conditions that had made it possible were already gone. By the late 1990s, the state legislature had moved decisively away from bottom-up restructuring and authentic assessment philosophy that had animated the early part of the decade.

The pivot was not gradual. California was simultaneously expanding the charter school movement — Senator Gary Hart, the author of SB 1274, had a hand in that, too — and preparing for the standardized accountability mandates that would culminate nationally in No Child Left Behind.

The irony is structural. The same legislature that had, in 1990, bet on teacher professional judgment as the engine of school improvement had, by 1998, reverted to Nation at Risk (1983) and concluded that teacher professional judgment was precisely what needed to be controlled.

The warm and cool and hard lenses that had focused teacher attention on student work during the restructuring era were replaced by a single instrument, not even a lens but an external probe: the standardized test. One temperature. No texture. No protocol for what happens when the score and the student don’t match.

Clarissa had learned to use experimentation as an answer to getting stuck when she writes, not as a criterion to perform for a grade. Richard had stopped caring more about his grade than what he was learning. Sophie had identified public criteria as the mechanism by which her class had grown on the local authentic reading assessment given to every student more than peers getting traditional grades. None of that evidence had a place in the accountability system that followed.

What the restructuring era demonstrated — briefly, incompletely, at real scale in California schools — was that teachers examining student work together, under a protocol that sequenced appreciation and analysis before judgment, could build the kind of professional community that made genuine learning criteria functional and powerful.

For me, the timing couldn’t have been worse. By 1998, any support for responsive teaching using authentic assessment strategies had been quashed. There was absolutely zero interest in portfolio assessment. Yet the next year, my book titled The Portfolio Project was published by NCTE.

By 2010, schools were moving toward the Common Core State Standards with little interest in engaging students in self-regulated and self-aware learning in relation to clear and actionable public criteria.

David Coleman, the chief architect of the Common Core, gave that indifference a voice when he famously proclaimed that adolescents simply need to get used to the fact that “nobody gives a sh***t about what they feel or think.”

Coleman’s project decided on one temperature, not warm, not cool, but hot. The Common Core rejected feelings, no celebration, no wondering, no critical friendship. The problem was not simply that tests replaced portfolios. A public culture for making disciplined judgments about student learning had been abandoned — and nobody, it seemed at the time, gave a sh***t.

---

References

Barth, R. S. (1991). Improving schools from within: Teachers, parents, and principals can make the difference. Jossey-Bass.

Carnegie Forum on Education and the Economy, Task Force on Teaching as a Profession. (1986). A Nation prepared: Teachers for the 21st century. Carnegie Corporation. [Available in excerpts via The Chronicle of Higher Education, May 21, 1986]

Krovetz, M. L. (1993). Collegial learning communities: The road to school restructuring. The School Community Journal, 3(2), 71–82. https://www.adi.org/journal/fw93/KrovetzFall1993.pdf

McDonald, J. P., Mohr, N., Dichter, A., & McDonald, E. C. (2003). The power of protocols: An educator’s guide to better practice. Teachers College Press.

National Commission on Excellence in Education. (1983). A nation at risk: The imperative for educational reform. U.S. Department of Education.

Sizer, T. R. (1984). Horace’s compromise: The dilemma of the American high school. Houghton Mifflin.

Underwood, T. (1998). The consequences of portfolio assessment: A case study. Educational Assessment, 5(3), 147–194.

Underwood, T. (1999). The portfolio project: A study of assessment, instruction, and middle school reform. National Council of Teachers of English. https://eric.ed.gov/?id=ED463359

California Legislative Analyst’s Office. (1991). School restructuring in California [Analysis]. https://lao.ca.gov/1991/010191_school_restructuring/010191_school_restructuring_in_california.html

California Center for School Restructuring — final summative reports cited in: Finnan, C., & Meza, J. (1999). School restructuring and the problem of implementation. Education Policy Analysis Archives. https://digitalcommons.usf.edu/cgi/viewcontent.cgi?article=1413&context=usf_EPAA

SB 1274 implementation timeline and sunset documentation: https://files.eric.ed.gov/fulltext/ED425500.pdf