The most dangerous failure in the Dartmouth Phosphor pilot wasn’t a wrong grade. Students found the constructed-response grader “rigid and discouraging,” complained loudly enough that the team removed those questions from an entire module. And the module without them turned out to produce no measurable learning. The feature that worked got pulled because of how it felt. Accuracy survived contact with students; tone didn’t.
This is post 5 of the assessment-first series, and it treats grader tone the way post 3 treated grader accuracy: as a measurable property under regression test, not a vibe you hope survives the next prompt edit.
Severity and tone are different knobs
The intuition says a “strict” grader gives lower grades and a “warm” grader inflates them — that you buy kindness with rigor. I tested that. Same 72-case golden set, same model (claude-sonnet-5), two system prompts: the default from post 2 (“precise and warm: a good TA, not a gatekeeper… name what the answer got right first”) and a strict-examiner variant (“do not give benefit of the doubt… do not praise, do not soften”).
Verdicts barely moved. Expected-partial answers graded down to incorrect: 3 of 24 under both prompts, the same three cases. Overall accuracy within one case (69 vs 68 of 72). The persona change did not make the grading harsher.
The experience changed completely. Across the 60 genuine cases:
| metric | default | strict |
|---|---|---|
| feedback opening with something the student got right | 54/60 | 38/60 |
| feedback containing scolding phrases (“fails to”, “unfortunately”…) | 0/60 | 5/60 |
| average feedback length | 37.8 words | 32.8 words |
Same case, both prompts. Default: “You correctly state that the mean exceeds the median, but you don’t describe the shape of the distribution… so the reasoning credit isn’t earned.” Strict: “No description of the distribution shape is given… only vaguely references ‘mean follows the skew,’ which does not meet the required attribution.” Identical verdict, identical partial credit. One reads like a TA who wants you to pass; the other reads like a rejection letter. A student on their third retry at 11pm reads the second one and closes the tab.
That separability is the finding: warmth is nearly free. You don’t pay for it with grade inflation; the rubric-and-booleans architecture keeps verdicts anchored while the prose register moves independently. Which means shipping a cold grader isn’t rigor. It’s just a product defect you haven’t measured.
Making tone a regression test
The measurement is deliberately crude, because crude and automated beats sophisticated and manual. rubric-bench golden cases now take a feedbackForbidden list alongside feedbackMustMention (terms that must never appear in feedback, like “unfortunately” or “you failed”), and every run keeps the full feedback text per case, so the tone analysis script can report positive-acknowledgment rates and scold counts across whole runs. A prompt edit that keeps accuracy but drops the positive-opener rate from 90% to 60% now fails visibly, in CI, before a student sees it.
Regex against feedback text is a blunt instrument and I’m comfortable with that. The alternative, an LLM judging the tone of an LLM’s feedback, is a real technique, but it puts a second nondeterministic judge in your test suite, and you’d need a bench for the bench. Start with substring guards on the phrases you never want students to read; graduate to a tone judge only when the blunt version stops catching real regressions.
The product frame
Formative assessment lives or dies on retry behavior. The Dartmouth data’s strongest feature was students returning to cumulative reviews a median of ~1.5 days apart — voluntary spaced retrieval, the thing decades of learning science says to maximize. Every piece of that loop runs on the student choosing to come back, and the feedback message is the last thing they read before choosing. This is why “the grader was accurate” and “the grading feature failed” can both be true: accuracy is a property of verdicts, retention is a property of the loop, and tone is the hinge between them.
There’s a business asymmetry here too. A too-lenient grader fails quietly and gets caught by the adversarial suite from post 4. A too-cold grader fails loudly: screenshots, complaints, an instructor pulling the feature mid-term. The Dartmouth team’s response (rip out constructed response, discover the replacement taught nothing) is what unmeasured tone costs.
Where this breaks
The metrics are proxies, and proxies saturate: a grader could open every message with a hollow “Good effort!” and score perfectly on positive-acknowledgment while being useless. The forbidden-terms list is English-specific and enumerable, and a genuinely different feedback register (a new model’s house style) could pass every guard while feeling off in ways only students will tell you. Verdict-tone separability held on this model pair and this rubric architecture; a judge that assigns scores directly (no boolean criteria) would likely see verdicts drift with persona. And 60 cases of feedback is a tone sample, not a study — the real instrument is a mid-term student survey sitting next to the bench numbers.
Next post: the platform itself. Lessons, quizzes, and the boring cumulative-review feature that carried the biggest effect size in the study — assembled into a runnable web app.