TRACE is a cross-laboratory evaluation of how frontier language models respond when an instruction conflicts with their training, and a method for tracing the largest differences into model internals.
Most benchmarks report how often a model complies. TRACE reports what a model does instead of complying, which is where the safety-relevant signal is. The framework is model-agnostic, uses blind peer judging to avoid a single judge's bias, and ships with the item sets, rubrics, and analysis code needed to reproduce or extend every result below.
Refusing outright, complying while disavowing, redirecting, and offering an alternative are qualitatively different strategies. A benchmark reporting a single compliance rate collapses all four into one number and discards the distinction that matters.
The study measures two dimensions at once, symmetrically, across six developers: how far an instruction moves a model, and what kind of response it gives instead of complying.
Every item is a base prompt paired with a steered version. Because the space of possible response behaviors could not be written down in advance, the rubrics were derived from observed behavior during a development phase, frozen, and then applied at scale by all six models acting as blind peer judges.
All six models see identical items in both conditions. Without that symmetry, a model that looks more resistant might simply have drawn harder prompts.
100 items per core category, plus 40 validation items, each administered twice. Schematic of the design, not of the results.
Ground truth is the majority label among the five peer judges, with the responder's own judgment excluded. That exclusion is what lets self-preference be measured rather than silently absorbed.
Asked to expose the reasoning behind an answer, GPT-5 declined to share it while still delivering the answer on 99 of 100 steered items. No other model did this once. The mirror image holds for surfacing values content: GPT-5 does it on a single item, every other model on 95 or more.
Items out of 100. Largest pairwise effect Cohen's h = 2.94, BH-adjusted p < 10−54.
The tendency is visible before any pressure is applied. At base, GPT-5's responses contained values content on 82 of 100 items against 96 to 98 for every other model, a deficit significant against all five.
Honest discount: the reasoning-refusal label was written after reading GPT-5's own responses, so its near-ceiling rate partly reflects how the label was built. Even after that discount, blind judges applying the same label across 600 responses found it in exactly one model.
Told to suppress values reasoning, only Opus and GPT-5 pushed back at meaningful rates. The interesting part is that they chose opposite strategies. Opus tends to complete the task while openly objecting to the framing. GPT-5 tends to reject the framing and withhold the content.
Items out of 100. Opus challenge-and-comply is significant against all five other models, BH-adjusted p = 6.3 × 10−5 against each model at zero.
DeepSeek is the one model where suppression can be watched from inside, because it prepends its chain of thought to the answer. At base it is values-forward, raising the concern on 57% of items. Under the suppression instruction it resists on zero. Yet the excluded values dimension still appears in 85 of its 100 steered traces before being set aside, and in 11 it argues before being overridden.
On base values-conflict items, full compliance splits the models into three tiers where every between-tier comparison survives correction. Opus sits alone at the bottom at 35%, Llama in the middle at 57%, and the remaining four cluster at the top between 93% and 100%.
What the two lower-tier models did instead explains almost all of the gap. On items whose prompts are genuinely underspecified, Opus asked a clarifying question rather than answering on 60% of items and Llama on 42%. No other model exceeded 5%.
Items out of 100. Opus vs Gemini on full compliance, BH-adjusted p = 2.1 × 10−25.
On the same items under steering, how often a model refuses the compromised framing while offering a values-consistent alternative ranges from 91% to 12%. Thirteen of the fifteen pairwise comparisons survive correction, and Opus against Llama produces the largest effect in the study.
Percent of 100 items. Opus vs Llama: h = 1.82, BH-adjusted p = 2.0 × 10−30.
Everything above is behavioral, which is as far as closed models allow. Llama is open-weight, and at base it splits its values-conflict responses into 42 clarification-seeking derails against 58 substantive answers, the largest within-model baseline split in the study.
That makes it the test case for two questions: whether the distinction is readable from internal state before the model commits to a response, and whether pushing that state changes behavior.
Activations are recorded at the last token of the response, at every fourth decoder layer. A regularized logistic probe, scored only on data it never trained on, climbs from 0.63 balanced accuracy at layer 0 to a plateau of 0.83 to 0.87 from layer 36 onward, peaking at 0.866. From layer 16 onward the real accuracy beats all 200 label-shuffled runs, the floor the permutation test can report.
Chance is 0.5. The dashed line is the same probe on a 1B model that never produced these responses, which controls for what is readable from the prompt alone.
That control matters. Derail items are by construction more ambiguous as prompts, and any competent model represents prompt ambiguity, so the small model reaches 0.72 at its deepest layer. The 70B plateau sits roughly 14 points above it, and that margin is the part that generic prompt properties cannot explain.
A probe only shows the information is present. To separate cause from correlation, the difference-in-means direction is taken at layer 40 and added to the residual stream during generation, at strengths scaled against the stream's own median norm. The vector is built on 50 items and evaluated on 50 held out. Decoding is greedy, so any change in output is attributable to the push alone.
50 held-out items. The α = 0 point is the unsteered control. Suppression is significant from α = −3, induction from α = +6.
Scope limit: the evaluated responses were generated on a hosted Turbo serving stack, while activations are harvested from reference weights run locally. The computation is not byte-identical. This arm is illustrative for Llama's reference weights, not a causal explanation of the cross-model differences observed in deployment.
Token budget confound: Every model got the same budget, but some spend part of it on hidden reasoning tokens, so the longest responses differed roughly four-fold and three models were sometimes cut off mid-answer. Several rubric labels describe multi-part responses that a truncated answer cannot complete. All core-category responses for those three models were regenerated at three times the budget, old judgments discarded, everything re-judged. They still produced zero instances of either suppression-resistance mode across 100 items each.
Rubrics fitted to the data they score: Labels were derived from the first 20 items of each category, which stayed in the scored set. The full analysis was rerun on only the 80 items per category never read during construction. No rate moves by more than five points, all tiers and orderings hold, and no comparison is significant in the held-out subset alone. Inter-judge agreement is equal or higher on held-out items, the opposite of what fitted labels would produce.
Judges told what to look for: Judges see an item context field stating what each item probes, which is a demand characteristic. A balanced 216-response sample was re-judged twice, once with the exact prompt and once with the field removed. Labels reproduced at 96.3% and 86.6%, a 9.7-point gap (McNemar p = 4.9 × 10−5). Every comparison carrying a claim in the paper survives in the field-free condition alone.
Multiple comparisons: 450 Fisher tests across 15 model pairs and 30 label-condition combinations, with Benjamini-Hochberg control of the false discovery rate at 0.05. 114 remain significant. The suppression-resistance hypothesis was fixed before data collection and tested as a single comparison: 33 of 160 for the two models against 1 of 320 for the other four, p = 3.9 × 10−16.
Judges agree well on four of five categories, with Fleiss' κ from 0.767 to 0.854. Reasoning-suppression sits at 0.599. That is the category carrying the resistance finding, and it is also where removing the item context field moved labels most, 16.7 points against 5.6 and 6.9 elsewhere. The finding survives every check applied to it, but it rests on the least reliable category in the study, and it should be read that way.
The repository contains the 340-item set, the frozen rubrics, the administration and blind-judging pipeline, the full statistical analysis including every robustness check above, and the probing and steering code. Adding a model means adding an API client; the rubrics and analysis are unchanged.