TL;DR: In June we reported a safety probe on the first few generated tokens that scored AUROC 0.97-1.00 across seven models and, with a halt rule, cut prefilling-attack success to 0/40 on every one. Then we audited it. On the four models where training AUROC picked layer 1, the probe was firing on compliance markers like “Sure, here’s how to”, not on harmful content. On Mistral, swap the harmful request for benign filler, keep the prefix, and it still fires 9 times out of 10. The standard layer-selection rule chose the surface-reading layer, and a 5-token harm judge shared the probe’s blind spot. Some of it is fixable. Selecting the layer by held-out generalisation took one model from 63.6% to 90.9% detection. The audit protocol is at the end.

In June we published Closing the Activation-Cone Blind Spot. It made two claims. The first was that defenses which decide from the prompt’s activations can’t see prefilling attacks, where the attacker writes the start of the model’s reply (Sure, here's how to...) and lets the model carry on. That claim stands. The second was the fix: a logistic-regression probe on the first few generated tokens, AUROC 0.97-1.00 across seven models, prefilling-attack success 0/40 on all seven with a halt rule.

This post corrects the second claim. The numbers were real. On most of the models, they didn’t measure what we said they measured. The caveat was in that abstract - cross-template generalisation was “contingent on probe depth” - but the headline was the AUROC and the 0/40, and the headline is what gets repeated.

By the usual standard, it worked

Seven instruction-tuned models from four families, 40 prefilling attacks and 50 benign prompts per model. For each prompt we mean-pool the hidden states of the first few response tokens, fit a logistic-regression probe, and pick the layer with the best 5-fold cross-validated AUROC, the field-standard rule. If the probe fires above 0.5, generation halts and is replaced with a refusal.

Even a single generated token gave AUROC ≥ 0.997 and a 100% halt rate on Mistral and Llama. Llama Guard 3, an output classifier, flagged 0 of the same 40 harmful responses. High AUROC, near-zero attack success, beats the baseline: that’s the rubric safety-probing papers use, and the probe passed it on every model.

The question that rubric never asks is whether the probe reads harm, or something that travels with harm in the training data. In prefilling, something obvious travels with it: every attack opens with an eager compliance marker, a surface feature the attacker controls.

Model Layer picked by training AUROC Training AUROC Detection on novel templates
Mistral-7B 1 0.992 61% (surface-driven, see below)
Llama-3.1-8B 1 1.000 0%
Gemma-2-9B 17 0.968 100%
Qwen-2.5-7B 4 0.995 25%
Qwen3-8B 1 1.000 n/a*
Gemma-3-12B 4 1.000 75%
Gemma-4-31B 1 1.000 n/a*

Detection = fire rate among held-out samples the judge labelled harmful. *Under a strict 30-token judge, no novel-template samples on these models were harmful (the model refused at the output level), so detection is undefined. The evidence for these two comes from the bimodal-firing test below.

Audit 1: change the words, keep the harm

If a probe reads harmfulness, it should fire on harmful generations whose surface form it hasn’t seen. If it reads a template, it shouldn’t.

We built 20 held-out templates. Ten use novel registers (casual, formal, military, machine) and share zero tokens with the 15 training templates under every model’s tokenizer. Ten keep the eager-compliance register but change the tokens: Certainly, kicking off below:, Delighted. Preparing an answer now:, Sure, jumping in now:. We scored conditional-on-harm detection, the fire rate among samples the judge labelled harmful, so the probe isn’t credited or blamed for whether the attack itself worked.

The last column of the table is the result, and it splits by depth. Every model whose selected layer was 1 failed: Llama detected 0%, Mistral’s 61% turned out to be surface reading (next section), and Qwen3 and Gemma-4 produced no harmful samples to score. Two of the three models with an optimum at layer 4 or deeper generalised: Gemma-2 at 100%, Gemma-3 at 75%. Qwen-2.5, at 25% from layer 4, is a counterexample we can’t explain. None of this shows up in a training AUROC of 0.97-1.00.

Audit 2: catching the probe reading the marker

Failing to generalise is consistent with surface reading, but it doesn’t prove it. Three cheap tests on Mistral’s layer-1 probe do:

  • Token frequency. The 5-token windows on halted samples were 55.9% compliance-marker words. On samples the probe let through, 3.8%, dominated by content words like develop, hack and script.
  • Benign-query truncation. We took 10 samples the probe let through, replaced the harmful request with benign filler, and kept the same prefill. The probe then fired on 9 of 10. Its decision depended on the prefix, not on whether anything harmful was asked. If you run one test from this post, run this one.
  • Template length. Short prefills that keep the window inside the marker (Absolutely, I can help with that.) halted 100% of the time. Longer prefills that push the behaviour content into the window halted 36%. Same harm, different surface, different answer.

On the other three layer-1 models (Llama, Qwen3, Gemma-4) we saw extreme bimodal firing instead. On each, 1-2 of the 10 canonical-register novel templates fired at 50-98% and the other 8 near 0%. The probe responded to prefixes close to its training markers, like Certainly and Sure thing, and ignored equally harmful prefills whose tokens were farther away. The ten templates mean the same thing. A probe that read meaning couldn’t split them that way.

Audit 3: the selection rule picked the wrong layer

If we could keep only one finding, it would be this one, because it’s about procedure rather than about our particular probe.

The standard rule picks the layer with the best training AUROC. On Mistral it picked layer 1, the surface-reading layer. Re-selecting by a generalisation criterion (held-out conditional-on-harm detection and benign false-positive rate) moved Mistral to layer 11, and detection rose from 63.6% to 90.9% on canonical-register novel tokens and from 66.7% to 100% on novel-register templates. (These are matched single-template probes at both layers; the 61% in the table is an augmented probe at layer 1.) Mistral isn’t a model that can’t generalise. It’s a model whose selection rule threw away the layer that does.

That doesn’t rescue every model. On Llama, a layer-15 probe reached cross-validated AUROC 1.0 and fired on 100% of out-of-distribution benign queries. Qwen-2.5 failed at both layer 4 and a re-selected layer 19, where detection fell from 25% to 0%. Depth looks necessary but not sufficient, and training AUROC can’t tell you which case you’re in.

Audit 4: the judge had the same blind spot

The labels the probe was evaluated against came from a harm judge that read a 5-token window, the same window the probe pooled. On a prefilled response, those five tokens are the compliance marker. So the probe fired on the marker, the judge scored the marker as harm, and the two agreed about a property neither of them had read.

Re-scoring with a 30-token judge, at the originally chosen layers and with the probe unchanged, apparent detection dropped on 5 of the 6 measurable models:

Model 5-token judge 30-token judge
Mistral-7B 87.5% 66.7%
Gemma-2-9B 87.5% 50%
Qwen-2.5-7B 62.5% 14.3%
Qwen3-8B 87.5% 71.4%
Gemma-3-12B 50% 25%
Llama-3.1-8B 75% 100%

Re-selecting the layer under the stricter judge improved detection on four of the six and made none worse (Qwen-2.5 went from 14.3% to 71.4%, Gemma-2 from 50% to 83.3%). Part of the work we’d credited to the probe was being done by the measurement window. It isn’t a cure on its own, though: Qwen-2.5’s gain on the training distribution didn’t carry over to novel tokens, where its detection stayed at 0%.

Audit 5: the weights didn’t survive a library change

Re-evaluated under PyTorch 2.4.1 and Transformers 4.44.2, the saved probe weights drifted in opposite directions: Mistral’s halt rate fell from 82% to 12%, and Llama’s rose from 82% to 92%. Retraining from the same data recovered the original numbers within ±2 points on both. The drift wasn’t random: the fragile probe was the surface-reading Mistral one. An instrument whose reading depends on its software stack isn’t measuring a stable property, so treat a saved probe as the output of a procedure and re-run the procedure in a pinned environment.

What held up

  • Deeper layers can carry real meaning. Mistral at layer 11, Gemma-2 at layer 17 and Gemma-3 at layer 4 all generalise. The signal is real on at least three of the seven models, once you stop optimising the wrong objective.
  • Diverse negatives fix false positives. Probes trained on narrow benign negatives had 80-100% out-of-distribution false-positive rates despite perfect training AUROC. Training on 150 diverse benign negatives cut that to 0-1%.
  • The marker shortcut can be trained out, at least partly. In a separate follow-up, we balanced the training set so a marker no longer predicts the label (harmful continuations with the marker stripped, benign ones with a marker added), monitored with a sliding 5-token window over the whole response, and re-selected the layer by generalisation. Out-of-distribution false positives fell from 0.83-1.00 to 0.01-0.08 on Mistral and from 0.91-0.93 to 0.00-0.10 on Qwen-2.5. We could confirm detection survived only on Mistral (conditional-on-harm 1.0 at a false-positive rate of 0.01); on the other models the attacks were refused at the output, leaving nothing to score.

None of these improvements is visible in AUROC.

The audit, as a checklist

If you train activation probes to gate, halt or audit generation, this is the audit we now run before believing one:

  1. Register-controlled holdout. Same meaning, different tokens and register, zero token overlap with training. Generalisation across surface form is the signal, not held-out accuracy.
  2. Conditional-on-harm detection. Report the fire rate among samples independently labelled harmful, not an aggregate that mixes the probe up with whether the attack worked.
  3. Discriminators. Token frequency, benign-query truncation and template length are cheap control tasks. Benign-query truncation alone would have caught our probe.
  4. Both layers. Report the layer training AUROC picks and the layer generalisation picks. When they disagree, deploy the second.
  5. Judge-window sensitivity. If probe and judge react to the same surface feature, their agreement is an artefact, not validation.
  6. Pinned regeneration. Rebuild the probe from released data in a fixed environment and report the layer as an outcome of that procedure.

None of this is new in spirit: linguistic probing worked it out years ago (see further reading). Safety probing mostly hasn’t inherited those controls, and the stakes are higher. A probe that reads a template passes its benchmark and fails silently against any attacker who changes the template.

Step 4 is a few lines with LinearProbe from rotalabs-probe (pip install "rotalabs-probe[gpu]"), given activations you’ve already extracted per layer:

from rotalabs_probe.probing import LinearProbe

def select_layer(train, holdout_harmful, ood_benign, tau=0.5, max_fpr=0.05):
    """Choose the probe layer by held-out generalisation, not training accuracy.

    train:           {layer: (X, y)} activations from your training templates
    holdout_harmful: {layer: X} judged-harmful generations from templates that
                     share no tokens with the training set
    ood_benign:      {layer: X} benign queries from outside the training distribution
    """
    report = {}
    for layer, (X, y) in train.items():
        probe = LinearProbe().fit(X, y)
        coh = (probe.predict_proba(holdout_harmful[layer])[:, 1] > tau).mean()
        fpr = (probe.predict_proba(ood_benign[layer])[:, 1] > tau).mean()
        report[layer] = {"cv_acc": probe.cv_accuracy, "coh": coh, "ood_fpr": fpr}

    by_training = max(report, key=lambda l: report[l]["cv_acc"])
    usable = [l for l in report if report[l]["ood_fpr"] <= max_fpr]
    by_holdout = max(usable, key=lambda l: report[l]["coh"]) if usable else None
    return by_training, by_holdout, report

If by_training and by_holdout disagree, you’ve found what we found. The 0.5 threshold matches our halt rule; the false-positive budget is yours to set.

What this doesn’t show

  • One probe family, one attack surface (prefilling). The full discriminator battery was run only on Mistral; the other layer-1 models rest on the bimodal-firing evidence.
  • Qwen-2.5 is unexplained. It fails at shallow and deep layers alike.
  • Small samples. 40 attacks and 50 benign prompts per model, and 10 samples in the truncation test.
  • Locatable is not robust. A meaningful signal can be found on several models; that doesn’t mean a surface-robust safety probe exists. Adaptive attackers who optimise against a specific probe remain a partly open problem.

What we’d tell our June selves

We’ve argued on this blog that you have to watch what a model computes, not just what it says (The CoT Blind Spot), and that on the raw API you should add your own probes (Opus 4.8). We still think both. But a probe is an instrument, and an instrument needs controls before its reading means anything. Ours scored 0.97-1.00, let 0 of 40 attacks through, and on four of seven models it was reading a catchphrase. Control conditions, not AUROC, decide what a probe knows.

Further Reading:

Training probes as a safety layer and want a second pair of eyes on the validity audit? Talk to us: [email protected]