Persona Vectors: Chen et al. (2025), replicated and extended through 2025

Chen and coauthors identified directions in a chat model's activations associated with traits such as sycophancy and hallucination. These persona vectors can change the model's behaviour, predict traits in its responses, and identify training examples likely to encourage those traits. Our extension through 2025 finds that steering and training-data screening remain effective across five models. Response prediction is much weaker on the reasoning-distilled DeepSeek-R1 model, suggesting that this use of persona vectors may depend on how a model is trained to respond.

The paper's main claims

1. Steering along a persona vector drives the trait from near zero to roughly 80 to 100 on a 0 to 100 judge scale, rising steadily with the steering coefficient.
2. The projection of the last prompt token predicts the trait score of the response that follows, at correlations of 0.75 to 0.83.
3. How far finetuning shifts the model along the vector predicts how strongly the finetuned model expresses the trait.
4. A projection difference computed on a training set, before any finetuning, predicts which datasets will induce the trait.
5. Individual trait-inducing training samples separate from normal ones by their projection alone.

What we were able to do

We used the authors' code and trait-definition and evaluation prompts for evil, sycophantic, and hallucinating behaviour, starting with Qwen2.5-7B-Instruct. We ran vector extraction, steering, response monitoring, and training-data screening. The prompt sets were then reused across the five-model extension.

We patched the projection utility to use finetuned model weights. The initial run used a substitute open-model judge; every response reported here was later evaluated with the paper's gpt-4.1-mini judge to make the comparisons consistent. For the reasoning model, we generated monitoring responses first and evaluated them afterwards.

What we got when we re-ran it

Our re-run on Qwen2.5-7B-Instruct reproduces the main pattern: persona vectors can steer traits, predict traits in subsequent responses, and identify trait-inducing training examples. The extraction, steering, monitoring, and data-screening steps all ran end to end. The initial run used a substitute model to judge responses; we later evaluated every response with the paper's gpt-4.1-mini judge so the values below are comparable. We also patched a projection utility that ignored finetuned model weights. Some figure comparisons used numerical results without recreating the plots, and we did not repeat analyses requiring human judgments or external data unavailable to this run.

Table 1: The paper's core numbers on Qwen2.5-7B-Instruct. Notes: the published "80–100" and "0.75–0.83" are ranges the paper reports across traits and models, not per-trait values. Our trait measurements are shown after evaluating every response with the paper's own judge, gpt-4.1-mini, so both rows are on the same instrument; the monitoring correlation is a Pearson correlation between the last-prompt-token projection and the judged trait score of the response.

The same analysis on today's models

We carried the design forward across five chat models spanning 2023 to 2025, each run through the same pipeline and every response, old and new, scored by the same gpt-4.1-mini judge the paper used. The short answer: the paper survives, with one amendment. Steering works on every model, pushing traits to 72 to 100 out of 100. Training samples separate almost perfectly everywhere, with AUC of at least 0.996, where AUC is the chance a randomly chosen trait-inducing sample projects higher than a normal one, so 1.0 is perfect separation. Projection monitoring stays strong on every standard chat model, at correlations of 0.65 to 0.99, but collapses to 0.25 to 0.48 on the reasoning-distilled DeepSeek-R1 model, whose long thinking trace sits between the prompt and the final answer and decouples the two.

Table 2: The paper's three mechanisms on the 2023 to 2025 ladder. Notes: each cell reads evil / sycophantic / hallucinating; the steered score is the trait score at the largest steering coefficient at that model's selected layer; the AUC column is the lowest of the three traits' values; every score comes from the paper's own gpt-4.1-mini judge; the Qwen2.5 row is reused verbatim from the replication artifacts.

A follow-up theory study then asked what governs the strength of these effects, working only from the ladder's artifacts. Era is not the answer. The selected steering layer sits at about 0.55 of network depth on every model, from 0.44 to 0.71 with no trend across generations (slope 0.005 per era, p = 0.79), and the steering coefficient needed for a half-maximal effect is likewise stable at about 1.2. What moves monitoring is alignment style: grouping the models as ordinary RLHF, newer RLHF, and reasoning-distilled, a standard test of whether group means differ (ANOVA) gives F = 27.7 with p < 0.001, with group mean correlations of 0.92, 0.84, and 0.39. Depth of the chosen layer is the one continuous predictor with any grip on both steering and monitoring; model size and release year have none.

Table 3: What predicts each mechanism's strength, over 15 model-and-trait points. Notes: R² is the share of the variation the predictor explains on its own; all five models are 7 to 8 billion parameters, so the size row has almost nothing to work with; treat every row as descriptive, since there are only five independent models.

Finally, the directions themselves travel within a family but not across families. Between Llama-3.1 and its own distillation DeepSeek-R1, the persona vectors align at a mean cosine of 0.478, rising to 0.73 at matched layers, while every pair of independently trained families sits at 0.002, about 200 times weaker. The updated claim, then: persona vectors are not a quirk of one 2024 model but a stable feature of chat models through 2025, living at a fixed relative depth, portable within an architecture family, with monitoring reliability set by how the model was aligned rather than by when it was built, and reasoning-style training is the one recipe that breaks the monitor.

Notes

These are the judgment calls and data limits behind our numbers, so you can decide how much weight each result deserves.

• The first run of the five-model extension hit an 11-hour session timeout partway through. The continuation reused the four completed models' outputs verbatim, finished only the missing cheap stages for DeepSeek-R1, and ran the one missing model, Qwen3-8B. Nothing was silently redone; an independent QA pass recomputed the reported numbers from disk and they reproduce.
• Reasoning models hang the monitoring pipeline's judge call, so for them the monitor responses were generated with the judge disabled and scored afterwards. Same instrument, different order.
• The replication itself ran with a substitute open-model judge because no OpenAI key was available; every number on this page was later re-scored with the paper's own gpt-4.1-mini judge so old and new are comparable. The substitution also exposed a portability bug: the released scoring code assumes the judge's tokenizer keeps a number in one token, and silently corrupts scores otherwise.
• QA verified the DeepSeek monitoring collapse is a genuine correlation break, not a parsing artifact: the judge scored nearly all reasoning responses successfully and the scores have healthy spread. But it is one reasoning model, measured only at its steering-selected layer, so "reasoning models break monitoring" is suggestive rather than established, and whether another layer restores monitoring there is unmeasured.
• The 200-fold within-family alignment rests on a single related pair, Llama-3.1 and its own distillation. The near-zero cross-family cosines reflect that independently trained models do not share a coordinate basis, not that their trait concepts differ, and no live cross-model steering transfer was run (the theory study had no GPU).
• The regressions in Table 3 summarize five models, so era and alignment style are partly confounded, and model size is effectively untested since all five are 7 to 8 billion parameters.
• The paper's finetuning-dependent claims were not re-run on the new models (each would need a 24-model finetuning suite); the replication's Qwen2.5 values remain the only anchor for those. DeepSeek-R1's elevated baseline hallucination score of 72.2 reflects its thinking-style output being judged hallucinatory.

Data and model sources for the extension studies

The extension studies ran on September 9 and 10, 2026, loading each model from a local snapshot of its Hugging Face release (snapshots verified against the Hugging Face API on September 9, 2026). In full:

• Mistral AI, Mistral-7B-Instruct-v0.2 (Hugging Face model mistralai/Mistral-7B-Instruct-v0.2), released December 2023.
• Meta, Llama-3.1-8B-Instruct (Hugging Face model meta-llama/Llama-3.1-8B-Instruct), released July 2024.
• Alibaba Qwen team, Qwen2.5-7B-Instruct (Hugging Face model Qwen/Qwen2.5-7B-Instruct), released September 2024; the paper's main model, reused from the replication with no new compute.
• DeepSeek, DeepSeek-R1-Distill-Llama-8B (Hugging Face model deepseek-ai/DeepSeek-R1-Distill-Llama-8B), released January 2025.
• Alibaba Qwen team, Qwen3-8B (Hugging Face model Qwen/Qwen3-8B), released April 2025, run in its non-thinking chat mode from a local snapshot labelled Qwen3-8B-Nonthinking.
• OpenAI, gpt-4.1-mini (gpt-4.1-mini-2025-04-14), accessed via the OpenRouter API as the trait judge, with the paper's exact settings (temperature 0, seed 0, single-token log-probability scoring); the same judge scored old and new responses.
• The paper's trait definition and evaluation prompt sets for evil, sycophantic, and hallucinating (20 questions and 5 positive and negative instruction pairs per trait, with the judge templates), used unchanged on every model.

The theory update fetched no new data; it works entirely from the living update's artifacts and the persona vectors already on disk.