Function Vectors: Todd, Li, Sharma, Mueller, Wallace and Bau (2024), replicated and extended to 2025

Todd and coauthors found that averaging the outputs of a small set of attention heads produces a function vector that represents a task, such as giving antonyms or naming country capitals. Adding this vector to the model's activations can make it perform the task without examples in the prompt. Our extension through 2025 finds that function vectors remain effective across model families, but the best insertion point shifts deeper in Qwen models, which also require more heads to construct the vectors.

The paper's main claims

1. Averaging the outputs of the top ten or so causally important attention heads over in-context prompts yields a function vector that triggers the task zero-shot: GPT-J's mean top-1 accuracy on six tasks rises from 5.5 to 57.5 percent, and from 39.1 to 90.8 percent on prompts whose example labels have been deliberately scrambled.
2. The same recipe works in other models: GPT-NeoX-20B reaches 57.1 percent zero-shot and Llama-2-70B reaches 83.8.
3. The effect is layer-dependent: it peaks when the vector is inserted around one third of network depth and collapses in late layers.
4. About ten heads suffice, and several of them are induction heads, heads that copy patterns they have already seen in the prompt.
5. The vector is more than its vocabulary readout (a synthetic vector matching its decoded word distribution does much worse), and it keeps working in free-form natural text and across unusual prompt templates.

What we were able to do

We used GPT-J-6B and the six word-task datasets supplied with the paper's public code: antonym, capitalization, country capitals, English-to-French translation, present-to-past tense, and singular-to-plural conversion. The reported GPT-J comparison evaluates function-vector insertion at layer 9 and averages results over three seeds.

We ran the authors' function-vector pipeline, including causal mediation to identify attention heads and interventions that insert the resulting vectors. For the extension, we applied the pipeline to newer models on three tasks—antonym, country-capital, and present-past—with one seed. We also included the existing GPT-NeoX result on the antonym task.

What we got when we re-ran it

We re-ran the analysis on GPT-J and reproduced the main finding: adding a function vector substantially improves task accuracy. Mean zero-shot accuracy rises from 5.7% to 55.0%, compared with 5.5% to 57.5% in the paper; accuracy with shuffled-label prompts reaches 90.8% in both runs. Results differ more on individual tasks, especially English-to-French translation. We did not repeat the appendix visualizations, vector-composition experiments, or Llama-2 experiments, and some figure comparisons used the underlying numbers without recreating the plots.

Table 1: The paper's core intervention on GPT-J. Notes: mean top-1 accuracy in percent over the six core tasks, function vector inserted at layer 9, our runs averaged over three seeds; shuffled-label prompts carry examples with scrambled answers.

Table 2: Zero-shot function-vector accuracy per task, GPT-J at layer 9. Notes: five of six tasks land within about two standard deviations of the published value; english-french replicates far above the paper, a deviation stable across all three seeds (see Notes).

The supporting mechanics reproduced too: the three flagged induction heads reappear in our top-10 head list with prefix-matching scores of 0.545, 0.569 and 0.333 against the paper's 0.49, 0.56 and 0.31, the vocabulary-reconstruction control behaves as published, and accuracy plateaus at roughly ten heads.

The same analysis on today's models

We carried the paper's exact pipeline forward across an era ladder of open models from 2021 to 2025, spanning three families and the industry's shift from multi-head attention, where every head keeps its own memory of the context, to grouped-query attention (GQA), where heads share it to save memory. The comparison uses three of the six tasks (antonym, country-capital, present-past) under a compute cap, with the GPT-J anchor recomputed on the same three tasks.

Table 3: Zero-shot function-vector accuracy, 2021 to 2025. Notes: mean top-1 accuracy on the three-task subset at each model's best insertion layer and at the paper's canonical one-third-depth layer; GPT-NeoX is reused from the replication and covers antonym only; Qwen3-8B's mean is dragged down by a total failure on country-capital (5.1 percent, equal to baseline) while its other two tasks are healthy at 48.4 and 81.4 percent; new models run in bf16, single seed.

The survival answer is yes. In every model tested, summing the mean outputs of a handful of high-effect attention heads still produces a vector that triggers the task from a near-zero baseline, and the newer models are mostly stronger: Llama-2 at 92.2 percent and OLMo-2 at 89.3 against the GPT-J anchor's 65.9. Head localization is intact too; the top heads stay concentrated in the first two thirds of the layers everywhere (0.61 to 1.00 of them).

But the paper's rule for where to insert the vector breaks in the Qwen family. GPT-J, Llama-2 and OLMo-2 peak at 0.26 to 0.37 of depth, right where the paper says. Qwen2.5 and Qwen3 peak at 0.64 and 0.55 of depth, and inserting at the paper's one-third-depth layer collapses their accuracy to about 6 percent while the true peak stays high. And one cell fails outright: Qwen3-8B's country-capital function vector does nothing at any layer, a real failure rather than a bug (see Notes).

The updated theory

A follow-up re-analysis of the ladder asked what actually governs the effect, and the answer reshapes the paper's claims without overturning them.

• How strong the effect is depends on model family and pre-training, not on year or size. Peak accuracy sits in the 45 to 92 percent band with no trend over time (about one point per year, R² of 0.01) and a weakly negative trend with parameter count.
• Where to insert depends on the attention architecture. Across the four MHA models, three families and five years, the peak sits at 0.315 of depth give or take 0.05, confirming the paper's one-third heuristic; the GQA Qwen models sit at roughly 0.59, about twice as deep, so no single depth rule fits the whole ladder.
• How many heads you need also tracks the architecture: 8 to 11 for the MHA models, matching the paper's plateau of about ten, but 25 to 27 for the Qwen models, where the task is spread across most of the heads.
• The induction-head connection fades. The top heads are enriched for induction behavior in every model except OLMo-2, but the enrichment falls from a ratio of 4.2 in GPT-J to around 1.2 to 2.0 in the newer models, and OLMo-2 shows none at all.

In short: the function vector is a real, causal, transferable object across families and the MHA-to-GQA shift, but the paper's tidy universals, one third of depth and ten heads and induction heads, are properties of the 2021-era MHA design rather than of function vectors themselves.

Notes

These are the judgment calls and data limits behind our numbers, so you can decide how much weight each result deserves.

• An initial run stopped before completing the analysis. The results reported here come from the subsequent completed run.
• The re-run did not cover appendix figures, vector composition, or Llama-2 models. Numerical differences in the experiments we did repeat are shown in the tables above.
• The one odd replication cell is english-french at 70.3 versus the published 44.7, consistent across all three seeds; the direction favors the paper's thesis, but the gap is unexplained.
• The Qwen3-8B country-capital failure was adversarially QA-checked and confirmed real, not a code bug: a shape guard verified the head decomposition on every model, and the same model's antonym and present-past vectors work fine. Why that one task fails on that one model was not probed.
• The replication loaded GPT-J in fp16 and the extension ran new models in bf16, both disclosed budget choices; the accuracy metrics are rank-based and robust to this, but exact magnitudes could shift at the margins.
• Extension limits: three tasks not six, one seed, all base models (the planned instruct rung, Llama-3.1-8B-Instruct, was gated and skipped), and GPT-NeoX enters as a reused antonym-only rung.
• One disclosed bookkeeping wrinkle: a stored summary file records Qwen3's peak-layer fraction as 0.361 by naively averaging in the failed task's spurious layer-0 peak; the reported 0.55 excludes that task, and QA judged the choice correct.

Data and model sources for the extension studies

The extension studies pulled model weights from Hugging Face on September 9 and 10, 2026. In full:

• EleutherAI, GPT-J-6B (Hugging Face model EleutherAI/gpt-j-6b), 2021 release; all GPT-J numbers reused from our replication artifacts rather than recomputed.
• EleutherAI, GPT-NeoX-20B (EleutherAI/gpt-neox-20b), 2022 release; antonym-only results reused from the replication artifacts.
• Meta, Llama 2 7B (meta-llama/Llama-2-7b-hf), 2023 release, loaded from a local copy of the official weights.
• Alibaba Cloud, Qwen2.5-7B base model (Qwen/Qwen2.5-7B), 2024 release, downloaded from Hugging Face.
• Alibaba Cloud, Qwen3-8B base model (Qwen/Qwen3-8B-Base), 2025 release, downloaded from Hugging Face.
• Allen Institute for AI, OLMo-2-7B base model (allenai/OLMo-2-1124-7B), 2025 release, downloaded from Hugging Face.
• In-context task datasets: the six core word-task files (antonym, capitalize, country-capital, english-french, present-past, singular-plural) shipped with the paper's public replication code; the extension used the antonym, country-capital and present-past tasks.

The theory update fetched no new data or models; it works entirely from the living update's artifacts and the replication anchors on disk.