Activation Patching, causal claims from swapping activations
How to turn "this component seems important" into a causal claim: run a model twice, swap one activation between the runs, and measure what changes. The operation, its two directions, the choices that change the answer, and the ways a clean-looking result can still mislead.
Looking at which components of a model are active when it does something tells you what correlates with the behaviour. Activation patching asks the causal question instead: if this one activation had been different, would the output have changed? It is a core experiment behind many circuit-level claims in mechanistic interpretability.
This is a reading note. It organises published work, and it reports no experiments of the studio's own. The core reference is How to use and interpret activation patching (Heimersheim and Nanda, 2024), a practical guide to the method's variants and pitfalls.
- Activation
- An intermediate value the model computes while reading a prompt: the output of one layer, head or MLP at one token position.
- Clean / corrupted
- Two prompts that differ minimally. The clean one produces the behaviour under study; the corrupted one does not.
- Logit difference
- The model's score for the correct token minus its score for a specific wrong one.
- Foil
- The specific wrong token the correct one is compared against.
- Residual stream
- The running vector each layer reads from and adds to; the model's main internal pathway.
- Ablation
- Removing a component's contribution by overwriting its activation with a stand-in value.
- Circuit
- A set of components that together implement a behaviour.
Two runs, one swap
Take a clean prompt that produces the behaviour and a corrupted prompt that differs minimally and does not. Cache the activations from both runs. Then re-run one of them while overwriting a single activation, at one site, one token position and one layer, with the value cached from the other run. Measure the change in a chosen metric.
Because only that one value differs, any change in the metric is attributable to it: this activation, at this site, on this input, causally mediates the difference in output. Everything else in the method is a choice about which direction to patch, what to overwrite with, and how to score the result.
Denoising and noising answer different questions
Denoising patches a clean activation into the corrupted run and "shows whether the patched activations were sufficient to restore the model behaviour". Noising patches a corrupted activation into the clean run and shows whether it was "necessary to maintain" it. The two "are usually not complements of each other, nor does one imply the other".
The consequence follows from how a circuit combines its parts. If the circuit is an OR, any one component is enough, so denoising finds every component in the first pass while noising finds none, because the others cover for the one you removed. If it is an AND, every component is required, so noising finds them all while denoising finds none. The same mechanism gives opposite pictures depending on the direction you ran. Pick a structure and a direction below, then patch each component.
Each cell shows what patching that component alone does. Select several cells to patch them together.
Neither direction is ground truth on its own, which is why the guide recommends reporting which one was run and reading the result as sufficiency or necessity accordingly. An experiment that quietly mixes the two assumes a symmetry that usually is not there.
Localised is not the same as editable
Causal tracing is the canonical denoising experiment. Meng et al. (2022) corrupt the subject tokens of a factual prompt with noise, then restore one clean hidden state at a time and track how much of the correct answer returns. It pointed to mid-layer feed-forward modules at the last subject token, and motivated an editing method that rewrites those modules.
The next question was whether the localisation tells you where to edit. Hase et al. (2023) found that the localisation results "do not provide any insight into which model MLP layer would be best to edit" to replace a fact, and that even for the one editing method where the two did relate, the choice of layer was a far better predictor of editing performance. Sufficiency to restore a behaviour under one corruption is a different causal quantity from effectiveness as an intervention point, so a patching map should not be read as a map of where information is stored.
From components to paths, and to automation
A single-site patch measures a component's total effect on the output. Path patching isolates one edge, from component A to component B, separating direct effects on the metric from effects mediated through later components. It is the machinery used to decompose the indirect object identification circuit in GPT-2 small into groups of attention heads with distinct roles (Wang et al. 2022).
Doing this by hand does not scale, so the same idea is automated. ACDC prunes the edges whose removal least changes the metric (Conmy et al. 2023), at the cost of a forward pass per candidate edge. Attribution patching replaces the real patch with a first-order approximation computed from two forward passes and one backward pass. That is far cheaper, but it is a linearisation, weakest for large activations such as a whole residual-stream vector, so it works best as a filter whose top candidates are then confirmed with real patches.
What you overwrite with, and how you score it
Two choices that look like settings change the conclusions. The first is what to overwrite with. Zero ablation hands the rest of the network a value it never saw in training, which can make a component look important simply because it was removed in an unnatural way. Mean ablation, which overwrites with the dataset mean, is "slightly more principled than zero ablating since there is no special meaning to activations being zero".
The second is the metric. The guide trusts logit difference most: it is "a mostly linear function of the residual stream (unlike probability-based metrics)". Probabilities track the logits exponentially, so they saturate once the answer is already ranked first, and a patch that clearly moves the model's confidence can barely move the probability. Slide the baseline below to see one fixed patch scored both ways.
Open questions
- QHow do you tell redundancy from irrelevance? A null from noising is ambiguous when other components can take over, and the evidence that distinguishes the two is usually another experiment, not the patch itself.
- QWhat is the right corruption? Different corrupted prompts, or noise instead of a prompt swap, ask different questions, and the field has no single default that is both on-distribution and minimal.
- QWhat changes when the unit being patched is a learned feature rather than a component? Clamping a feature of a sparse autoencoder is an intervention of the same family, but its meaning depends on how faithfully the dictionary reflects the model.
- QWhat would count as a formal test of a circuit hypothesis, beyond showing that individual patches move the metric? Single-site patches are necessary evidence, not a proof that the described mechanism is the one the model uses.
When a clean result is still wrong
Three traps recur. Self-repair: ablating one attention layer can cause another to compensate, the Hydra effect (McGrath et al. 2023), so a small effect from noising can mean redundancy rather than unimportance. Localisation read as an edit target: the Hase et al. result above. Metric and corruption as hidden hyperparameters: the choices in Figure 02, plus which corrupted prompt was used, can reorder components, so a ranking that survives only one choice should not be trusted.
In short: a patching result is evidence about one swap under one setup. The claim it supports is only as wide as the direction, the ablation, the metric and the corruption that produced it.