Relatedly, increasing the amount of sparse autoencoder data related to a particular topic can be used to increase the resolution of features related to that topic. https://
transformer-circuits.pub/2024/september
-update/index.html#oversampling
…
@anthropicai
-
Sparse Autoencoders Resolution Enhancement Through Data Oversampling
By
–
-
Decomposing Successor Heads Using Independent Component Analysis
By
–
We replicated Gould et al.'s “successor heads” paper, and found that independent component analysis can further decompose these heads into a "successor" component, induction component, and category components. https://
transformer-circuits.pub/2024/september
-update/index.html#successor-heads
… -
Dictionary Learning Features for Dangerous Behavior Detection
By
–
Can dictionary learning features be used to detect dangerous behavior? It turns out that they're competitive with (and sometimes beat) linear probes, but can discover spurious correlations that might make a classifier vulnerable to adversarial attacks. https://
transformer-circuits.pub/2024/features-
as-classifiers/index.html
… -
Crosscoders: Finding Shared Features Across Model Layers
By
–
Crosscoders (published today: https://
transformer-circuits.pub/2024/crosscode
rs/index.html
…) are a new method allowing us to find features shared across different layers in a model, or even across different models. Identifying the same feature when it persists across layers can simplify our understanding of models. -
Crosscoders as Model Comparison Tools: Experimental Interpretability Approach
By
–
One neat thing is that, very experimentally, crosscoders can be used to "diff" models: comparing between, say, a pretrained model and a fine-tuned one, to see how they differ at a more basic level.
-
Interpretability Team Shares Recent Research Updates
By
–
Over the past few months, our Interpretability team has put out a number of smaller research updates. Here’s a thread of some of the things we've been up to:
-
Research on AI Steering Methods for Safer Model Outputs
By
–
We hope our preliminary findings inspire further research on steering methods for safer, more reliable model outputs. Read the full research blog for detailed results, insights, and limitations:
-
Feature Steering vs Prompt Engineering: Unexpected Off-Target Effects
By
–
We also compared feature steering with prompt engineering and found some surprising similarities: for example, prompt engineering also showed unexpected off-target effects.
-
Feature Steering Reduces AI Bias Across Nine Social Dimensions
By
–
Finally, we discovered a feature that significantly reduces bias scores across nine social dimensions within the sweet spot. This did come with a slight capability drop, which highlights potential trade-offs in feature steering.
-
AI Gender Bias Mitigation Unexpectedly Increases Age Discrimination
By
–
However, we also observed unexpected "off-target effects". For example, to our surprise, we found that dialing up the "Gender bias awareness" feature also increased age bias.