Synthetic Model for SAE Training

This repository contains a SyntheticModel for use with SAELens.

This is v2 of the SynthSAEBench-16k model. It is identical to v1 (same feature vectors, firing probabilities, correlation matrix, and hierarchy structure), except with scale_children_by_parent=True in the hierarchy, as described in the SynthSAEBench paper. v1 was mistakenly uploaded with scale_children_by_parent=False.

Model Info

Number of features: 16,384
Hidden dimension: 768
Hierarchy: Yes
- Root nodes: 128
- Total nodes: 10,880
- Max depth: 3
Feature correlation: Yes (scale 0.1)

Usage

from sae_lens.synthetic import SyntheticModel

model = SyntheticModel.from_pretrained("decoderesearch/synth-sae-bench-16k-v2")

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support