ArthurConmyGDM
commited on
Commit
•
0127b34
1
Parent(s):
2232397
add experimental embedding SAEs (#7)
Browse files- add embedding sae parameters (4cdad7faa506b5602d2b3cbb79af3d16c28b6214)
- fold in scaling by sqrt(d_model) into params (9ff4e7b3ef5f24a9f7777c1732ac0bde3523a46f)
.gitattributes
CHANGED
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
36 |
+
embedding/width_4k filter=lfs diff=lfs merge=lfs -text
|
37 |
+
embedding/width_4k/*.npz filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
@@ -16,6 +16,7 @@ See our [landing page](https://huggingface.co/google/gemma-scope) for details on
|
|
16 |
- `gemma-scope-`: See 1.
|
17 |
- `2b-pt-`: These SAEs were trained on Gemma v2 2B base model.
|
18 |
- `res`: These SAEs were trained on the model's residual stream.
|
|
|
19 |
|
20 |
|
21 |
# 3. Which SAE is in the [Neuronpedia demo](https://www.neuronpedia.org/gemma-scope)?
|
@@ -37,4 +38,4 @@ https://huggingface.co/ArthurConmyGDM
|
|
37 |
|
38 |
# Citation
|
39 |
|
40 |
-
Paper: https://arxiv.org/abs/2408.05147
|
|
|
16 |
- `gemma-scope-`: See 1.
|
17 |
- `2b-pt-`: These SAEs were trained on Gemma v2 2B base model.
|
18 |
- `res`: These SAEs were trained on the model's residual stream.
|
19 |
+
- We include experimental SAEs trained on token embeddings in the ./embedding folder.
|
20 |
|
21 |
|
22 |
# 3. Which SAE is in the [Neuronpedia demo](https://www.neuronpedia.org/gemma-scope)?
|
|
|
38 |
|
39 |
# Citation
|
40 |
|
41 |
+
Paper: https://arxiv.org/abs/2408.05147
|
embedding/width_4k/.gitattributes
ADDED
@@ -0,0 +1,4 @@
|
|
|
|
|
|
|
|
|
|
|
1 |
+
average_l0_6/params.npz filter=lfs diff=lfs merge=lfs -text
|
2 |
+
average_l0_111/params.npz filter=lfs diff=lfs merge=lfs -text
|
3 |
+
average_l0_21/params.npz filter=lfs diff=lfs merge=lfs -text
|
4 |
+
average_l0_44/params.npz filter=lfs diff=lfs merge=lfs -text
|
embedding/width_4k/average_l0_111/params.npz
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:fb17c05ab4c749d5fc2f78892241c1abfe8200e3cef2c875b47320062897cc2a
|
3 |
+
size 75540696
|
embedding/width_4k/average_l0_21/params.npz
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:dd54f7bb23d4c9b05eaea3cfb44eba76f1a009d9a3ab5d1e1f9d3b7ef1b64e79
|
3 |
+
size 75540696
|
embedding/width_4k/average_l0_44/params.npz
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:b28594f38a0be308bca2cf31f1891338de1f4e8f0e2b7d43c1f4eb8e27a22062
|
3 |
+
size 75540696
|
embedding/width_4k/average_l0_6/params.npz
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:8c663031d10179005329025fdb572a49a16d50270a8e724829bb0628ff8a02d1
|
3 |
+
size 75540696
|