Katherine commited on
Commit
0a96f74
1 Parent(s): 2441500

Add information to README.md

Browse files
Files changed (1) hide show
  1. README.md +4 -4
README.md CHANGED
@@ -23,10 +23,10 @@ pipe = StableDiffusionPipeline.from_pretrained(model, vae=vae)
23
  ```
24
 
25
  ## Decoder Finetuning
26
- We publish two kl-f8 autoencoder versions, finetuned from the original [kl-f8 autoencoder](https://github.com/CompVis/latent-diffusion#pretrained-autoencoding-models).
27
- The first, _ft-EMA_, was resumed from the original checkpoint, trained for 313198 steps and uses EMA weights.
28
- The second, _ft-MSE_, was resumed from _ft-EMA_ and uses EMA weights and was trained for another 280k steps using a re-weighted loss, with more emphasis
29
- on MSE reconstruction (producing somewhat ``smoother'' outputs).
30
  To keep compatibility with existing models, only the decoder part was finetuned; the checkpoints can be used as a drop-in replacement for the existing autoencoder.
31
 
32
  _Original kl-f8 VAE vs f8-ft-EMA vs f8-ft-MSE_
 
23
  ```
24
 
25
  ## Decoder Finetuning
26
+ We publish two kl-f8 autoencoder versions, finetuned from the original [kl-f8 autoencoder](https://github.com/CompVis/latent-diffusion#pretrained-autoencoding-models) on a 1:1 ratio of [LAION-Aesthetics](https://laion.ai/blog/laion-aesthetics/) and LAION-Humans, an unreleased subset containing only SFW images of humans. The intent was to fine-tune on the Stable Diffusion training set (the autoencoder was originally trained on OpenImages) but also enrich the dataset with images of humans to improve the reconstruction of faces.
27
+ The first, _ft-EMA_, was resumed from the original checkpoint, trained for 313198 steps and uses EMA weights. It uses the same loss configuration as the original checkpoint (L1 + LPIPS).
28
+ The second, _ft-MSE_, was resumed from _ft-EMA_ and uses EMA weights and was trained for another 280k steps using a different loss, with more emphasis
29
+ on MSE reconstruction (MSE + 0.1 * LPIPS). It produces somewhat ``smoother'' outputs. The batch size for both versions was 192 (16 A100s, batch size 12 per GPU).
30
  To keep compatibility with existing models, only the decoder part was finetuned; the checkpoints can be used as a drop-in replacement for the existing autoencoder.
31
 
32
  _Original kl-f8 VAE vs f8-ft-EMA vs f8-ft-MSE_