Ukeiyo-style Diffusion

This is the fine-tuned Stable Diffusion model trained on traditional Japanese Ukeiyo-style images. Use the tokens ukeiyoddim style in your prompts for the effect. The model repo also contains a ckpt file , so that you can use the model with your own implementation of stable diffusion.

🧨 Diffusers

This model can be used just like any other Stable Diffusion model. For more information, please have a look at the Stable Diffusion.

You can also export the model to ONNX, MPS and/or FLAX/JAX.

#!pip install diffusers transformers scipy torch
from diffusers import StableDiffusionPipeline
import torch
model_id = "salmonhumorous/ukeiyo-style-diffusion"
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)
pipe = pipe.to("cuda")
prompt = "illustration of ukeiyoddim style landscape"
image = pipe(prompt).images[0]
image.save("./ukeiyo_landscape.png")

Training procedure and data

The training for this model was done using a RTX 3090. The training was completed in 28 minutes for a total of 2000 steps. A total of 33 instance images (Images of the style I was aiming for) and 1k Regularization images was used. Regularization images dataset used by ProGamerGov.

Training notebook used by Shivam Shrirao.

Training hyperparameters

The following hyperparameters were used during training:

number of steps : 2000
learning_rate: 1e-6
train_batch_size: 1
scheduler_type: DDIM
number of instance images : 33
number of regularization images : 1000
lr_scheduler : constant
gradient_checkpointing

Results

Below are the sample results for different training steps :

Sample images by model trained for 2000 steps :

prompt = "landscape" prompt = "ukeiyoddim style landscape" prompt = " illustration of ukeiyoddim style landscape"

Acknowledgement

Many thanks to nitrosocke, for inspiration and for the guide. Also thanks, to all the amazing people making stable diffusion easily accessible for everyone.

salmonhumorous
/

ukeiyo-style-diffusion