README.md

---
license: cc-by-nc-sa-4.0
language:
  - ja
tags:
  - clip
  - ja
  - japanese
  - japanese-clip
pipeline_tag: feature-extraction
---

# Japanese CLIP ViT-H/14 (Wider)

## Table of Contents

1. [Overview](#overview)
1. [Usage](#usage)
1. [Model Details](#model-details)
1. [Evaluation](#evaluation)
1. [Limitations and Biases](#limitations-and-biases)
1. [Citation](#citation)
1. [See Also](#see-also)
1. [Contact Information](#contact-information)

## Overview

* **Developed by**: [HAKUHODO Technologies Inc.](https://www.hakuhodo-technologies.co.jp/)
* **Model type**: Contrastive Language-Image Pre-trained Model
* **Language(s)**: Japanese
* **LICENSE**: [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/)

Presented here is a Japanese [CLIP (Contrastive Language-Image Pre-training)](https://arxiv.org/abs/2103.00020) model,
mapping Japanese texts and images to a unified embedding space.
Capable of multimodal tasks including zero-shot image classification,
text-to-image retrieval, and image-to-text retrieval,
this model extends its utility when integrated with other components,
contributing to generative models like image-to-text and text-to-image generation.

## Usage

### Dependencies

```bash
python3 -m pip install pillow sentencepiece torch torchvision transformers
```

### Inference

The usage is similar to [`CLIPModel`](https://huggingface.co/docs/transformers/model_doc/clip)
and [`VisionTextDualEncoderModel`](https://huggingface.co/docs/transformers/model_doc/vision-text-dual-encoder).

```python
import requests
import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor, BatchEncoding

# Download
model_name = "hakuhodo-tech/japanese-clip-vit-h-14-bert-wider"
device = "cuda" if torch.cuda.is_available() else "cpu"
model = AutoModel.from_pretrained(model_name, trust_remote_code=True).to(device)
processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True)

# Prepare raw inputs
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

# Process inputs
inputs = processor(
    text=["犬", "猫", "象"],
    images=image,
    return_tensors="pt",
    padding=True,
)

# Infer and output
outputs = model(**BatchEncoding(inputs).to(device))
probs = outputs.logits_per_image.softmax(dim=1)
print([f"{x:.2f}" for x in probs.flatten().tolist()])  # ['0.00', '1.00', '0.00']
```

## Model Details

### Components

The model consists of a frozen ViT-H image encoder from
[laion/CLIP-ViT-H-14-laion2B-s32B-b79K](https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K)
and a 12-layer 24-head BERT text encoder initialized from
[hakuhodo-tech/japanese-clip-vit-h-14-bert-base](https://huggingface.co/hakuhodo-tech/japanese-clip-vit-h-14-bert-base)
with [Model Fusion](https://www.anlp.jp/proceedings/annual_meeting/2024/pdf_dir/B6-5.pdf).

### Training

Model training is done by Zhi Wang with 8 A100 (80 GB) GPUs.
[Locked-image Tuning (LiT)](https://arxiv.org/abs/2111.07991) is adopted.
See more details in [the paper](https://www.anlp.jp/proceedings/annual_meeting/2024/pdf_dir/B6-5.pdf).

### Dataset

The Japanese subset of the [laion2B-multi](https://huggingface.co/datasets/laion/laion2B-multi) dataset containing ~120M image-text pairs.

## Evaluation

### Testing Data

The 5K evaluation set (val2017) of [MS-COCO](https://cocodataset.org/)
with [STAIR Captions](http://captions.stair.center/).

### Metrics

Zero-shot image-to-text and text-to-image recall@1, 5, 10.

### Results

|                                                                                                                         |          |          |          |          |          |          |
| :---------------------------------------------------------------------------------------------------------------------- | :------: | :------: | :------: | :------: | :------: | :------: |
|                                                                                            <td colspan=3 align=center>Text Retrieval</td> <td colspan=3 align=center>Image Retrieval</td> |
|                                                                                                                         |   R@1    |   R@5    |   R@10   |   R@1    |   R@5    |   R@10   |
| [recruit-jp/japanese-clip-vit-b-32-roberta-base](https://huggingface.co/recruit-jp/japanese-clip-vit-b-32-roberta-base) |   23.0   |   46.1   |   57.4   |   16.1   |   35.4   |   46.3   |
| [rinna/japanese-cloob-vit-b-16](https://huggingface.co/rinna/japanese-cloob-vit-b-16)                                   |   37.1   |   63.7   |   74.2   |   25.1   |   48.0   |   58.8   |
| [rinna/japanese-clip-vit-b-16](https://huggingface.co/rinna/japanese-clip-vit-b-16)                                     |   36.9   |   64.3   |   74.3   |   24.8   |   48.8   |   60.0   |
| [**Japanese CLIP ViT-H/14 (Base)**](https://huggingface.co/hakuhodo-tech/japanese-clip-vit-h-14-bert-base)              |   39.2   |   66.3   |   76.6   |   28.9   |   53.3   |   63.9   |
| [**Japanese CLIP ViT-H/14 (Deeper)**](https://huggingface.co/hakuhodo-tech/japanese-clip-vit-h-14-bert-deeper)          | **48.7** |   74.0   |   82.4   |   36.5   |   61.5   |   71.8   |
| [**Japanese CLIP ViT-H/14 (Wider)**](https://huggingface.co/hakuhodo-tech/japanese-clip-vit-h-14-bert-wider)            |   47.9   | **74.2** | **83.2** | **37.3** | **62.8** | **72.7** |

\* [Japanese Stable CLIP ViT-L/16](https://huggingface.co/stabilityai/japanese-stable-clip-vit-l-16) is excluded for zero-shot retrieval evaluation as
[the model was partially pre-trained with MS-COCO](https://huggingface.co/stabilityai/japanese-stable-clip-vit-l-16#training-dataset).

## Limitations and Biases

Despite our data filtering, it is crucial
to acknowledge the possibility of the training dataset
containing offensive or inappropriate content.
Users should be mindful of the potential societal impact
and ethical considerations associated with the outputs
generated by the model when deploying in production systems.
It is recommended not to employ the model for applications
that have the potential to cause harm or distress
to individuals or groups.

## Citation

If you found this model useful, please consider citing:

```bibtex
@article{japanese-clip-vit-h,
 author = {王 直 and 細野 健人 and 石塚 湖太 and 奥田 悠太 and 川上 孝介},
 journal = {言語処理学会年次大会発表論文集},
 month = {Mar},
 pages = {1547--1552},
 title = {日本語特化の視覚と言語を組み合わせた事前学習モデルの開発 Developing Vision-Language Pre-Trained Models for {J}apanese},
 volume = {30},
 year = {2024}
}
```

## See Also

* [Japanese CLIP ViT-H/14 (Base)](https://huggingface.co/hakuhodo-tech/japanese-clip-vit-h-14-bert-base)
* [Japanese CLIP ViT-H/14 (Deeper)](https://huggingface.co/hakuhodo-tech/japanese-clip-vit-h-14-bert-deeper)

## Contact Information

Please contact
[hr-koho\@hakuhodo-technologies.co.jp](mailto:[email protected]?subject=Japanese%20CLIP%20ViT-H/14%20Models)
for questions and comments about the model,
and/or
for business and partnership inquiries.

お問い合わせは
[hr-koho\@hakuhodo-technologies.co.jp](mailto:[email protected]?subject=日本語CLIP%20ViT-H/14モデルについて)
にご連絡ください。