init commit

README.md

@@ -1,3 +1,119 @@
 ---
-license: cc-by-3.0
+library_name: generic
+license: cc-by-sa-3.0
+pipeline_tag: image-classification
+tags:
+- ai_or_not
+- sumsub
+- image_classification
+- sumsubaiornot
+- aiornot
+- deepfake
+- synthetic
+- generated
+- pytorch
+metrics:
+- accuracy
 ---
+
+# For Fake's Sake: a set of models for detecting generated and synthetic images
+
+Many people on the internet have recently been tricked by fake images of Pope Francis wearing a coat or of Donald Trump's arrest. 
+To help combat this issue, we provide detectors for such images generated by popular tools like Midjourney and Stable Diffusion.
+
+| ![Image1](images/3.jpg) | ![Image2](images/2.jpg) | ![Image3](images/4.webp) |
+|-------------------------|-------------------------|--------------------------|
+
+## Model Details
+
+### Model Description
+
+- **Developed by:** [Sumsub AI team](https://sumsub.com/)
+- **Model type:** Image classification
+- **License:** CC-By-SA-3.0
+- **Types:** 
+- **Finetuned from model:** *convnext_large_mlp.clip_laion2b_soup_ft_in12k_in1k_384*
+
+## Demo
+
+The demo page can be found [here](https://huggingface.co/spaces/Sumsub/Sumsub-ffs-demo).
+
+## How to Get Started with the Model & Model Sources
+
+Use the code below to get started with the model:
+
+```bash
+git lfs install
+git clone https://huggingface.co/Sumsub/Sumsub-ffs-synthetic-2.0 sumsub-ffs-synthetic-v2
+```
+
+```python
+from sumsub-ffs-synthetic-v2.pipeline import PreTrainedPipeline
+from PIL import Image
+
+pipe = PreTrainedPipeline("sumsub-ffs-synthetic-v2/")
+
+img = Image.open("sumsub-ffs-synthetic-v2/images/2.jpg")
+
+result = pipe(img)
+print(result)
+```
+
+You may need these prerequsites installed:
+
+```bash
+pip install -r requirements.txt
+pip install "git+https://github.com/rwightman/pytorch-image-models"
+pip install "git+https://github.com/huggingface/huggingface_hub"
+```
+
+## Training Details
+
+### Training Data
+
+The models were trained on the following datasets:
+
+TBD
+
+### Training Procedure
+
+TBD
+
+## Evaluation
+
+TBD
+
+## Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+TBD
+
+## Limitations
+
+- It should be noted that achieving 100% accuracy is not possible. Therefore, the model output should only be used as an indication that an image may have been (but not definitely) artificially generated.
+- Our models may face challenges in accurately predicting the class for real-world examples that are extremely vibrant and of exceptionally high quality. In such cases, the richness of colors and fine details may lead to misclassifications due to the complexity of the input. This could potentially cause the model to focus on visual aspects that are not necessarily indicative of the true class.
+
+![Image1](images/1.jpg)
+
+## Citation
+
+If you find this useful, please cite as:
+
+```text
+@misc{sumsubaiornot, 
+    publisher = {Sumsub},
+    url       = {https://huggingface.co/Sumsub/Sumsub-ffs-synthetic-2.0},
+    year      = {2023},
+    author    = {Savelyev, Alexander and Toropov, Alexey and Goldman-Kalaydin, Pavel and Samarin, Alexey},
+    title     = {For Fake's Sake: a set of models for detecting deepfakes, generated images and synthetic images}
+}
+```
+
+## References
+
+- Stöckl, Andreas. (2022). Evaluating a Synthetic Image Dataset Generated with Stable Diffusion. 10.48550/arXiv.2211.01777.
+- Lin, Tsung-Yi & Maire, Michael & Belongie, Serge & Hays, James & Perona, Pietro & Ramanan, Deva & Dollár, Piotr & Zitnick, C.. (2014). Microsoft COCO: Common Objects in Context.
+- Howard, Andrew & Zhu, Menglong & Chen, Bo & Kalenichenko, Dmitry & Wang, Weijun & Weyand, Tobias & Andreetto, Marco & Adam, Hartwig. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications.
+- Liu, Zhuang & Mao, Hanzi & Wu, Chao-Yuan & Feichtenhofer, Christoph & Darrell, Trevor & Xie, Saining. (2022). A ConvNet for the 2020s.
+- Wang, Zijie & Montoya, Evan & Munechika, David & Yang, Haoyang & Hoover, Benjamin & Chau, Polo. (2022). DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models. 10.48550/arXiv.2210.14896.

config.json

@@ -0,0 +1,12 @@
+{
+  "id2label": {
+    "0": "by AI",
+    "1": "by human"
+  },
+  "label2id": {
+    "by AI": "0",
+    "by human": "1"
+  },
+  "pretrained": false,
+  "timm_model": "convnext_large_mlp.clip_laion2b_soup_ft_in12k_in1k_384"
+}

pipeline.py

@@ -0,0 +1,53 @@
+from typing import Dict, List, Any
+from PIL import Image
+
+import os
+import json
+import torch
+from torch.nn import functional as F
+import timm
+
+from synthetic import SyntheticModel
+
+class PreTrainedPipeline():
+    def __init__(self, path=""):
+        self.model = SyntheticModel()
+        ckpt = torch.load(os.path.join(path, "synthetic.pt"), map_location=torch.device('cpu'))
+        self.model.load_state_dict(ckpt)
+        self.model.eval()
+
+        with open(os.path.join(path, "config.json")) as config:
+            config = json.load(config)
+        self.id2label = config["id2label"]
+
+        transform_config = {'input_size': (3, 384, 384),
+                        'interpolation': 'bicubic',
+                        'mean': (0.48145466, 0.4578275, 0.40821073),
+                        'std': (0.26862954, 0.26130258, 0.27577711),
+                        'crop_pct': 1.0,
+                        'crop_mode': 'squash'}
+
+        self.transform_m = timm.data.create_transform(**transform_config, is_training=False)
+
+    def __call__(self, inputs: "Image.Image") -> List[Dict[str, Any]]:
+        """
+        Args:
+            inputs (:obj:`PIL.Image`):
+                The raw image representation as PIL.
+                No transformation made whatsoever from the input. Make all necessary transformations here.
+        Return:
+            A :obj:`list`:. The list contains items that are dicts should be liked {"label": "XXX", "score": 0.82}
+                It is preferred if the returned list is in decreasing `score` order
+        """
+        img = self.transform_m(inputs)
+        return self.predict_from_model(img)
+
+    def predict_from_model(self, img):
+        y = self.model.forward(img.unsqueeze(0).to('cpu'))
+        y_1 = F.softmax(y, dim=1)[:, 1].cpu().detach().numpy()
+        y_2 = F.softmax(y, dim=1)[:, 0].cpu().detach().numpy()
+        labels = [
+            {"label": str(self.id2label["0"]), "score": y_1.tolist()[0]},
+            {"label": str(self.id2label["1"]), "score": y_2.tolist()[0]},
+        ]
+        return labels

requirements.txt

@@ -0,0 +1,4 @@
+timm==0.9.5
+torch==2.0.1
+torchvision==0.15.2
+pytorch-lightning==2.0.9

synthetic.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:89a955ec54bddab759228757e437d300b6b86bbba9f45cfd5ecd0e3d7dec83a2
+size 795263437

synthetic.py

@@ -0,0 +1,22 @@
+import timm
+from torch import nn
+from torch.nn import functional as F
+import pytorch_lightning as pl
+from pytorch_lightning.core.mixins import HyperparametersMixin
+
+
+class SyntheticModel(pl.LightningModule, HyperparametersMixin):
+    def __init__(self):
+        super().__init__()
+        self.model = timm.create_model('convnext_large_mlp.clip_laion2b_soup_ft_in12k_in1k_384',
+                                       pretrained=False,
+                                       num_classes=0)
+
+        self.clf = nn.Sequential(
+            nn.Linear(1536, 128),
+            nn.ReLU(inplace=True),
+            nn.Linear(128, 2))
+
+    def forward(self, image):
+        image_features = self.model(image)
+        return self.clf(image_features)