Upload 7 files

README.md

@@ -1,13 +1,82 @@
 ---
-title: Salesforce Codegen 350M Multi
-emoji: 🐠
-colorFrom: indigo
-colorTo: green
-sdk: gradio
-sdk_version: 3.34.0
-app_file: app.py
-pinned: false
-license: mit
+license: afl-3.0
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# PyCodeGPT
+A pre-trained GPT model for Python code completion and generation
+
+## What is it?
+
+PyCodeGPT is efficient and effective GPT-Neo-based model for python code generation task, which is similar to [OpenAI Codex](https://openai.com/blog/openai-codex/), [Github Copliot](https://copilot.github.com/), [CodeParrot](https://huggingface.co/blog/codeparrot), [AlphaCode](https://deepmind.com/blog/article/Competitive-programming-with-AlphaCode).
+
+## Training Data
+Due to the small size of public released dataset, we proposed to collect data from GitHub from scratch. We first crawled 1.2M python-related repositories hosted by GitHub. Then, we used these repository URLs to download all contents of each repository from GitHub. After that, we got 60M raw python files under 1MB with a total size of 330GB. Finally, we carefully designed various strategies of data cleaning to get about 96GB data for training. Please refer to the following table for the details.
+
+|Model|Repositories|Size and file after filtering|
+|:------:|:---:|:---:|
+| CodeParrot | 0.56M | 12GB (compressed), 5.4M |
+| Codex | 54M | 159GB |
+| PyCodeGPT | 1.2M | 96GB, 13M |
+
+
+## Pretrained models
+
+we aims to train median-large pre-trained models (model size with 110M) based on GPT-Neo:
+- PyCodeGPT-110M: derived from GPT-Neo 125M with a vocabulary size of 32K. 
+
+## GitHub
+
+[https://github.com/microsoft/PyCodeGPT](https://github.com/microsoft/PyCodeGPT)
+
+## Evaluation Results
+
+Here's our evaluation result on HumanEval dataset:
+
+Note: our model can have a comparable accuracy with Codex of similar model size.
+
+|Model|Pass@1|Pass@10|Pass@100|
+|:------:|:---:|:---:|:---:|
+|PyCodeGPT-110M                             |**8.32%**  |**13.53%** |**18.3%**  |
+|||||
+|GPT-Neo 125M                               |0.75%  |1.88%  |2.97%  |
+|GPT-Neo 1.3B                               |4.97%  |7.47%  |16.3%  |
+|GPT-Neo 2.7B                               |6.41%  |11.27% |21.37% |
+|GPT-J 6B                                   |11.62% |15.74% |27.74% |
+|||||
+|TabNine                                    |2.58%  |4.35%  |7.59%  |
+|||||
+|CodeParrot 110M                            |3.80%  |6.57%  |12.78% |
+|CodeParrot 1.5B                            |3.58%  |8.03%  |14.96% |
+|||||
+|Codex 12M                                  |2.00%  |3.62%  |8.58%  |
+|Codex 25M                                  |3.21%  |7.1%   |12.89% |
+|Codex 42M                                  |5.06%  |8.8%   |15.55% |
+|Codex 85M                                  |8.22%  |12.81% |22.4%  |
+|Codex 300M                                 |13.17% |20.37% |36.27% |
+|Codex 679M                                 |16.22% |25.7%  |40.95% |
+|Codex 2.5B                                 |21.36% |35.42% |59.5%  |
+|Codex 12B                                  |28.81% |46.81% |72.31% |
+|||||
+|Pretrained Decoder-only 13M (AlphaCode)    |1.5%   |3.6%   |8.6%   |
+|Pretrained Decoder-only 29M (AlphaCode)    |3.4%   |5.8%   |11.2%  |
+|Pretrained Decoder-only 55M (AlphaCode)    |4.2%   |8.2%   |16.9%  |
+|Pretrained Decoder-only 89M (AlphaCode)    |4.3%   |12.2%  |20.0%  |
+|Pretrained Decoder-only 302M (AlphaCode)   |11.6%  |18.8%  |31.8%  |
+|Pretrained Decoder-only 685M (AlphaCode)   |14.2%  |24.4%  |38.8%  |
+|Pretrained Decoder-only 1.1B (AlphaCode)   |17.1%  |28.2%  |45.3%  |
+|||||
+|PolyCoder 160M                             |2.13%  |3.35%  |4.88%  |
+|PolyCoder 400M                             |2.96%  |5.29%  |11.59% |
+|PolyCoder 2.7B                             |5.59%  |9.84%  |17.68% |
+
+## Reference
+If you want to use the models, you need to cite our following paper:
+
+```
+@inproceedings{CERT,
+  title={{CERT}: Continual Pre-training on Sketches for Library-oriented Code Generation},
+  author={Zan, Daoguang and Chen, Bei and Yang, Dejian and Lin, Zeqi and Kim, Minsu and Guan, Bei and Wang, Yongji and Chen, Weizhu and Lou, Jian-Guang},
+  booktitle={The 2022 International Joint Conference on Artificial Intelligence},
+  year={2022}
+}
+```

config.json

@@ -0,0 +1,54 @@
+{
+  "_name_or_path": "//amlt0a3b8c9fa72c7a7e36e6cd517fb7abe6/data/pycode_func_0214_17M_codepy-110M/model",
+  "activation_function": "gelu_new",
+  "architectures": [
+    "GPTNeoForCausalLM"
+  ],
+  "attention_dropout": 0,
+  "attention_layers": [
+    "global",
+    "local",
+    "global",
+    "local",
+    "global",
+    "local",
+    "global",
+    "local",
+    "global",
+    "local",
+    "global",
+    "local"
+  ],
+  "attention_types": [
+    [
+      [
+        "global",
+        "local"
+      ],
+      6
+    ]
+  ],
+  "bos_token_id": 1,
+  "embed_dropout": 0,
+  "eos_token_id": 0,
+  "gradient_checkpointing": false,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": null,
+  "layer_norm_epsilon": 1e-05,
+  "max_position_embeddings": 2048,
+  "model_type": "gpt_neo",
+  "num_heads": 12,
+  "num_layers": 12,
+  "resid_dropout": 0,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "torch_dtype": "float32",
+  "transformers_version": "4.12.5",
+  "use_cache": true,
+  "vocab_size": 32000,
+  "window_size": 256
+}

gitattributes.txt

@@ -0,0 +1,34 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

special_tokens_map.json

@@ -0,0 +1 @@
+{"bos_token": "<|beginoftext|>", "eos_token": "<|endoftext|>", "unk_token": "<|unkoftext|>", "pad_token": "<|padoftext|>"}