Update README.md

README.md

@@ -1,199 +1,242 @@
 ---
 library_name: transformers
-tags: []
+license: bigcode-openrail-m
+datasets:
+- AlfredPros/smart-contracts-instructions
+language:
+- en
 ---
 
-# Model Card for Model ID
-
-<!-- Provide a quick summary of what the model is/does. -->
-
-
+# Model Card for Starcoder2-15B Fine-Tuned with QLORA on Solidity Dataset
 
 ## Model Details
 
 ### Model Description
 
-<!-- Provide a longer summary of what this model is. -->
-
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+This model has been fine-tuned using QLORA to generate Solidity smart contracts from natural language instructions. Training was done using the same dataset used to train [https://huggingface.co/yoniebans/starcoder2-3b-qlora-solidity](starcoder2-3b-qlora-solidity). This should allow for a direct comparison between the two models. The `starcoder2-15b` base model benefits from having being trained on a lot more code that also included solidity code for reference.
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
+- **Developed by:** @yoniebans
+- **Model type:** Transformer-based Causal Language Model
+- **Language(s) (NLP):** English with programming language syntax (Solidity)
 - **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+- **Finetuned from model** bigcode/starcoder2-15b
 
 ### Model Sources [optional]
 
-<!-- Provide the basic links for the model. -->
-
 - **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
 - **Demo [optional]:** [More Information Needed]
 
 ## Uses
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
 ### Direct Use
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
+This model was created to demonstrate how fine-tuning a base model such as starcoder2-15b using qlora can increase the model's ability to create Solidity code.
 
 ### Out-of-Scope Use
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+This model is not intended for:
 
-[More Information Needed]
+- Deployment in production systems without rigorous testing.
+- Use in non-technical text generation or any context outside smart contract development.
 
 ## Bias, Risks, and Limitations
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
+The training data consists of code from publicly sourced Solidity projects which may not encompass the full diversity of programming styles and techniques. The Solidity source code originates from mwritescode's Slither Audited Smart Contracts (https://huggingface.co/datasets/mwritescode/slither-audited-smart-contracts).
 
 ### Recommendations
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+Users are advised to use this model as a starting point for development and not as a definitive solution. Generated code should always be reviewed by experienced developers to ensure security and functionality.
 
 ## How to Get Started with the Model
 
 Use the code below to get started with the model.
 
-[More Information Needed]
+```python
+import sys, torch, accelerate
+from peft import PeftModel
+from transformers import BitsAndBytesConfig, AutoTokenizer, AutoModelForCausalLM
+
+use_4bit = True
+bnb_4bit_compute_dtype = "float32"
+bnb_4bit_quant_type = "nf4"
+use_double_nested_quant = True
+compute_dtype = getattr(torch, bnb_4bit_compute_dtype)
+
+device_map = "auto"
+max_memory = '75000MB'
+n_gpus = torch.cuda.device_count()
+max_memory = {i: max_memory for i in range(n_gpus)}
+
+checkpoint = "bigcode/starcoder2-15b"
+model = AutoModelForCausalLM.from_pretrained(
+    checkpoint,
+    cache_dir=None,
+    device_map=device_map,
+    max_memory=max_memory,
+    quantization_config=BitsAndBytesConfig(
+        load_in_4bit=use_4bit,
+        llm_int8_threshold=6.0,
+        llm_int8_has_fp16_weight=False,
+        bnb_4bit_compute_dtype=compute_dtype,
+        bnb_4bit_use_double_quant=use_double_nested_quant,
+        bnb_4bit_quant_type=bnb_4bit_quant_type
+    ),
+    torch_dtype=torch.float32,
+    trust_remote_code=False
+)
+
+tokenizer = AutoTokenizer.from_pretrained(
+        checkpoint,
+        cache_dir=None,
+        padding_side="right",
+        use_fast=False,
+        tokenizer_type=None, # Needed for HF name change
+        trust_remote_code=False,
+        use_auth_token=False,
+    )
+
+if tokenizer._pad_token is None:
+    num_new_tokens = tokenizer.add_special_tokens(dict(pad_token="[PAD]"))
+    model.resize_token_embeddings(len(tokenizer))
+    
+    if num_new_tokens > 0:
+        input_embeddings_data = model.get_input_embeddings().weight.data
+        output_embeddings_data = model.get_output_embeddings().weight.data
+
+        input_embeddings_avg = input_embeddings_data[:-num_new_tokens].mean(dim=0, keepdim=True)
+        output_embeddings_avg = output_embeddings_data[:-num_new_tokens].mean(dim=0, keepdim=True)
+
+        input_embeddings_data[-num_new_tokens:] = input_embeddings_avg
+        output_embeddings_data[-num_new_tokens:] = output_embeddings_avg
+
+local_adapter_weights = "yoniebans/starcoder2-15b-qlora-solidity"
+model = PeftModel.from_pretrained(model, local_adapter_weights)
+
+input='Make a smart contract for a memecoin named 'LLMAI', adhering to the ERC20 standard. The contract should enforce a purchase limit where no individual wallet can acquire more than 1% of the total token supply, which is set at 10 billion tokens. This purchasing limit should be modifiable and can only be disabled by the contract owner at their discretion. Note that the interfaces for ERC20, Ownable, and any other dependencies should be assumed as already imported and do not need to be included in your code response.'
+
+prompt = f"""### Instruction:
+Use the Task below and the Input given to write the Response, which is a programming code that can solve the following Task:
+
+### Task:
+{input}
+
+### Solution:
+"""
+
+input_ids = tokenizer(prompt, return_tensors="pt", truncation=True).input_ids.cuda()
+outputs = model.generate(
+    input_ids=input_ids, 
+    max_new_tokens=2048, 
+    do_sample=True, 
+    top_p=0.9, 
+    temperature=0.001, 
+    pad_token_id=1
+)
+
+output_text = tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True)[0]
+output_text_without_prompt = output_text[len(prompt):]
+
+file_path = './smart_contract.sol'
+
+with open(file_path, 'w') as file:
+    file.write(output_text_without_prompt)
+
+print(f"Output written to {file_path}")
+```
 
 ## Training Details
 
 ### Training Data
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
+The model was trained on a dataset consisting of pairs of natural language instructions and their corresponding Solidity code implementations. This dataset includes over 6000 input and outputs.
 
 ### Training Procedure
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
 #### Training Hyperparameters
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+- Number of train epochs: 6
+- Double quant: true
+- Quant type: nf4
+- Bits: 4
+- Lora r: 64
+- Lora aplha: 16
+- Lora dropout: 0.0
+- Per device train batch size: 1
+- Gradient accumulation steps: 1
+- Max steps: 2000
+- Weight decay: 0.0 #use lora dropout instead for regularization if needed
+- Learning rate: 2e-4
+- Max gradient normal: 0.3
+- Gradient checkpointing: true
+- FP16: false
+- FP16 option level: O1
+- BF16: false
+- Optimizer: paged AdamW 32-bit
+- Learning rate scheduler type: constant
+- Warmup ratio: 0.03
 
 #### Speeds, Sizes, Times [optional]
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
-
-## Evaluation
-
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
+| Epoch | Grad Norm | Loss | Step |
+|-------|-----------|------|------|
+| 0.03 | 0.299587 | 0.9556 | 10 |
+| 0.30 | 0.496698 | 0.6772 | 100 |
+| 0.59 | 0.249761 | 0.5784 | 200 |
+| 0.89 | 0.233806 | 0.6166 | 300 |
+| 1.18 | 0.141580 | 0.3541 | 400 |
+| 1.48 | 0.129458 | 0.3517 | 500 |
+| 1.78 | 0.114603 | 0.3793 | 600 |
+| 2.07 | 0.085970 | 0.3937 | 700 |
+| 2.37 | 0.085016 | 0.3209 | 800 |
+| 2.67 | 0.097650 | 0.3716 | 900 |
+| 2.96 | 0.093905 | 0.3437 | 1000 |
+| 3.26 | 0.137684 | 0.3026 | 1100 |
+| 3.55 | 0.137903 | 0.3432 | 1200 |
+| 3.85 | 0.103362 | 0.3310 | 1300 |
+| 4.15 | 0.174095 | 0.3567 | 1400 |
+| 4.44 | 0.203337 | 0.3279 | 1500 |
+| 4.74 | 0.229325 | 0.4026 | 1600 |
+| 5.04 | 0.134137 | 0.1737 | 1700 |
+| 5.33 | 0.113009 | 0.2132 | 1800 |
+| 5.63 | 0.066551 | 0.2207 | 1900 |
+| 5.92 | 0.136091 | 0.2193 | 2000 |
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/65e20784f70c00af9691d07b/OxGiUiSUMZEsdLWuYU2jl.png)
 
 ### Results
 
-[More Information Needed]
-
 #### Summary
 
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
+Initial evaluations show promising results in generating Solidity code. More work required to understand effectiveness of input promt and max_length for training and inference.
 
 ## Environmental Impact
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
+- **Hardware Type:** NVIDIA A100 • 80GB (80 GB VRAM) • 117 GB RAM • 8 vCPU
+- **Hours used:** 8
+- **Cloud Provider:** https://www.runpod.io
 - **Compute Region:** [More Information Needed]
 - **Carbon Emitted:** [More Information Needed]
 
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
 ## Citation [optional]
 
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
 **BibTeX:**
-
-[More Information Needed]
+```
+@misc{lozhkov2024starcoder,
+      title={StarCoder 2 and The Stack v2: The Next Generation}, 
+      author={Anton Lozhkov and Raymond Li and Loubna Ben Allal and Federico Cassano and Joel Lamy-Poirier and Nouamane Tazi and Ao Tang and Dmytro Pykhtar and Jiawei Liu and Yuxiang Wei and Tianyang Liu and Max Tian and Denis Kocetkov and Arthur Zucker and Younes Belkada and Zijian Wang and Qian Liu and Dmitry Abulkhanov and Indraneil Paul and Zhuang Li and Wen-Ding Li and Megan Risdal and Jia Li and Jian Zhu and Terry Yue Zhuo and Evgenii Zheltonozhskii and Nii Osae Osae Dade and Wenhao Yu and Lucas Krauß and Naman Jain and Yixuan Su and Xuanli He and Manan Dey and Edoardo Abati and Yekun Chai and Niklas Muennighoff and Xiangru Tang and Muhtasham Oblokulov and Christopher Akiki and Marc Marone and Chenghao Mou and Mayank Mishra and Alex Gu and Binyuan Hui and Tri Dao and Armel Zebaze and Olivier Dehaene and Nicolas Patry and Canwen Xu and Julian McAuley and Han Hu and Torsten Scholak and Sebastien Paquet and Jennifer Robinson and Carolyn Jane Anderson and Nicolas Chapados and Mostofa Patwary and Nima Tajbakhsh and Yacine Jernite and Carlos Muñoz Ferrandis and Lingming Zhang and Sean Hughes and Thomas Wolf and Arjun Guha and Leandro von Werra and Harm de Vries},
+      year={2024},
+      eprint={2402.19173},
+      archivePrefix={arXiv},
+      primaryClass={cs.SE}
+}
+```
 
 **APA:**
 
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
+Starcoder2-15B: BigCode Team. (2024). Starcoder2-15B [Software]. Available from https://huggingface.co/bigcode/starcoder2-15b
 
-[More Information Needed]
+AlfredPros Smart Contracts Instructions: AlfredPros. (2023). Smart Contracts Instructions [Data set]. Available from https://huggingface.co/datasets/AlfredPros/smart-contracts-instructions
 
 ## Model Card Contact
 
-[More Information Needed]
+[email protected]