Model Card for Starcoder2-15B Fine-Tuned with QLORA on Solidity Dataset

Model Details

Model Description

This model has been fine-tuned using QLORA to generate Solidity smart contracts from natural language instructions. Training was done using the same dataset used to train https://huggingface.co/yoniebans/starcoder2-3b-qlora-solidity. This should allow for a direct comparison between the two models. The starcoder2-15b base model benefits from having being trained on a lot more code that also included solidity code for reference.

Developed by: @yoniebans
Model type: Transformer-based Causal Language Model
Language(s) (NLP): English with programming language syntax (Solidity)
License: [More Information Needed]
Finetuned from model bigcode/starcoder2-15b

Model Sources [optional]

Repository: [More Information Needed]
Demo [optional]: [More Information Needed]

Uses

Direct Use

This model was created to demonstrate how fine-tuning a base model such as starcoder2-15b using qlora can increase the model's ability to create Solidity code.

Out-of-Scope Use

This model is not intended for:

Deployment in production systems without rigorous testing.
Use in non-technical text generation or any context outside smart contract development.

Bias, Risks, and Limitations

The training data consists of code from publicly sourced Solidity projects which may not encompass the full diversity of programming styles and techniques. The Solidity source code originates from mwritescode's Slither Audited Smart Contracts (https://huggingface.co/datasets/mwritescode/slither-audited-smart-contracts).

Recommendations

Users are advised to use this model as a starting point for development and not as a definitive solution. Generated code should always be reviewed by experienced developers to ensure security and functionality.

How to Get Started with the Model

Use the code below to get started with the model.

import sys, torch, accelerate
from peft import PeftModel
from transformers import BitsAndBytesConfig, AutoTokenizer, AutoModelForCausalLM

use_4bit = True
bnb_4bit_compute_dtype = "float32"
bnb_4bit_quant_type = "nf4"
use_double_nested_quant = True
compute_dtype = getattr(torch, bnb_4bit_compute_dtype)

device_map = "auto"
max_memory = '75000MB'
n_gpus = torch.cuda.device_count()
max_memory = {i: max_memory for i in range(n_gpus)}

checkpoint = "bigcode/starcoder2-15b"
model = AutoModelForCausalLM.from_pretrained(
    checkpoint,
    cache_dir=None,
    device_map=device_map,
    max_memory=max_memory,
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=use_4bit,
        llm_int8_threshold=6.0,
        llm_int8_has_fp16_weight=False,
        bnb_4bit_compute_dtype=compute_dtype,
        bnb_4bit_use_double_quant=use_double_nested_quant,
        bnb_4bit_quant_type=bnb_4bit_quant_type
    ),
    torch_dtype=torch.float32,
    trust_remote_code=False
)

tokenizer = AutoTokenizer.from_pretrained(
        checkpoint,
        cache_dir=None,
        padding_side="right",
        use_fast=False,
        tokenizer_type=None, # Needed for HF name change
        trust_remote_code=False,
        use_auth_token=False,
    )

if tokenizer._pad_token is None:
    num_new_tokens = tokenizer.add_special_tokens(dict(pad_token="[PAD]"))
    model.resize_token_embeddings(len(tokenizer))
    
    if num_new_tokens > 0:
        input_embeddings_data = model.get_input_embeddings().weight.data
        output_embeddings_data = model.get_output_embeddings().weight.data

        input_embeddings_avg = input_embeddings_data[:-num_new_tokens].mean(dim=0, keepdim=True)
        output_embeddings_avg = output_embeddings_data[:-num_new_tokens].mean(dim=0, keepdim=True)

        input_embeddings_data[-num_new_tokens:] = input_embeddings_avg
        output_embeddings_data[-num_new_tokens:] = output_embeddings_avg

local_adapter_weights = "yoniebans/starcoder2-15b-qlora-solidity"
model = PeftModel.from_pretrained(model, local_adapter_weights)

input='Make a smart contract for a memecoin named 'LLMAI', adhering to the ERC20 standard. The contract should enforce a purchase limit where no individual wallet can acquire more than 1% of the total token supply, which is set at 10 billion tokens. This purchasing limit should be modifiable and can only be disabled by the contract owner at their discretion. Note that the interfaces for ERC20, Ownable, and any other dependencies should be assumed as already imported and do not need to be included in your code response.'

prompt = f"""### Instruction:
Use the Task below and the Input given to write the Response, which is a programming code that can solve the following Task:

### Task:
{input}

### Solution:
"""

input_ids = tokenizer(prompt, return_tensors="pt", truncation=True).input_ids.cuda()
outputs = model.generate(
    input_ids=input_ids, 
    max_new_tokens=2048, 
    do_sample=True, 
    top_p=0.9, 
    temperature=0.001, 
    pad_token_id=1
)

output_text = tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True)[0]
output_text_without_prompt = output_text[len(prompt):]

file_path = './smart_contract.sol'

with open(file_path, 'w') as file:
    file.write(output_text_without_prompt)

print(f"Output written to {file_path}")

Training Details

Training Data

The model was trained on a dataset consisting of pairs of natural language instructions and their corresponding Solidity code implementations. This dataset includes over 6000 input and outputs.

Training Procedure

Training Hyperparameters

Number of train epochs: 6
Double quant: true
Quant type: nf4
Bits: 4
Lora r: 64
Lora aplha: 16
Lora dropout: 0.0
Per device train batch size: 1
Gradient accumulation steps: 1
Max steps: 2000
Weight decay: 0.0 #use lora dropout instead for regularization if needed
Learning rate: 2e-4
Max gradient normal: 0.3
Gradient checkpointing: true
FP16: false
FP16 option level: O1
BF16: false
Optimizer: paged AdamW 32-bit
Learning rate scheduler type: constant
Warmup ratio: 0.03

Speeds, Sizes, Times [optional]

Epoch	Grad Norm	Loss	Step
0.03	0.299587	0.9556	10
0.30	0.496698	0.6772	100
0.59	0.249761	0.5784	200
0.89	0.233806	0.6166	300
1.18	0.141580	0.3541	400
1.48	0.129458	0.3517	500
1.78	0.114603	0.3793	600
2.07	0.085970	0.3937	700
2.37	0.085016	0.3209	800
2.67	0.097650	0.3716	900
2.96	0.093905	0.3437	1000
3.26	0.137684	0.3026	1100
3.55	0.137903	0.3432	1200
3.85	0.103362	0.3310	1300
4.15	0.174095	0.3567	1400
4.44	0.203337	0.3279	1500
4.74	0.229325	0.4026	1600
5.04	0.134137	0.1737	1700
5.33	0.113009	0.2132	1800
5.63	0.066551	0.2207	1900
5.92	0.136091	0.2193	2000

Results

Summary

Initial evaluations show promising results in generating Solidity code. More work required to understand effectiveness of input promt and max_length for training and inference.

Environmental Impact

Hardware Type: NVIDIA A100 • 80GB (80 GB VRAM) • 117 GB RAM • 8 vCPU
Hours used: 8
Cloud Provider: https://www.runpod.io
Compute Region: [More Information Needed]
Carbon Emitted: [More Information Needed]

Citation [optional]

BibTeX:

@misc{lozhkov2024starcoder,
      title={StarCoder 2 and The Stack v2: The Next Generation}, 
      author={Anton Lozhkov and Raymond Li and Loubna Ben Allal and Federico Cassano and Joel Lamy-Poirier and Nouamane Tazi and Ao Tang and Dmytro Pykhtar and Jiawei Liu and Yuxiang Wei and Tianyang Liu and Max Tian and Denis Kocetkov and Arthur Zucker and Younes Belkada and Zijian Wang and Qian Liu and Dmitry Abulkhanov and Indraneil Paul and Zhuang Li and Wen-Ding Li and Megan Risdal and Jia Li and Jian Zhu and Terry Yue Zhuo and Evgenii Zheltonozhskii and Nii Osae Osae Dade and Wenhao Yu and Lucas Krauß and Naman Jain and Yixuan Su and Xuanli He and Manan Dey and Edoardo Abati and Yekun Chai and Niklas Muennighoff and Xiangru Tang and Muhtasham Oblokulov and Christopher Akiki and Marc Marone and Chenghao Mou and Mayank Mishra and Alex Gu and Binyuan Hui and Tri Dao and Armel Zebaze and Olivier Dehaene and Nicolas Patry and Canwen Xu and Julian McAuley and Han Hu and Torsten Scholak and Sebastien Paquet and Jennifer Robinson and Carolyn Jane Anderson and Nicolas Chapados and Mostofa Patwary and Nima Tajbakhsh and Yacine Jernite and Carlos Muñoz Ferrandis and Lingming Zhang and Sean Hughes and Thomas Wolf and Arjun Guha and Leandro von Werra and Harm de Vries},
      year={2024},
      eprint={2402.19173},
      archivePrefix={arXiv},
      primaryClass={cs.SE}
}

APA:

Starcoder2-15B: BigCode Team. (2024). Starcoder2-15B [Software]. Available from https://huggingface.co/bigcode/starcoder2-15b

AlfredPros Smart Contracts Instructions: AlfredPros. (2023). Smart Contracts Instructions [Data set]. Available from https://huggingface.co/datasets/AlfredPros/smart-contracts-instructions

Model Card Contact

[email protected]

yoniebans
/

starcoder2-15b-qlora-solidity