metadata

license: openrail
datasets:
  - lucasmccabe-lmi/CodeAlpaca-20k
language:
  - en
library_name: adapter-transformers

Model Card for `opt350m-codealpaca20k`

Model Description

An opt-350m model trained on the CodeAlpaca 20k dataset using quantization and Progressive Embedding Fine-Tuning (PEFT). The resulting model is designed to understand and generate code-related responses based on the prompts provided.

original model car

Model Architecture

Base Model: facebook/opt-350m
Fine-tuning: Parameter-Efficient Fine-Tuning (PEFT)

Training Data

The model was trained on the lucasmccabe-lmi/CodeAlpaca-20k dataset. This dataset contains code-related prompts and their corresponding outputs. Script used for training is avaiable here

Training Procedure

Quantization Configuration:

Quantization Type: 4-bit
Compute Dtype: float16
Double Quant: Enabled

PEFT Configuration:

Lora Alpha: 16
Lora Dropout: 0.5
Bias: None
Task Type: CAUSAL_LM
Target Modules: q_proj, v_proj, k_proj

Training Arguments:

Batch Size: 4 (per device)
Gradient Accumulation Steps: 2
Number of Epochs: 10
Optimizer: adamw_bnb_8bit
Learning Rate: 2e-5
Max Gradient Norm: 0.3
Warmup Ratio: 0.03
Learning Rate Scheduler: Cosine
Logging Steps: 10
Save Steps: 250
FP16 Precision: Enabled

Training information from wandb

train/total_flos: 72,761,086,854,758,400
train/train_loss: 1.5557164267259171
train/train_runtime: 5892.7285 seconds
train/train_steps_per_second: 4.248
_runtime: 5891.33976650238 seconds
_timestamp: 1,695,390,058.0198596
train/epoch: 10
train/global_step: 25,030
train/learning_rate: 8.371592860045851e-12
train/train_samples_per_second: 33.977
_step: 2,503
_wandb.runtime: 5890 seconds
train/loss: 1.4114

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("facebook/opt350m")
model = AutoModelForCausalLM.from_pretrained("harpomaxx/opt350m-codealpaca20k)

prompt = "Question: [Your code-related question here] ### Answer: "
inputs = tokenizer.encode(prompt, return_tensors="pt")
outputs = model.generate(inputs)
decoded_output = tokenizer.decode(outputs[0], skip_special_tokens=True)

print(decoded_output)

License

OpenRail

Model Card for opt350m-codealpaca20k