VaibhavSahai
commited on
Commit
•
d207946
1
Parent(s):
302c4f5
Update README.md
Browse files
README.md
CHANGED
@@ -1,3 +1,72 @@
|
|
1 |
---
|
2 |
license: cc-by-4.0
|
|
|
|
|
|
|
3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
---
|
2 |
license: cc-by-4.0
|
3 |
+
datasets:
|
4 |
+
- verifiers-for-code/CodeNet-16K
|
5 |
+
- verifiers-for-code/CodeNet-Planner
|
6 |
---
|
7 |
+
# Llama-3-LlamaPlanner
|
8 |
+
|
9 |
+
## Model Description
|
10 |
+
|
11 |
+
LlamaPlanner is a fine-tuned version of Meta's Llama-8B model, specifically designed for generating high-quality plans for code generation tasks. The model was trained on CodeNet-16k, a curated dataset of competitive programming problems, and their corresponding plans generated using Llama-3-70B. By leveraging the power of Parameter Efficient Fine-Tuning (PEFT), LlamaPlanner achieves performance comparable to much larger models in generating effective plans for code generation.
|
12 |
+
|
13 |
+
## Model Details
|
14 |
+
|
15 |
+
- **Base Model:** Llama-8B Instruct
|
16 |
+
- **Fine-Tuning Approach:** Parameter Efficient Fine-Tuning (PEFT) using Unsloth
|
17 |
+
- **Training Data:** CodeNet-16k, a filtered and deduplicated dataset of 16,500 competitive programming problems and their plans generated using Llama-3-70B
|
18 |
+
- **Training Infrastructure:** H100-SXM5 GPU
|
19 |
+
- **Evaluation Benchmarks:** HumanEval and EvalPlus
|
20 |
+
|
21 |
+
## How to Use
|
22 |
+
|
23 |
+
To use LlamaPlanner with the Hugging Face Transformers library, follow these steps:
|
24 |
+
|
25 |
+
```python
|
26 |
+
import transformers
|
27 |
+
import torch
|
28 |
+
|
29 |
+
model_id = "verifiers-for-code/LlamaPlanner"
|
30 |
+
|
31 |
+
pipeline = transformers.pipeline(
|
32 |
+
"text-generation", model=model_id, model_kwargs={"torch_dtype": torch.bfloat16}, device_map="auto"
|
33 |
+
)
|
34 |
+
|
35 |
+
prompt = "Generate a plan for a program that sorts an array of integers in ascending order."
|
36 |
+
pipeline(prompt)
|
37 |
+
```
|
38 |
+
|
39 |
+
## Training Details
|
40 |
+
|
41 |
+
LlamaPlanner was trained using the following steps:
|
42 |
+
|
43 |
+
1. Filtering and preprocessing the CodeNet dataset to create CodeNet-16k
|
44 |
+
2. Generating plans for each problem using Llama-3-70B
|
45 |
+
3. Formatting the problem description, input description, output description, and samples as input, and the generated plans as output
|
46 |
+
4. Performing PEFT on the Llama-8B Instruct base model using Unsloth with different ranks and alpha values
|
47 |
+
5. Training on an H100-SXM5 GPU for varying epochs
|
48 |
+
|
49 |
+
## Evaluation Results
|
50 |
+
|
51 |
+
LlamaPlanner was evaluated on the HumanEval and EvalPlus benchmarks using various methods, including zero-shot, self-planning, base planner model, and fine-tuned planner model. The results demonstrated that LlamaPlanner outperforms the base Llama-3-8B model by 14% on HumanEval and 11% on EvalPlus. Additionally, plans generated by LlamaPlanner helped boost the performance of Llama-3-70B on HumanEval.
|
52 |
+
|
53 |
+
## Citation
|
54 |
+
|
55 |
+
If you use LlamaPlanner in your research or applications, please cite the model using the following BibTeX entry:
|
56 |
+
|
57 |
+
```bibtex
|
58 |
+
@misc{llamaplanner,
|
59 |
+
title={LlamaPlanner: A Fine-Tuned Llama-8B Model for Effective Plan Generation in Code Generation Tasks},
|
60 |
+
author={Abhinav Chinta and Sumuk Shashidhar and Vaibhav Sahai},
|
61 |
+
year={2023},
|
62 |
+
howpublished={\url{https://huggingface.co/verifiers-for-code/LlamaPlanner}},
|
63 |
+
}
|
64 |
+
```
|
65 |
+
|
66 |
+
## License
|
67 |
+
|
68 |
+
LlamaPlanner is released under the Apache License 2.0.
|
69 |
+
|
70 |
+
## Acknowledgements
|
71 |
+
|
72 |
+
We would like to thank Meta for releasing the Llama model family and the open-source community for their contributions to the development of large language models and their applications in code generation tasks.
|