|
--- |
|
license: cc-by-4.0 |
|
datasets: |
|
- verifiers-for-code/CodeNet-16K |
|
- verifiers-for-code/CodeNet-Planner |
|
--- |
|
# Llama-3-LlamaPlanner |
|
|
|
## Model Description |
|
|
|
LlamaPlanner is a fine-tuned version of Meta's Llama-8B model, specifically designed for generating high-quality plans for code generation tasks. The model was trained on CodeNet-16k, a curated dataset of competitive programming problems, and their corresponding plans generated using Llama-3-70B. By leveraging the power of Parameter Efficient Fine-Tuning (PEFT), LlamaPlanner achieves performance comparable to much larger models in generating effective plans for code generation. |
|
|
|
## Model Details |
|
|
|
- **Base Model:** Llama-8B Instruct |
|
- **Fine-Tuning Approach:** Parameter Efficient Fine-Tuning (PEFT) using Unsloth |
|
- **Training Data:** CodeNet-16k, a filtered and deduplicated dataset of 16,500 competitive programming problems and their plans generated using Llama-3-70B |
|
- **Training Infrastructure:** H100-SXM5 GPU |
|
- **Evaluation Benchmarks:** HumanEval and EvalPlus |
|
|
|
## How to Use |
|
|
|
To use LlamaPlanner with the Hugging Face Transformers library, follow these steps: |
|
|
|
```python |
|
import transformers |
|
import torch |
|
|
|
model_id = "verifiers-for-code/LlamaPlanner" |
|
|
|
pipeline = transformers.pipeline( |
|
"text-generation", model=model_id, model_kwargs={"torch_dtype": torch.bfloat16}, device_map="auto" |
|
) |
|
|
|
prompt = "Generate a plan for a program that sorts an array of integers in ascending order." |
|
pipeline(prompt) |
|
``` |
|
|
|
## Training Details |
|
|
|
LlamaPlanner was trained using the following steps: |
|
|
|
1. Filtering and preprocessing the CodeNet dataset to create CodeNet-16k |
|
2. Generating plans for each problem using Llama-3-70B |
|
3. Formatting the problem description, input description, output description, and samples as input, and the generated plans as output |
|
4. Performing PEFT on the Llama-8B Instruct base model using Unsloth with different ranks and alpha values |
|
5. Training on an H100-SXM5 GPU for varying epochs |
|
|
|
## Evaluation Results |
|
|
|
LlamaPlanner was evaluated on the HumanEval and EvalPlus benchmarks using various methods, including zero-shot, self-planning, base planner model, and fine-tuned planner model. The results demonstrated that LlamaPlanner outperforms the base Llama-3-8B model by 14% on HumanEval and 11% on EvalPlus. Additionally, plans generated by LlamaPlanner helped boost the performance of Llama-3-70B on HumanEval. |
|
|
|
## Citation |
|
|
|
If you use LlamaPlanner in your research or applications, please cite the model using the following BibTeX entry: |
|
|
|
```bibtex |
|
@misc{llamaplanner, |
|
title={LlamaPlanner: A Fine-Tuned Llama-8B Model for Effective Plan Generation in Code Generation Tasks}, |
|
author={Abhinav Chinta and Sumuk Shashidhar and Vaibhav Sahai}, |
|
year={2023}, |
|
howpublished={\url{https://huggingface.co/verifiers-for-code/LlamaPlanner}}, |
|
} |
|
``` |
|
|
|
## License |
|
|
|
LlamaPlanner is released under the Apache License 2.0. |
|
|
|
## Acknowledgements |
|
|
|
We would like to thank Meta for releasing the Llama model family and the open-source community for their contributions to the development of large language models and their applications in code generation tasks. |