metadata
license: llama2
datasets:
- gair-prox/open-web-math-pro
language:
- en
base_model:
- codellama/CodeLlama-7b-hf
CodeLlama-7B-ProXMath
ArXiv | Data: OpenWebMath-Pro | Code
CodeLlama-7B-ProXMath is a math-adapted language model that is continually pre-trained on OpenWebMath-Pro (a refined version by ProX) for 10B tokens.
Evaluations
ProX models are evaluated on 9 common math reasoning benchmarks.
| Model | asdiv | gsm8k | mathqa | mawps | minerva_math | mmlu_stem | sat_math | svamp | tabmwp | average |
|---|---|---|---|---|---|---|---|---|---|---|
| CodeLlama-7B | 50.7 | 11.8 | 14.3 | 62.6 | 5.0 | 20.4 | 21.9 | 44.2 | 30.6 | 29.1 |
| CodeLlama-7B-ProXMath | 67.9 | 35.6 | 38.9 | 82.7 | 17.6 | 42.6 | 62.5 | 55.8 | 41.3 | 49.4 |
Citation
@article{zhou2024programming,
title={Programming Every Example: Lifting Pre-training Data Quality like Experts at Scale},
author={Zhou, Fan and Wang, Zengzhi and Liu, Qian and Li, Junlong and Liu, Pengfei},
journal={arXiv preprint arXiv:2409.17115},
year={2024}
}