🤗 HuggingFace 🍷 FineWeb datasets

Read our technical report!

This organization hosts the 🍷 FineWeb datasets, a collection of text datasets sourced from the web (CommonCrawl), released under a permissive license (ODC-By).

The creation of 🍷 FineWeb involved careful processing and filtering of large amounts of web data with the aim of lowering the barriers to entry to anyone intending to pretrain high-performance large language models.

All code and artefacts needed for reproduction are public and built on top of open source libraries, such as the 🤗 libraries datatrove, nanotron or lighteval.

Version 1 of the 🍷 FineWeb dataset is available here. Our ablation models can be found here.