Our model for the 2024 BabyLM challenge 100M words track.

To download and use this model the fla package has to be installed:

pip install -U git+https://github.com/sustcsonglin/flash-linear-attention

Inference Examples

Inference API (serverless) does not yet support model repos that contain custom code.

Dataset used to train PatrickHaller/hgrn2_pile_100m_distill_babylm

Collection including PatrickHaller/hgrn2_pile_100m_distill_babylm