minor style changes

README.md

@@ -19,7 +19,7 @@ BERTIN is a series of BERT-based models for Spanish. The current model hub point
 This is part of the
 [Flax/Jax Community Week](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104), organized by [HuggingFace](https://huggingface.co/) and TPU usage sponsored by Google Cloud.
 
-The aim of this project was to pre-train a RoBERTa-base model from scratch for during the Flax/JAX Community Event in which Google Cloud provided free TPUv3-8 to do the training using Huggingface's Flax implementations of their library.
+The aim of this project was to pre-train a RoBERTa-base model from scratch during the Flax/JAX Community Event, in which Google Cloud provided free TPUv3-8 to do the training using Huggingface's Flax implementations of their library.
 
 ## Spanish mC4
 
@@ -50,7 +50,9 @@ In order to efficiently build this subset of data, we decided to leverage a tech
 <caption>Figure 1. Perplexity distributions by percentage CCNet corpus.</caption>
 </figure>
 
-In this work, we tested the hypothesis that perplexity sampling might help reduce training-data size and training times.
+In this work, we tested the hypothesis that perplexity sampling might help
+reduce training-data size and training times, while keeping the performance of
+the final model.
 
 ## Methodology
 
@@ -60,13 +62,15 @@ In order to test our hypothesis, we first calculated the perplexity of each docu
 
 ![](./images/perp-p95.png)
 
-<caption>Figure 2. Perplexity distributions and quartiles (red lines) of 100M samples of mc4-es.</caption>
+<caption>Figure 2. Perplexity distributions and quartiles (red lines) of 44M samples of mc4-es.</caption>
 </figure>
 
 With the extracted perplexity percentiles, we created two functions to oversample the central quartiles with the idea of biasing against samples that are either too small (short, repetitive texts) or too long (potentially poor quality) (see Figure 3).
 
 The first function is a `Stepwise` that simply oversamples the central quartiles using quartile boundaries and a factor for the desired sampling frequency for each quartile, obviously given larger frequencies for middle quartiles (oversampling Q2, Q3, subsampling Q1, Q4). 
-The second function was a Gaussian approximation of the `Stepwise` function to smooth out the sharp boundaries and give a better approximation of the underlying distribution (see Figure 4).
+The second function weighted the perplexity distribution by a Gaussian-like
+function, to smooth out the sharp boundaries of the `Stepwise` function and
+give a better approximation to the desired underlying distribution (see Figure 4).
 
 We adjusted the `factor` parameter of the `Stepwise` function, and the `factor` and `width` parameter of the `Gaussian` function to roughly be able to sample 50M samples from the 416M in `mc4-es` (see Figure 4). For comparison, we also sampled randomly `mC4-es` up to 50M samples as well. In terms of sizes, we went down from 1TB of data to ~200GB.
 
@@ -85,7 +89,8 @@ We adjusted the `factor` parameter of the `Stepwise` function, and the `factor`
 <caption>Figure 4. Expected perplexity distributions of the sample `mc4-es` after applying `Gaussian` function.</caption>
 </figure>
 
-Figure 5 shows the perplexity distributions of the 50M subsets for each of the approximations. All subsets can be easily accessed for reproducibility purposes using the `bertin-project/mc4-es-sampled` dataset. Since the validation set was too small to extract a 10% (5M) of the samples using perplexity-sampling with the same `factor` and `width`, in our experiments we decided to sample from the training sets. In the `bertin-project/mc4-es-sampled` dataset, the `validation` set pulls the samples from the original `mc4`.
+Figure 5 shows the actual perplexity distributions of the generated 50M subsets for
+each of the executed subsampling procedures. All subsets can be easily accessed for reproducibility purposes using the `bertin-project/mc4-es-sampled` dataset. Since the validation set was too small to extract a 10% (5M) of the samples using perplexity-sampling with the same `factor` and `width`, in our experiments we decided to sample from the training sets. In the `bertin-project/mc4-es-sampled` dataset, the `validation` set pulls the samples from the original `mc4`.
 
 ```python
 from datasets import load_dataset
@@ -106,7 +111,9 @@ for split in ("random", "stepwise", "gaussian"):
 
 ![](./images/datasets-perp.png)
 
-<caption>Figure 5. Experimental perplexity distributions of the sampled `mc4-es` after applying `Gaussian` and `Stepwise` functions.</caption>
+<caption>Figure 5. Experimental perplexity distributions of the sampled
+`mc4-es` after applying `Gaussian` and `Stepwise` functions, and the `Random`
+control sample.</caption>
 </figure>
 
 `Random` sampling displayed the same perplexity distribution of the underlying true distribution, as can be seen in Figure 6.
@@ -270,7 +277,8 @@ With roughly 10 days worth of access to 3xTPUv3-8, we have achieved remarkable r
 
 The experience has been incredible and we feel this kind of events provide an amazing opportunity for small teams on low or non-existent budgets to learn how the big players in the field pre-train their models. The trade-off between learning and experimenting, and being beta-testers of libraries (Flax/JAX) and infrastructure (TPU VMs) is a marginal cost to pay compared to the benefits such access has to offer.
 
-We hope our work will set the basis for more small teams playing and experimenting with language models training on small subsets of data with reduced training times, since the performance of our models is on par with those trained on big machines for longer times.
+We hope our work will set the basis for more small teams playing and
+experimenting with language models training on smaller subsets of huge datasets with reduced training times, since the performance of our models is on par with those trained on big machines for longer times.
 
 ## Team members