Add citations and other fixes (#4)

- citations and other fixes (76c98527e2cee753162f850963418522e87f476a)
- fixes (149e56a9f65052cc92c8f6c61bd543bdf7a007ff)
- more fixes (7bde7c000db400745b5710f7b7f31a5542a23107)

common.py

@@ -308,7 +308,7 @@ global_div = Div(
             "Deduplication is beneficial for LM pretraining in several ways, with the most important being controllable upsampling. With unique data, teams gain fine-grained control over the training data. Other benefits of deduplication include avoiding train-test overlap which prevents evaluation contamination."
         ),
         P(
-            "Duplicate data can lead to a strong double descent phenomenon, where repeated data causes test loss to increase midway through training [2]. Additionally, it reduces the risk of memorization [1]. By implementing deduplication and selective upsampling, we gain control over the pretraining data distribution, rather than relying on the inherent distribution of the source."
+            "Duplicate data can lead to a strong double descent phenomenon, where repeated data causes test loss to increase midway through training",  D_cite(bibtex_key="hernandez2022scaling"), ". Additionally, it reduces the risk of memorization", D_cite(bibtex_key="lee2022deduplicating"), ". By implementing deduplication and selective upsampling, we gain control over the pretraining data distribution, rather than relying on the inherent distribution of the source."
         ),
         P(
             "To illustrate the need for deduplication, below is the distribution of near-duplicate clusters, organized into buckets of 100. The first bucket contains clusters with sizes ranging from 2 to 100, as found in the Common Crawl dataset. Some clusters even reach up to a million documents."
@@ -406,6 +406,7 @@ global_div = Div(
             "In sizable clusters comprising 1000 or more documents, we observe a trend towards templatization. This involves the recurrent use of standardized language to convey general topics such as terms and conditions, warnings, and disclaimers. Such language is prevalent on commercial websites, offering a consistent and efficient way to communicate commonly encountered information."
         ),
         Img(src="images/image9.png", style="max-width: 100%;"),
+        id="section47",
     ),
     Section(
         H2("Personally Identifiable Information Removal"),
@@ -435,7 +436,7 @@ global_div = Div(
                 style="list-style-type: none",
             ),
         ),
-        id="section47",
+        id="section48",
     ),
     Section(
         H2("Normalization Form C"),
@@ -455,7 +456,7 @@ global_div = Div(
                 style="list-style-type: none",
             )
         ),  # "background-color= gray" "color= blue" maybe add this later
-        id="section48",
+        id="section49",
     ),
     Section(
         H3("NFC Examples"),

curated.py

@@ -296,7 +296,7 @@ table_div_hn = Div(NotStr(table_html_hn))
 uirc_filter = pd.DataFrame(
     {
         "Dataset": [
-            "Ubunutu IRC",
+            "Ubuntu IRC",
         ],
         "Lines Downloaded": [
             "37966",
@@ -609,7 +609,7 @@ data_preprocessing_div = Div(
         B("Unigram Log Probability Filter"),
         " calculates the log probability of each unigram to measure the significance of individual words. This step quantifies the importance of individual words but may not capture the semantic meaning of words. To calculate the average log word probability, we use word frequencies extracted from the ",
         A("1T Web-gram corpus", href="https://catalog.ldc.upenn.edu/LDC2006T13"),
-        ". Specifically, we use the list available created by ",
+        ". Specifically, we use the available list created by ",
         A(
             "Rachel Tatman",
             href="https://www.kaggle.com/datasets/rtatman/english-word-frequency",
@@ -713,7 +713,7 @@ filtering_process = Div(
                 ),
                 ". The",
                 D_code("huggingface dataset.to_json", language="python"),
-                " function was used to convert the original parqet format to the jsonl format.",
+                " function was used to convert the original parquet format to the jsonl format.",
             ),
             P(
                 B("Filtering: "),
@@ -854,7 +854,7 @@ filtering_process = Div(
                     style="margin-bottom: -3px",
                 ),
                 Li(
-                    "Paragraph Count Filter: The paper must have at least 5 paragraphs after removing paragraphs with less than -20 average log world probability",
+                    "Paragraph Count Filter: The paper must have at least 5 paragraphs after removing paragraphs with less than -20 average log word probability",
                     style="margin-bottom: -3px",
                 ),
                 Li(
@@ -1140,7 +1140,7 @@ filtering_process = Div(
         Raw single line in data: <P> Hi I am speaker
         After tag removal: P Hi I am speaker
         We remove everything that starts with ["P", "BRK", "CHAPTER", "/P"]
-        and only keep tagnae == SPEAKER
+        and only keep tagname == SPEAKER
         because line starting with <SPEAKER> TEXT TEXT ....... has the relevant text
         """,
                 style="block",
@@ -1217,7 +1217,7 @@ filtering_process = Div(
                     style="margin-bottom: -3px",
                 ),
                 Li(
-                    "As discussed above, the comment heirarchies required a thoughful approach to extracting meaningful data. ",
+                    "As discussed above, the comment hierarchies required a thoughtful approach to extracting meaningful data. ",
                     style="margin-bottom: -3px",
                 ),
                 Li(
@@ -1374,7 +1374,7 @@ filtering_process = Div(
             P(B("Unique Data Preparation Challenges: ")),
             Ul(
                 Li(
-                    "Handling code block was a required finding the specific blocks and exacting the details in one snippet.",
+                    "Handling code block was a required finding the specific blocks and extracting the details in one snippet.",
                     style="margin-bottom: -3px",
                 ),
                 Li(

main.py

@@ -346,16 +346,22 @@ def main():
                             ),
                             Li(
                                 A(
-                                    "Personally Identifiable Information Removal",
+                                    "Analysis of Near-Duplicate Clusters",
                                     href="#section47",
                                 )
                             ),
                             Li(
                                 A(
-                                    "Normalization Form C",
+                                    "Personally Identifiable Information Removal",
                                     href="#section48",
                                 )
                             ),
+                            Li(
+                                A(
+                                    "Normalization Form C",
+                                    href="#section49",
+                                )
+                            ),
                         ),
                     ),
                     Div(
@@ -876,7 +882,7 @@ def intro():
                 A(B("TxT360 (Trillion eXtracted Text),"), href="https://huggingface.co/datasets/LLM360/TxT360"),
                 " the first dataset to globally deduplicate 99 CommonCrawl snapshots and 14 high-quality data sources from diverse domains (e.g., FreeLaw, PG-19, etc.). The large-scale deduplication process and rich metadata stored enables precise control over data distribution. We demonstrate a simple but effective upsampling recipe that creates a 15+ trillion-token corpus, outperforming FineWeb 15T on several key metrics. With the information, TxT360 empowers pre-trainers to explore more advanced weighting techniques, a feature not commonly available in previous pre-training datasets. Our findings highlight the importance of both high-quality data sources and appropriate weighting for optimal blending in LLM training." 
             ),
-            P("In line with our 360° open source spirit, we document all detailed steps, reasons of our decisions, detailed statistics, our code (stay tuned!), analysis results and more, in additional to the dataset itself. We hope this can serve as a useful resource for future developers."
+            P("In line with our 360° open source spirit, we document all detailed steps, reasons of our decisions, detailed statistics, our code (stay tuned!), analysis results and more, in addition to the dataset itself. We hope this can serve as a useful resource for future developers."
             ),
             plotly2fasthtml(all_eval_res_figs["MMLU"]),
             P(
@@ -890,7 +896,7 @@ def intro():
                 D_cite(bibtex_key="dclm"),
                 "and RedPajama V2,",
                 D_cite(bibtex_key="redpajama-v2"),
-                "we also hope to provide a dataset at this scale that is ready to go, without requiring futher filtering."
+                "we also hope to provide a dataset at this scale that is ready to go, without requiring further filtering."
             ),
             P(
                 B("How to Read this Blog Post?"),
@@ -902,7 +908,7 @@ def intro():
         Section(
             H2("Why TxT360"),
             P(
-                "In this year we have seen excellent datasets released by the community. Among those, most datasets focus on one source (e.g., crawled websites, code bases, papers). However, it is not trivial to combine these sources together due to the potential duplicaiton across them. TxT360 is the first dataset to combine most of sources commonly used in pretraining."
+                "In this year we have seen excellent datasets released by the community. Among those, most datasets focus on one source (e.g., crawled websites, code bases, papers). However, it is not trivial to combine these sources together due to the potential duplication across them. TxT360 is the first dataset to combine most of sources commonly used in pretraining."
             ),
             new_table_div_1,
             # table_div_1,
@@ -911,7 +917,7 @@ def intro():
                 "In LLM pretraining, it is common to combine all possible text sources due to the Scaling Law. Crawled web pages are included to provide a vast quantity of data which can cover long tail and diverse information, while curated datasets such as Wikipedia are also used, which often provide the 'deep-dive' domain information. By integrating the reach of web data with the quality of curated sources, TxT360 meets and surpasses the rigorous standards required for state-of-the-art LLM pre-training."
             ),
             P(
-                "** TxT360 does not include very specific domains such as code and math. This decision was made due to the perceived low duplication code with other sources, and the different logic requiring to build those datasets. We leave those work to future work and recommend users refer to existing projects such as Stack V2",
+                "** TxT360 does not include very specific domains such as code and math. This decision was made due to the perceived low duplication code with other sources, and the different logic required to build those datasets. We leave that to future work and recommend users refer to existing projects such as Stack V2",
                 D_cite(bibtex_key="lozhkov2024starcoder2stackv2"),
                 ".",
             ),

web.py

@@ -657,7 +657,7 @@ def web_data():
             """, 
         ),
         P(B("Toxic Lines: "), """
-        When doing manual inspection on the data, we found that there are some adult ads in the beginning or end of the 
+        When manually inspecting the data, we found that there are some adult ads in the beginning or end of the
         document (with a sample shown below), which are hard to remove via document-level filtering strategies. Inspired 
         by this, we develop line-level detoxification using a bad word list from LDNOOBW (+ rule: word length < 10 + the 
         line is in the first 3 lines or in the last 3 lines) to remove toxic lines. Specifically, we do not only consider 
@@ -1455,7 +1455,7 @@ def web_data():
         ),
         P("""
         Both Dolma and RedPajama V2 split texts into words using white spaces and newline symbols. However, 
-        DataTrove employs a tokenizer to split texts into words and ignore punctuations, resulting in a higher 
+        DataTrove employs a tokenizer to split texts into words and ignore punctuation, resulting in a higher
         word count compared to simple `text.split()`.
         We decided to use simple `len(text.split())` to compute the word count.
         """),