Change pipeline_tag to text generation, add placeholders for paper links, incorporate SL change recommendations

README.md

@@ -7,10 +7,11 @@ tags:
 license: apache-2.0
 datasets:
 - the_pile
-
+pipeline_tag: text-generation
 ---
 
 # Cerebras-GPT 111M
+[TODO: arXiv paper](https://www.cerebras.net), [TODO: Blog Post](https://www.cerebras.net) 
 
 ## Model Description
 
@@ -18,7 +19,7 @@ The Cerebras-GPT family is released to facilitate research into LLM scaling laws
 
 The family includes 111M, 256M, 590M, 1.3B, 2.7B, 6.7B, and 13B models.
 
-All models in the Cerebras-GPT family have been trained in accordance with [Chinchilla scaling laws](https://arxiv.org/abs/2203.15556) (20 tokens per model parameter) which yields improved performance at smaller model size.
+All models in the Cerebras-GPT family have been trained in accordance with [Chinchilla scaling laws](https://arxiv.org/abs/2203.15556) (20 tokens per model parameter) which is compute-optimal.
 
 These models were trained on the [Andromeda](https://www.cerebras.net/andromeda/) AI supercomputer comprised of 16 CS-2 wafer scale systems. Cerebras' [weight streaming technology](https://www.cerebras.net/blog/linear-scaling-made-possible-with-weight-streaming) simplifies the training of LLMs by disaggregating compute from model storage. This allowed for efficient scaling of training across nodes using simple data parallelism.
 
@@ -28,7 +29,7 @@ Cerebras systems for pre-training and fine tuning are available in the cloud via
 * Developed by: [Cerebras Systems](https://www.cerebras.net/)
 * License: Apache 2.0
 * Model type: Transformer-based Language Model
-* Architecture: GPT-2 model architecture with hyperparameters more similar to GPT-3.
+* Architecture: GPT-3 style architecture
 * Data set: The Pile
 * Tokenizer: Byte Pair Encoding
 * Vocabulary Size: 50257
@@ -109,7 +110,7 @@ Our tokenized version of the Pile has 371B tokens. We used byte-pair encoding, a
 
 ## Training procedure
 
-We use the GPT-2 model architecture with hyperparameters more similar to GPT-3. All of our layers use full attention as opposed to the GPT-3 style sparse banded attention. The model shapes were selected to either follow aspect ratio 80 or are the same shape as GPT3 models. Learning rate warmed up for 375M tokens (1500 steps for 111M and 256M models) and 10x cosine decayed. No dropout was used and weight decay was set to 0.1.
+We use the GPT-3 style model architecture. All of our layers use full attention as opposed to the GPT-3 style sparse banded attention. The model shapes were selected to either follow aspect ratio 80 or are the same shape as GPT-3 models. Learning rate warmed up for 375M tokens (1500 steps for 111M and 256M models) and 10x cosine decayed. No dropout was used and weight decay was set to 0.1.
 
 All models were trained to Chinchilla point: 20x more tokens than model parameters. Number of steps changed based on fixed batch size (2048) and sequence length (varied by model). See Training Table, below, for detail. 
 
@@ -192,14 +193,15 @@ We evaluate our models on the PILE validation set comprising 380M tokens. We als
 ## Uses and Limitations
 
 ### Intended Use
-The models we train are being open-sourced to further research into LLM scaling laws but are not intended for use as production models. You may fine-tune and adapt Cerebras-GPT models for deployment via either Cerebras [Model Studio](https://www.cerebras.net/product-cloud/) or the Hugging Face Transformers Library. We recommend assessing potential bias and harms prior to deployment of any LLM.
+The models we train are being open-sourced to further research into LLM scaling laws, but release these models with a fully permissive Apache license for the community to use freely.
+
+You may fine-tune and adapt Cerebras-GPT models for deployment via either Cerebras [Model Studio](https://www.cerebras.net/product-cloud/) or the Hugging Face Transformers Library. We recommend assessing potential bias and harms prior to deployment of any LLM.
 
-The primary intended users of these models are AI researchers and practitioners interested in testing the behaviors, capabilities, and limitations of large-scale generative language models.
 
 ### Out of Scope Use
 Cerebras-GPT models are trained on the Pile, with English language only, and are not suitable for machine translation tasks.
 
-Cerebras-GPT models have not been tuned for human-facing dialog applications like chatbots and will not respond to prompts in a similar way to models that have received instruction tuning or Reinforcement Learning from Human Feedback (RLHF) like Flan-T5 or ChatGPT. Cerebras-GPT models can be tuned using those methods.
+Cerebras-GPT models have not been tuned for human-facing dialog applications like chatbots and will not respond to prompts in a similar way to models that have received instruction tuning or reinforcement learning from human feedback (RLHF) like Flan-T5 or ChatGPT. Cerebras-GPT models can be tuned using those methods.
 
 ### Risk and Bias
 Like many large text corpora, the Pile contains offensive text. Cerebras-GPT models trained on this text may create offensive or undesirable text outputs regardless of whether the initial prompt is offensive. Human filtering of responses is recommended.