|
--- |
|
license: apache-2.0 |
|
tags: |
|
- snowflake |
|
- arctic |
|
--- |
|
|
|
## Model Details |
|
|
|
Arctic is a Dense-MoE Hybrid transformer architecture pre-trained from scratch by the Snowflake AI |
|
Research Team. We are releasing model checkpoints for both the base and instruct-tuned versions of |
|
Arctic under an Apache-2.0 license. This means you can use them freely in your own research, |
|
prototypes, and products. |
|
|
|
* [Arctic-Base](link-here) |
|
* [Acrtic-Instruct](link-to-instruct) |
|
|
|
**Model developers** Snowflake |
|
|
|
**License** Apache-2.0 |
|
|
|
**Input** Models input text only. |
|
|
|
**Output** Models generate text and code only. |
|
|
|
**Model Release Date** April, 24th 2024. |
|
|
|
## Model Architecture |
|
|
|
Arctic combines a 10B dense transformer model with a residual 128x3.66B MoE MLP resulting in 480B |
|
total and 17B active parameters chosen using a top-2 gating. For more details about Arctic's model |
|
architecture please see our cookbook |
|
|
|
|
|
## Usage |
|
|
|
As of 4/24/2024 we are actively working with the maintainers of `transformers` to include the Arctic |
|
model implementation. Until this support is released please follow these instructions to get the |
|
required dependencies for using Arctic: |
|
|
|
```python |
|
pip install git+https://github.com/Snowflake-Labs/transformers.git |
|
``` |
|
|
|
Arctic leverages several features from [DeepSpeed](https://github.com/microsoft/DeepSpeed), you will need to |
|
install the latest version of DeepSpeed to get all of these required features: |
|
|
|
```python |
|
pip install "deepspeed>=0.15.0" |
|
``` |
|
|
|
### Inference |
|
|
|
To get the best performance with Arctic we highly recommend using TRT-LLM or vLLM for inference. However you |
|
can also use `transformers` to load |
|
the model for text generation. Due to the model size we recommend using a single 8xH100 instance from your |
|
favorite cloud provider such as: AWS [p5.48xlarge](https://aws.amazon.com/ec2/instance-types/p5/), |
|
Azure [ND96isr_H100_v5](https://learn.microsoft.com/en-us/azure/virtual-machines/nd-h100-v5-series), etc. |
|
|
|
In addition, if you would like to access Acrtic via API we have colloborated with several inference API |
|
providers to host Acrtic such as AWS, Microsoft Azure, NVIDIA Foundry, Lamini, Perplexity, Replicate and Together. |
|
|
|
```python |
|
import torch |
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
|
|
tokenizer = AutoTokenizer.from_pretrained("snowflake/arctic") |
|
model = AutoModelForCausalLM.from_pretrained("snowflake/arctic", device_map="auto", torch_dtype=torch.bfloat16) |
|
|
|
input_text = "Hello my name is " |
|
input_ids = tokenizer(input_text, return_tensors="pt").to("cuda") |
|
|
|
outputs = model.generate(**input_ids, max_new_tokens=20) |
|
print(tokenizer.decode(outputs[0])) |
|
``` |
|
|
|
### Fine-Tuning |
|
|
|
TODO: add link and extra details about fine-tuning scripts |
|
|
|
## Metrics |
|
|
|
TODO: add summary of metrics here, we don't necessarily need to compare to others but we can if we want |
|
|
|
## Training Data |
|
|
|
TODO: add short description and links to training data related cookbook(s) |
|
|
|
|
