Required Sagemaker instance size?
#5
by
lcrane
- opened
If I wanted to run opt-66b in an AWS Sagemaker instance, what are the memory/GPU/CPU requirements? I have tried a couple experiments (example code snippet below) but it seems to run out of memory loading the model and fail silently.
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-66b", use_fast=False)
model = AutoModelForCausalLM.from_pretrained("facebook/opt-66b", torch_dtype=torch.float16).cuda()