Edit model card

NOTE: This LoRA was trained on Llama-30b AFTER additional pretraining. I intend on providing the LoRA of that pretraining too. Applying this LoRA to base Llama-30b will likely result in a performance reduction. I have uploaded the fp16 merged weights here

Mostly untested!

Find GPTQ quantized weights and full model card here: https://huggingface.co/bhenrym14/airoboros-33b-gpt4-1.4.1-lxctx-PI-16384-GPTQ

RoPE Scaled QLoRA Fine-tune of Llama-33b on airoboros-gpt4-1.4.1 (LoRA)

Overview

This is Jon Durbin's Airoboros 33B GPT4 1.4 (LoRA) with several key modifications:

  • Context length extended to 16384 by RoPE Scaled Embeddings.
  • The Llama-33b base model is pretrained for additional 100 steps on 8192 length sequences from the pile dataset.
  • Used airoboros-gpt4-1.4.1 dataset instead of airoboros-gpt4-1.4

This is a QLoRA fine-tune

Pretraining took 10 hours. Finetuning took ~41 hours on 1x RTX 6000 Ada.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference API
Unable to determine this model's library. Check the docs .

Dataset used to train bhenrym14/airoboros-33b-gpt4-1.4.1-lxctx-PI-16384-LoRA