metadata
language:
- en
license: apache-2.0
library_name: transformers
tags:
- transformers
datasets:
- mwitiderrick/AlpacaCode
base_model: mwitiderrick/open_llama_3b_code_instruct_0.1
inference: true
model_type: llama
prompt_template: |
<s>[INST]
{prompt}
[/INST]
created_by: mwitiderrick
pipeline_tag: text-generation
model-index:
- name: mwitiderrick/open_llama_3b_instruct_v_0.2
results:
- task:
type: text-generation
dataset:
name: hellaswag
type: hellaswag
metrics:
- type: hellaswag (0-Shot)
value: 0.66
name: hellaswag(0-Shot)
- task:
type: text-generation
dataset:
name: winogrande
type: winogrande
metrics:
- type: winogrande (0-Shot)
value: 0.6322
name: winogrande(0-Shot)
- task:
type: text-generation
dataset:
name: arc_challenge
type: arc_challenge
metrics:
- type: arc_challenge (0-Shot)
value: 0.3447
name: arc_challenge(0-Shot)
source:
url: https://huggingface.co/mwitiderrick/open_llama_3b_instruct_v_0.2
name: open_llama_3b_instruct_v_0.2 model card
- task:
type: text-generation
name: Text Generation
dataset:
name: AI2 Reasoning Challenge (25-Shot)
type: ai2_arc
config: ARC-Challenge
split: test
args:
num_few_shot: 25
metrics:
- type: acc_norm
value: 40.7
name: normalized accuracy
source:
url: >-
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mwitiderrick/open_llama_3b_glaive_assistant_v0.1
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: HellaSwag (10-Shot)
type: hellaswag
split: validation
args:
num_few_shot: 10
metrics:
- type: acc_norm
value: 67.45
name: normalized accuracy
source:
url: >-
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mwitiderrick/open_llama_3b_glaive_assistant_v0.1
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MMLU (5-Shot)
type: cais/mmlu
config: all
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 27.74
name: accuracy
source:
url: >-
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mwitiderrick/open_llama_3b_glaive_assistant_v0.1
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: TruthfulQA (0-shot)
type: truthful_qa
config: multiple_choice
split: validation
args:
num_few_shot: 0
metrics:
- type: mc2
value: 35.86
source:
url: >-
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mwitiderrick/open_llama_3b_glaive_assistant_v0.1
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: Winogrande (5-shot)
type: winogrande
config: winogrande_xl
split: validation
args:
num_few_shot: 5
metrics:
- type: acc
value: 64.72
name: accuracy
source:
url: >-
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mwitiderrick/open_llama_3b_glaive_assistant_v0.1
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: GSM8k (5-shot)
type: gsm8k
config: main
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 1.97
name: accuracy
source:
url: >-
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mwitiderrick/open_llama_3b_glaive_assistant_v0.1
name: Open LLM Leaderboard
OpenLLaMA Glaive: An Open Reproduction of LLaMA
This is an OpenLlama model Code Instruct that has been fine-tuned on 1 epoch of the Glaive Assistsnt dataset.
Prompt Template
<s>[INST] {{ user_msg }} [/INST]
Usage
from transformers import AutoTokenizer, AutoModelForCausalLM,pipeline
tokenizer = AutoTokenizer.from_pretrained("mwitiderrick/open_llama_3b_glaive_code_v0.1")
model = AutoModelForCausalLM.from_pretrained("mwitiderrick/open_llama_3b_glaive_v0.1")
query = "Write a quick sort algorithm in Python"
text_gen = pipeline(task="text-generation", model=model, tokenizer=tokenizer, max_length=200)
output = text_gen(f"<s>[INST]{query}[/INST]")
print(output[0]['generated_text'])
"""
<s>[INST]Write a quick sort algorithm in Python[/INST]
Quick sort is a divide and conquer algorithm that sorts an array in-place.
It works by repeatedly dividing the array into two sub-arrays, sorting
them, and then merging them back together.
Here's a Python implementation of the quick sort algorithm:
def quick_sort(arr):
if len(arr) <= 1:
return arr
else:
pivot = arr[len(arr) // 2]
left = [x for x in arr if x < pivot]
right = [x for x in arr if x > pivot]
return quick_sort(left) + [pivot] + quick_sort
"""
Metrics
| Tasks |Version|Filter|n-shot| Metric |Value | |Stderr|
|---------|-------|------|-----:|--------|-----:|---|-----:|
|hellaswag|Yaml |none | 0|acc |0.4974|± |0.0050|
| | |none | 0|acc_norm|0.6600|± |0.0047|
| Groups |Version|Filter|n-shot| Metric | Value | |Stderr|
|----------|-------|------|-----:|-----------|-------:|---|-----:|
|truthfulqa|N/A |none | 0|bleu_max | 23.5771|± |0.5407|
| | |none | 0|bleu_acc | 0.2754|± |0.0002|
| | |none | 0|bleu_diff | -8.1019|± |0.5137|
| | |none | 0|rouge1_max | 49.5707|± |0.6501|
| | |none | 0|rouge1_acc | 0.2607|± |0.0002|
| | |none | 0|rouge1_diff| -9.8962|± |0.5492|
| | |none | 0|rouge2_max | 33.0399|± |0.8237|
| | |none | 0|rouge2_acc | 0.2313|± |0.0002|
| | |none | 0|rouge2_diff|-11.9054|± |0.7963|
| | |none | 0|rougeL_max | 46.3168|± |0.6705|
| | |none | 0|rougeL_acc | 0.2521|± |0.0002|
| | |none | 0|rougeL_diff|-10.1301|± |0.5669|
| | |none | 0|acc | 0.3191|± |0.0405|
| Tasks |Version|Filter|n-shot|Metric|Value | |Stderr|
|----------|-------|------|-----:|------|-----:|---|-----:|
|winogrande|Yaml |none | 0|acc |0.6322|± |0.0136|
| Tasks |Version|Filter|n-shot| Metric |Value | |Stderr|
|-------------|-------|------|-----:|--------|-----:|---|-----:|
|arc_challenge|Yaml |none | 0|acc |0.3234|± |0.0137|
| | |none | 0|acc_norm|0.3447|± |0.0139|
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 39.74 |
| AI2 Reasoning Challenge (25-Shot) | 40.70 |
| HellaSwag (10-Shot) | 67.45 |
| MMLU (5-Shot) | 27.74 |
| TruthfulQA (0-shot) | 35.86 |
| Winogrande (5-shot) | 64.72 |
| GSM8k (5-shot) | 1.97 |