|
--- |
|
inference: false |
|
pipeline_tag: image-text-to-text |
|
license: apache-2.0 |
|
datasets: |
|
- VIMA/VIMA-Data |
|
tags: |
|
- llara |
|
- llava |
|
- robotics |
|
- vlm |
|
--- |
|
<br> |
|
<be> |
|
|
|
# LLaRA Model Card |
|
|
|
This model is released with paper **[LLaRA: Supercharging Robot Learning Data for Vision-Language Policy](https://arxiv.org/abs/2406.20095)** |
|
|
|
[Xiang Li](https://xxli.me)<sup>1</sup>, [Cristina Mata](https://openreview.net/profile?id=~Cristina_Mata1)<sup>1</sup>, [Jongwoo Park](https://github.com/jongwoopark7978)<sup>1</sup>, [Kumara Kahatapitiya](https://www3.cs.stonybrook.edu/~kkahatapitiy)<sup>1</sup>, [Yoo Sung Jang](https://yjang43.github.io/)<sup>1</sup>, [Jinghuan Shang](https://elicassion.github.io/)<sup>1</sup>, [Kanchana Ranasinghe](https://kahnchana.github.io/)<sup>1</sup>, [Ryan Burgert](https://ryanndagreat.github.io/)<sup>1</sup>, [Mu Cai](https://pages.cs.wisc.edu/~mucai/)<sup>2</sup>, [Yong Jae Lee](https://pages.cs.wisc.edu/~yongjaelee/)<sup>2</sup>, and [Michael S. Ryoo](http://michaelryoo.com/)<sup>1</sup> |
|
|
|
<sup>1</sup>Stony Brook University <sup>2</sup>University of Wisconsin-Madison |
|
|
|
## Model details |
|
|
|
**Model type:** |
|
D-RT2-Style is one of the baselines in our LLaRA paper, following the style of [RT2](https://robotics-transformer.github.io/). |
|
This is an open-source visuomotor policy trained by fine-tuning [LLaVA-7b-v1.5](https://huggingface.co/liuhaotian/llava-v1.5-7b) on instruction-following data `D-RT2-Style`, converted from [VIMA-Data](https://huggingface.co/datasets/VIMA/VIMA-Data). |
|
For the conversion code, please refer to [convert_vima.ipynb](https://github.com/LostXine/LLaRA/blob/main/datasets/convert_vima.ipynb) |
|
|
|
**Model date:** |
|
llava-1.5-7b-llara-D-RT2-Style-VIMA-80k was trained in June 2024. |
|
|
|
**Paper or resources for more information:** |
|
https://github.com/LostXine/LLaRA |
|
|
|
**Where to send questions or comments about the model:** |
|
https://github.com/LostXine/LLaRA/issues |
|
|
|
## Intended use |
|
**Primary intended uses:** |
|
The primary use of LLaRA is research on large multimodal models for robotics. |
|
|
|
**Primary intended users:** |
|
The primary intended users of the model are researchers and hobbyists in robotics, computer vision, natural language processing, machine learning, and artificial intelligence. |
