Model Card: LlavaOLMoBitnet1B

Multimodal Large Language Models (MM-LLMs) have seen significant advancements in the last year, demonstrating impressive performance across tasks. However, to truly democratize AI, models must exhibit strong capabilities and be able to run efficiently on small compute footprints accessible by most. Part of this quest, we introduce LLaVaOLMoBitnet1B - the first Ternary Multimodal LLM capable of accepting Image(s)+Text inputs to produce coherent textual responses. The model is fully open-sourced along with training scripts to encourage further research in this space. We also release a technical report highlighting the training process, eval details, challenges associated with ternary models and future opportunities.

Authors: Jainaveen Sundaram, Ravi Iyer

Training details and Evaluation

Two step training pipeline outlined in the LLaVa1.5 paper, consisting of two phases: (1) A Pre-training phase for feature alignment followed by an (2) End-to-end instruction fine-tuning The pre-training phase involves 1 epoch on a filtered subset of 595K Conceptual Captions [2], with only the projection layer weights updated. For instruction fine-tuning, we use 1 epoch of the LLaVa-Instruct-150K dataset, with both projection layer and LLM weights updated. For more details and model evaluation, please refer to the technical report.

How to use

Start off by cloning the repository:

git clone https://huggingface.co/IntelLabs/LlavaOLMoBitnet1B
cd LlavaOLMoBitnet1B

Install all the requirements by following instructions on requirements.txt

You are all set! Run inference by calling:

python llava_olmo.py 

To pass in your own query, modify the following lines within the llava_olmo.py file:

#Define Image and Text inputs..  

text = "Be concise. What are the four major tournaments of the sport shown in the image?"

url = "https://farm3.staticflickr.com/2157/2439959136_d932f4e816_z.jpg"

Model Sources

View the Technical report here

Ethical Considerations

Intel is committed to respecting human rights and avoiding causing or contributing to adverse impacts on human rights. See Intel’s Global Human Rights Principles. Intel’s products and software are intended only to be used in applications that do not cause or contribute to adverse impacts on human rights.

Ethical Considerations	Description
Data	The model was trained using the LLaVA-v1.5 data mixture as described above.
Human life	The model is not intended to inform decisions central to human life or flourishing.
Mitigations	No additional risk mitigation strategies were considered during model development.
Risks and harms	This model has not been assessed for harm or biases, and should not be used for sensitive applications where it may cause harm.
Use cases	-

Citation

If you found our work useful, please cite us at:

@misc{sundaram2024llavaolmobitnet1bternaryllmgoes,
      title={LLaVaOLMoBitnet1B: Ternary LLM goes Multimodal!}, 
      author={Jainaveen Sundaram and Ravishankar Iyer},
      year={2024},
      eprint={2408.13402},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2408.13402}, 
}

License

Apache-2.0