Update README

README.md

@@ -17,15 +17,24 @@ library_name: transformers
 </p>
 
 
+# Model Overview
 
-## Model Details
+## Description
+This family of models performs vision-language and text-only tasks including optical character recognition, multimodal reasoning, localization, common sense reasoning, world knowledge utilization, and coding.
+
+## License/Terms of Use
+[Creative Commons Attribution: Non-Commercial 4.0 International](https://spdx.org/licenses/CC-BY-NC-4.0) <br>
+
+# Model Details
 
 Today (September 17th, 2024), we introduce [NVLM 1.0](https://arxiv.org/abs/2409.11402), a family of frontier-class multimodal large language models (LLMs) that achieve state-of-the-art results on vision-language tasks, rivaling the leading proprietary models (e.g., GPT-4o) and open-access models (e.g., Llama 3-V 405B and InternVL 2). Remarkably, NVLM 1.0 shows improved text-only performance over its LLM backbone after multimodal training. 
 
-In this repo, we are open-sourcing NVLM-1.0-D-72B (decoder-only architecture), the decoder-only model weights and code for the community. 
+In this repo, we are open-sourcing NVLM-1.0-D-72B (decoder-only architecture), the decoder-only model weights and code for the community.
+
+
 
-## Other Resources
-[Inference Code (HF)](https://huggingface.co/nvidia/NVLM-D-72B/tree/main) &ensp; [Training Code (Coming soon)]() &ensp; [Website](https://research.nvidia.com/labs/adlr/NVLM-1/) &ensp; [Paper](https://arxiv.org/abs/2409.11402)
+## Reference(s)
+[Paper](https://arxiv.org/abs/2409.11402) &ensp; [Inference Code (HF)](https://huggingface.co/nvidia/NVLM-D-72B/tree/main) &ensp; [Training Code (Coming soon)]() &ensp; [Website](https://research.nvidia.com/labs/adlr/NVLM-1/) 
 
 ## Benchmark Results
 We train our model with legacy [Megatron-LM](https://github.com/NVIDIA/Megatron-LM/tree/main/megatron/legacy) and adapt the codebase to Huggingface for model hosting, reproducibility, and inference.
@@ -73,6 +82,22 @@ Results (as of September 17th, 2024) in the multimodal benchmarks are as follows
 | NVLM-D 1.0 72B (Huggingface) | (b)          | 81.7 | 93.2  | 73.1 | 89.0      | 🥳 84.3   (+4.5) |
 
 
+## Model Architectures
+
+**Network Architecture:** Decoder-Only Transformer 
+
+### Input
+**Input Type(s):** Text, Image <br>
+**Input Format(s):** String, [Pillow Library-Supported Formats](https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html) <br>
+**Input Dimensions:** One-Dimensional (1D), Two Dimensional (2D) <br>
+**Other Properties Related to Input:** Maximum Token Length = 128K Tokens <br>
+
+### Output
+**Output Type(s):** Text <br>
+**Output Format:** String <br>
+**Model Output:** 1D <br>
+**Other Properties Related to Output:** None <br> 
+
 ## How to use
 
 When converting Megatron checkpoint to Huggingface, we adapt [InternVL codebase](https://huggingface.co/OpenGVLab/InternVL2-Llama3-76B) to support model loading and multi-GPU inference in HF. 
@@ -291,6 +316,66 @@ response = model.chat(tokenizer, pixel_values, question, generation_config)
 print(f'User: {question}\nAssistant: {response}')
 ```
 
+## Software Integration
+**Runtime Engine(s)** 
+* PyTorch <br>
+
+**Supported Hardware Microarchitecture Compatibility:** <br>
+* NVIDIA Hopper <br>
+
+**[Preferred/Supported] Operating System(s):** <br>
+* Linux <br>
+
+## Inference
+**Engine:** PyTorch <br>
+**Test Hardware:** <br>
+* H100 <br>
+
+## Model Version(s)
+* v1.0-D (NVLM-D)
+
+## Training, Testing, and Evaluation Datasets 
+
+### Pre-Training Dataset
+
+**Link** <br>
+* [See Table 4](https://arxiv.org/abs/2409.11402) <br>
+
+**Data Collection Method by dataset** <br>
+* Hybrid: Automated, Human, Synthetic, Unknown <br>
+
+**Labeling Method by dataset** <br>
+* Hybrid: Automated, Human, Synthetic, Unknown <br>
+
+**Properties** 
+* Trained on image captions, image-text pairs, natural images, charts, documents, scene descriptions, and mathematical reasoning. <br>
+
+### Supervised Fine-Tuning Dataset
+**Link** <br>
+* [See Table 6](https://arxiv.org/abs/2409.11402) <br>
+
+**Data Collection Method by dataset** <br>
+* Hybrid: Automated, Human, Synthetic, Unknown <br>
+
+**Labeling Method by dataset** <br>
+* Hybrid: Automated, Human, Synthetic, Unknown <br>
+
+**Properties** 
+* Trained on image captions; general knowledge; image-text pairs; natural images; charts; diagrams; documents; scene descriptions; science diagrams, lessons, textbook data, and question-answer pairs; visual instruction tuning; and mathematical reasoning. <br>
+
+### Evaluation Dataset
+**Link** <br>
+* [See Section 6.1, "Benchmark"](https://arxiv.org/abs/2409.11402) <br>
+
+**Data collection method by dataset** <br>
+* Human <br>
+
+**Labeling method by dataset** <br>
+* Human <br>
+
+**Properties** <br>
+* Evaluated on general knowledge, visual answering, chart understanding, table, optical character recognition, and mathematical reasoning. <br> 
+
 
 ## Correspondence to
 Wenliang Dai* ([email protected]), Nayeon Lee* ([email protected]), Boxin Wang* ([email protected]), Zhuolin Yang* ([email protected]), Wei Ping* ([email protected])
@@ -307,9 +392,6 @@ Wenliang Dai* ([email protected]), Nayeon Lee* ([email protected]), Boxin Wang* (
 </pre>
 
 
-## License
-The use of this model is governed by the [cc-by-nc-4.0](https://spdx.org/licenses/CC-BY-NC-4.0)
-
 ## Ethical Considerations
 NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications.  When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.