|
--- |
|
base_model: cmarkea/bloomz-3b-dpo-chat |
|
library_name: peft |
|
license: apache-2.0 |
|
datasets: |
|
- cmarkea/table-vqa |
|
language: |
|
- fr |
|
- en |
|
pipeline_tag: text-generation |
|
--- |
|
|
|
## Model Description |
|
|
|
**cmarkea/bloomz-3b-dpo-table-qa-latex** is a fine-tuned version of the **[cmarkea/bloomz-3b-dpo-chat](https://huggingface.co/cmarkea/bloomz-3b-dpo-chat)** model, specialized for table-based question answering (QA) tasks. The model has been trained on the **[table-vqa](https://huggingface.co/datasets/cmarkea/table-vqa)** dataset, which was developed by Crédit Mutuel Arkéa, and it processes tables provided in their LaTeX source format. |
|
|
|
This model is optimized for multilingual environments, supporting both French and English, and is especially effective in extracting and interpreting tabular data from documents. It has been fine-tuned for 2 days on an A100 40GB GPU and operates in bfloat16 precision to maximize resource efficiency. |
|
|
|
### Key Features |
|
- **Domain:** Table-based question answering (QA), particularly for extracting information from LaTeX-format tables. |
|
- **Language Support:** French and English, making it suitable for multilingual environments. |
|
- **Model Type:** Text-to-text language model. |
|
- **Precision:** bfloat16, optimizing computational efficiency. |
|
- **Training Duration:** 2 days on A100 40GB GPU. |
|
- **Fine-Tuning Method:** Full fine-tuning. |
|
|
|
This model is highly applicable in fields where tabular data needs to be queried and analyzed, such as financial reports, academic papers, and technical documentation. |
|
|
|
## Usage |
|
|
|
Here’s an example of how to use this model for table-based question answering: |
|
|
|
```python |
|
import torch |
|
from transformers import pipeline |
|
|
|
device = 0 if torch.cuda.is_available() else -1 |
|
table = '''\begin{tabular}{|c|c|c|} |
|
\hline |
|
Model & MAE-{$TKE$}-low & MAE-{$TKE$}-high\\ |
|
\hline |
|
U-FNET & 0.0048 & $1.09 \times 10^{-5}$\\ |
|
\hline |
|
\end{tabular}''' |
|
question = "What is the MAE-TKE-high value for the U-FNET model?" |
|
prompt = table + '\n' + question |
|
|
|
model = pipeline("text-generation", "cmarkea/bloomz-3b-dpo-chat", device=device) |
|
result = model(f"</s>{prompt}<s>", max_new_tokens=512) |
|
print(result) |
|
``` |
|
|
|
The model processes tables written in LaTeX format, so be sure to provide your tables in that form. |
|
|
|
## Performance |
|
|
|
This model was evaluated on 200 question-answer pairs extracted from 100 tables in the **[table-vqa](https://huggingface.co/datasets/cmarkea/table-vqa)** test set. Each table had two question-answer pairs: one in French and one in English. |
|
|
|
The evaluation used the **[LLM-as-Juries](https://arxiv.org/abs/2404.18796)** method, employing three judge models (GPT-4o, Gemini1.5 Pro, and Claude 3.5-Sonnet). The scoring was adapted to the table QA context, with a scale from 0 to 5 to ensure precision in assessing the model’s performance. |
|
|
|
Here’s a visualization of the results: |
|
|
|
 |
|
|
|
|
|
## Citation |
|
|
|
```bibtex |
|
@online{AgDePaligemmaTableQALatex, |
|
AUTHOR = {Tom Agonnoude, Cyrile Delestre}, |
|
URL = {https://huggingface.co/cmarkea/bloomz-3b-dpo-table-qa-latex}, |
|
YEAR = {2024}, |
|
KEYWORDS = {Table understanding, LaTeX, Multilingual, QA}, |
|
} |
|
``` |
