evaluation/README.md

# Benchmarks evaluations for tablegpt

<p align="center">
    <a href="#-About">🔥About</a> •

    <a href="#-Usage">💻Usage</a> •

</p>


## About

</div>

This is a repo opened for evaluation on different table-related benchmarks for tablegpt. 

Given the complexity of table QA tasks and the uncertainty of input instructions,  we provide evaluation datasets and scripts for 7 capabilities: 

- ✨Code correction based on tables 
- ✨Refusal of ambiguous questions
- ✨Table & field recall in multi-table scenarios
- ✨Table QA output code executable
- ✨Table-Bench.
- ✨Text2Sql.
- ✨TableInstruct, which includes a series of table-related evaluation benchmarks.

In addition, we have integrated other general abilities benchmarks like HumanEval, MBPP and MMLU/CMMLU.
We have built an inference method based on the  local model path using vLLM as the backend, and defined a set of example prompts templates for the above benchmarks. 

## Usage

</div>
</details>

⏬ To use this framework, please first install the repository from GitHub:

```shell

git clone https://github.com/tablegpt/tablegpt-eval

cd tablegpt-eval

pip install -r requirements.txt

```

</div>
</details>

[!Tips]
1. You can run all the benchmarks with the default params by running command `bash run_benchmarks.sh`.
2. If you want more configuration options for running parameters, refer to the typical Python script.
3. Download the .db files before running text2sql evaluation scripts. Download urls refer to `/table_related_benchmarks/evalset/bird_data/dev_databases/README.md`(Bird dev) & `table_related_benchmarks/evalset/spider_data/dev_database/README.md` (Spider dev) & `table_related_benchmarks/evalset/spider_data/test_database/README.md` (Spider test).


### Code correction eval

We provide a non-executable eval dataset based on the Python language. Eval dataset path:

```python

table_related_benchmarks/evalset/code_correction_test/correction_set.json

```

We use the  ***executable_pass_rate*** and ***absolute_match_rate***  of the corrected code in pass-1 to evaluate the model's code correction ability. You can perform code-correction evaluation by running the following Python command:

```bash

python table_related_benchmarks/run_code_correction_eval.py \

		--model_path  <EVAL MODEL PATH> \

		--template  <CHAT_TEMPLATE_NAME, support [llama3, baichuan, chatglm, None], default None> \

    	--eval_results_save_path <PATH TO SAVE THE EVAL RESULTS> \

        --gpus_num <NUMBER OF GPU TO RUN INFERENCE> \

        --temperature <ONE OF THE INFERENCE PARAMETER>

```

### Ambiguous reject eval

We provide 298 table-based queries, with a ratio of about 1:3 between queries marked as ambiguous (to be rejected) and queries that should be accepted and correctly answered. Dataset path:

```python

# test queries

evalset/reject_test/test_query.json

# queries with ground truth

evalset/reject_test/ground_truth.json

```

We use **accuracy**, **recall**, and **F1 score** as metrics to evaluate the LLM's ability in this task. You can perform reject evaluation by  running the following Python command:

```bash

python table_related_benchmarks/run_reject_eval.py \

    --model_path <EVAL MODEL PATH>  \

    --save_path <LLM OUTPUT CONTENT SAVE PATH> \

    --gpus_num <NUMBER OF GPU TO RUN INFERENCE> \

    --temperature <ONE OF THE INFERENCE PARAMETER>

```

### Table&Fields recall eval

The provided eval dataset path:

```python

table_related_benchmarks/evalset/retrieval_test/recall_set.json

```

We use a series of evaluation metrics such as **recall**, **precision**, **Jaccard similarity**, and **Hamming loss** to assess the LLM's performance in table and field retrieval tasks.  You can perform recall evaluation by  running the following Python command:

```bash

python table_related_benchmarks/run_recall_eval.py \

    --model_path <EVAL MODEL PATH> \

    --temperature <TEMPERATURE> \

    --gpus_num <NUMBER OF GPU TO RUN INFERENCE> 

```

### Table QA executable 

Provide 2178 table based queries,  eval dataset path:

```python

table_related_benchmarks/evalset/table_qa_execuate_test/tableqa_samples_with_paths.jsonl

```

We employ ***executable_pass_rate***  of pass-1 to employ the model's tableQA code generation ability. You can perform tableQA evaluation by  running the following Python command:

```bash

python table_related_benchmarks/run_tableqa_execution_eval.py \

    --model_path <EVAL MODEL PATH> \

    --temperature <ONE OF THE INFERENCE PARAMETER> \

    --gpus_num <NUMBER OF GPU TO RUN INFERENCE> 

```

### TableBench evaluation

The provided eval dataset path:

```python

table_related_benchmarks/evalset/TableBench

```

In the evaluation of TableBench, Rough-L was used to assess general QA questions, while pass@1 was used as the evaluation metric for visualization-type samples.  You can perform TableBench evaluation by the following command:

```bash

python table_related_benchmarks/run_table_bench_eval.py \

    --model_path <EVAL MODEL PATH> \

    --temperature <ONE OF THE INFERENCE PARAMETER> \

    --gpus_num <NUMBER OF GPU TO RUN INFERENCE> 

```

### TableInstruct

The provided eval dataset path:

```python

table_related_benchmarks/evalset/TableInstruct

```

You can perform TableInstruct evaluation by the following command:

```bash

python table_related_benchmarks/run_table_instruct_eval.py \

    --model_path <EVAL MODEL PATH> \

    --temperature <ONE OF THE INFERENCE PARAMETER> \

    --gpus_num <NUMBER OF GPU TO RUN INFERENCE> 

```

### Text2Sql
```bash

python table_related_benchmarks/run_text2sql_eval.py --model_path <EVAL MODEL PATH> 

```

### HumanEval
Perform HumanEval evaluation by the following command:

```bash

python general_benchmarks/HumanEval/eval_instruct_vllm.py --model_path <EVAL MODEL PATH> 

```

### MBPP
Perform MBPP evaluation by the following command:

```bash

python general_benchmarks/MBPP/eval_instruct_vllm.py --model_path <EVAL MODEL PATH> 

```

### MMLU & CMMLU

```bash

python general_benchmarks/MMLU/evaluator.py --task <mmlu or cmmlu> --lang <en or zh> --model_path <EVAL MODEL PATH> 

```