fine_grained_analysis.py

[XuankeShi]

When running python fine_grained_analysis.py with mode=zh, only the reasoning and style results are output.I can successfully reproduce the results in the paper's tables for the EN mode, but I am unable to reproduce the ZH results.Could you please provide guidance on how to reproduce the ZH results as shown in the paper?

[XuankeShi]

When running python fine_grained_analysis.py with mode=zh, only the reasoning and style results are output.I can successfully reproduce the results in the paper's tables for the EN mode, but I am unable to reproduce the ZH results.Could you please provide guidance on how to reproduce the ZH results as shown in the paper?

The Chinese version of OneIG-Bench-ZH.csv lacks columns for prompt_length, type, and class, but the English version does, and the sample IDs in the Chinese and English versions correspond one-to-one. Could you please provide the correct information?

[XuankeShi]

Could you provide a method for calculating the overall table? I can currently obtain the individual sub-tables. How can I further refine this to obtain the overall table?

[OneIG-Bench]

Hi, XuankeShi,

The classification based on prompt length is determined by the number of English words in the prompts. Therefore, in OneIG-Bench-ZH, we did not simply map the original prompt length categories in a rough way; the same applies to the type category. As a result, when running fine_grained_analysis.py in "ZH" mode, only fine-grained analysis results related to reasoning and style can be obtained.

If needed, you can add relevant attributes from "OneIG-Bench.csv" to OneIG-Bench-ZH.csv and enable the corresponding categories (e.g., alignment, diversity, etc.) in fine_grained_analysis.py to obtain the corresponding fine-grained analysis results.

I’m not entirely sure what you mean by the overall table here. Do you mean combining the results from the sub-tables into a single table, or something else?

Best regards!
OneIG-Bench Team

[XuankeShi]

Hi, XuankeShi,

The classification based on prompt length is determined by the number of English words in the prompts. Therefore, in OneIG-Bench-ZH, we did not simply map the original prompt length categories in a rough way; the same applies to the type category. As a result, when running fine_grained_analysis.py in "ZH" mode, only fine-grained analysis results related to reasoning and style can be obtained.

If needed, you can add relevant attributes from "OneIG-Bench.csv" to OneIG-Bench-ZH.csv and enable the corresponding categories (e.g., alignment, diversity, etc.) in fine_grained_analysis.py to obtain the corresponding fine-grained analysis results.

I’m not entirely sure what you mean by the overall table here. Do you mean combining the results from the sub-tables into a single table, or something else?

Best regards! OneIG-Bench Team

Yes， I hope u can release the code about OneIG-Bench-ZH.
Overall table is this：
https://github.com/OneIG-Bench/OneIG-Benchmark/blob/main/assets/OneIG-Bench.png
Could please provide the convert code.

[OneIG-Bench]

Hi! We are so sorry that the fine-grained analysis code for the OneIG-Bench-ZH part may take some time to develop. If you're on a tight timeline, you can follow the instructions I mentioned in my previous reply to adapt it.

Also, are you referring to this table?
https://github.com/OneIG-Bench/OneIG-Benchmark/blob/main/assets/result.png

[XuankeShi]

Hi! We are so sorry that the fine-grained analysis code for the OneIG-Bench-ZH part may take some time to develop. If you're on a tight timeline, you can follow the instructions I mentioned in my previous reply to adapt it.

Also, are you referring to this table? https://github.com/OneIG-Bench/OneIG-Benchmark/blob/main/assets/result.png

yes. you are right