Skip to content

[Fix] MM-IFEval: import rule-based verify functions into module globals - #1601

Open
cdpark92 wants to merge 1 commit into
open-compass:mainfrom
cdpark92:fix-mmifeval
Open

[Fix] MM-IFEval: import rule-based verify functions into module globals#1601
cdpark92 wants to merge 1 commit into
open-compass:mainfrom
cdpark92:fix-mmifeval

Conversation

@cdpark92

Copy link
Copy Markdown

Bug

vlmeval/dataset/mmifeval.py dispatches rule-based constraint verifiers by name:

for func_dict in constraint["judge"]["verify_funcs"]:
    func = globals()[func_dict["func"]]        # ~line 300

The verify functions (check_whether_response_word_count_in_range,
check_whether_response_paragraph_number_in_range, ...) are defined in
vlmeval/dataset/utils/mmif/function_and_compare.py but are never imported into
mmifeval's module namespace, so the globals() lookup raises
KeyError: 'check_whether_response_word_count_in_range' for every item that
contains a rule-based constraint. The lookup is outside the surrounding
try/except blocks (those only wrap the GPT-judged branches), so MM-IFEval
evaluation cannot complete on any prediction set that includes rule-based
constraints.

Fix

Import the verify functions into the module globals (+5 lines including the
explanatory comment):

from .utils.mmif.function_and_compare import *  # noqa: F401,F403

Verification

  • Before: MM-IFEval evaluation fails with the KeyError above on the first
    rule-based item.
  • After: full MM-IFEval evaluation completes; rule-based constraints
    (word-count ranges, paragraph counts, punctuation rules, ...) score correctly.
    Verified end-to-end on local runs with sub-1B open VLMs (e.g.
    LiquidAI/LFM2.5-VL-450M).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant