Repository navigation
Add "agentic" model type to submission metadata & leaderboard (similar to GIFT-Eval) #153
Description
Add an agentic value to submission-level model-type metadata so agentic forecasting systems can self-declare and be filtered on the leaderboard. Metadata only. No change to tasks, metrics, EvaluationWindow, or aggregation.
Motivation
Agentic systems (tool-using, multi-step pipelines that produce forecasts) are now a distinct submission class, not plain pretrained or fine-tuned models. GIFT-Eval added an Agentic model type for this reason, and it gets used: the Moirai 2.0 paper filters out agentic solutions to get a clean foundation-model comparison. Without a tag, agentic submissions get excluded informally or silently skew a board readers assume is model v model.
Approach
- Extend the submission-level
model_typefield with anagenticvalue, surfaced as a filterable leaderboard column. - Submission-level, not task-level, so it describes how the forecast was produced, orthogonal to the task taxonomy.
- Same scoring and statistical treatment as everything else. The tag only enables include or exclude, not a separate ranking.
Happy to send a PR for this.
Activity
shchur commentedon Jun 18, 2026
Hi @Jameswlepage, thanks for the suggestion. Agentic submissions are welcome, and I agree that we should make a separate category for them.
Currently this metadata is stored ad-hoc in the leaderboard code at https://huggingface.co/spaces/autogluon/fev-bench/blob/main/src/utils.py#L27-L83. Therefore it's probably sufficient to just specify the submission type in the PR - we can then update the metadata on HF accordingly.
One important detail: we should have clear instructions for authors of agentic submissions on how to disclose leakage on the benchmark tasks. I think a reasonable requirement will be "you should set
trained_on_this_dataset=Trueif any of the base models used in the agentic system have seen this dataset during training". For example, if authors use TimesFM-2.5 within their system, they need to mark all of these datasets as leakyfev/models/timesfm-2.5/model.py
Lines 11 to 21 in 7c9b78d
This puts more transparency requirements on the submissions - the authors need to disclose which models the system uses under the hood. Do you think it's a reasonable requirement?
shchur commentedon Oct 6, 2026
Added a new submission type
Systemwhich covers agentic systems. See https://github.com/autogluon/fev/tree/main/models#submit-results-to-the-fev-bench-leaderboard for more details.