Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi, thank you for maintaining FEV-Bench.
This PR adds the Fracast-0 wrapper, requirements, and the complete 100-task fev-bench summary CSV. Fracast-0 is released as an 85K-parameter time-series foundation model under Apache-2.0, with weights at ztxtech/fracast-0.
The evaluation was run locally on Apple MPS with batch size 512 and the
w8weights:uv run python models/evaluate.py -m fracast-0 -b benchmarks/fev_bench/tasks.yaml -k '{"model_id":"ztxtech/fracast-0","weights":"w8","batch_size":512,"device":"mps"}'All 100 tasks completed without a task-level failure. The CSV contains 100 rows with no missing
SQLorMASE. The adapter has an explicit deterministic fallback for 572 of 235,039 sequence windows (0.243%) whose available context had fewer than 8 finite observations: it repeats the most recent finite value. This is recorded in the wrapper and avoids silently propagating NaNs from the sparse source contexts.trained_on_datasetscovers the completeautogluon/fev_datasetsconfiguration list used for Fracast-0 pretraining, and the CSV setstrained_on_this_dataset=True. Consequently, the leaderboard's leakage-imputation procedure replaces all Fracast-0 errors with Chronos-Bolt and gives an aggregate identical to that model. This submission is therefore an in-corpus result plus adapter, not a claim of an independent zero-shot FEV result.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.