Repository navigation
New benchmark: QV-Select3 for matbench_steels (77.31 MPa) - #409
Open
arhancanli wants to merge 1 commit into
Open
arhancanli wants to merge 1 commit into
arhancanli wants to merge 1 commit into
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Benchmark submission: QV-Select3 on matbench_steels
Brief description of the algorithm. Composition-only regression. Features are the 14 elemental atomic fractions and
the matminer Magpie ElementProperty statistics. Seven fixed candidate models are considered: random forests on Magpie and
on fractions, extra trees on fractions + Magpie, gradient boosting on fractions, an ARD Gaussian process on fractions,
and the public TabPFN-v2 regression checkpoint used as an in-context learner (no fine-tuning) on fractions and on
fractions + Magpie. Inside each official outer training fold, 5-fold inner cross-validation on the training rows only
ranks the candidates; the prediction is the mean of the three best by inner-CV MAE, refitted on the whole training fold.
Each outer test fold is predicted once, and the model set was fixed before any outer-fold score was seen.
Result. matbench_steels MAE 77.31 MPa (std 10.81 over the 5 official folds), recorded with
MatbenchBenchmark.recordon the official folds. The per-fold inner-CV ranks are ininfo.json.Included files (only the submission folder,
benchmarks/matbench_v0.1_QV-Select3/):results.json.gz: the recorded benchmark.notebook.ipynb: the code that produced it (it was run as a script; the notebook holds the same code).info.json: authors, algorithm description, references, notes and pinned requirements.Label: new_benchmark (I cannot set labels on this repository; please add it).
The pipeline code and this submission were prepared with an AI coding agent (Claude, Anthropic) for Quantavane; I
checked the protocol (selection by inner CV only, no feedback from the outer test folds) and take responsibility for
the submission. CPU only.
Arhan Canli (Quantavane)