Skip to content

New benchmark: QV-Select3 for matbench_steels (77.31 MPa) - #409

Open
arhancanli wants to merge 1 commit into
materialsproject:mainfrom
arhancanli:canli-2026-11-qv-select3
Open

arhancanli wants to merge 1 commit into
materialsproject:mainfrom
arhancanli:canli-2026-11-qv-select3

Conversation

@arhancanli

Copy link
Copy Markdown

Benchmark submission: QV-Select3 on matbench_steels

Brief description of the algorithm. Composition-only regression. Features are the 14 elemental atomic fractions and
the matminer Magpie ElementProperty statistics. Seven fixed candidate models are considered: random forests on Magpie and
on fractions, extra trees on fractions + Magpie, gradient boosting on fractions, an ARD Gaussian process on fractions,
and the public TabPFN-v2 regression checkpoint used as an in-context learner (no fine-tuning) on fractions and on
fractions + Magpie. Inside each official outer training fold, 5-fold inner cross-validation on the training rows only
ranks the candidates; the prediction is the mean of the three best by inner-CV MAE, refitted on the whole training fold.
Each outer test fold is predicted once, and the model set was fixed before any outer-fold score was seen.

Result. matbench_steels MAE 77.31 MPa (std 10.81 over the 5 official folds), recorded with
MatbenchBenchmark.record on the official folds. The per-fold inner-CV ranks are in info.json.

Included files (only the submission folder, benchmarks/matbench_v0.1_QV-Select3/):

  • results.json.gz: the recorded benchmark.
  • notebook.ipynb: the code that produced it (it was run as a script; the notebook holds the same code).
  • info.json: authors, algorithm description, references, notes and pinned requirements.

Label: new_benchmark (I cannot set labels on this repository; please add it).

The pipeline code and this submission were prepared with an AI coding agent (Claude, Anthropic) for Quantavane; I
checked the protocol (selection by inner CV only, no feedback from the outer test folds) and take responsibility for
the submission. CPU only.

Arhan Canli (Quantavane)

@github-actions github-actions Bot added the new_benchmark PR containing a new benchmark. May ONLY contain a new benchmark. label Nov 8, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

new_benchmark PR containing a new benchmark. May ONLY contain a new benchmark.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant