[bhaskargurram-ai]

New benchmark: DFT-Stack on the 4 matbench_v0.1 composition tasks

Label: new_benchmark (please add it; I cannot set labels on this repository)

External data disclosure (please read first)

This entry uses external computed data: Materials Project PBE DFT band gaps and formation energies, from
matminer's matbench_mp_gap and matbench_mp_e_form datasets, matched to each composition by reduced formula
and used as transfer-learning input features. No experimental external labels are used. On
matbench_expt_gap and matbench_expt_is_metal most compositions are matched; on matbench_steels and
matbench_glass almost none are (matbench_steels: 0/312 (0%); matbench_expt_gap: 3347/4604 (73%); matbench_expt_is_metal: 3613/4921 (73%); matbench_glass: 60/5680 (1%)). The entry is therefore not a like-for-like comparison with
composition-only entries that use no external data, and we would welcome it being flagged as such on the board.

Brief description of the algorithm

A cross-fitted, simplex-weighted stack of composition experts (Optuna-tuned LightGBM, XGBoost, CatBoost, extra
trees, kernel ridge / RBF SVM, k-NN, MLP; hurdle experts for the zero-inflated matbench_expt_gap) on
label-free matminer/CBFV composition descriptors plus the DFT block above. The DFT values are structure-derived
(each Materials Project entry is a DFT calculation on a crystal structure; per reduced formula the most stable
polymorph plus polymorph aggregates are used), so the DFT gap / metal flag are close proxies of the
matbench_expt_gap / matbench_expt_is_metal targets.

Fold use, stated exactly. Every choice is made inside the training rows of each official fold: the Optuna
hyperparameters, the simplex stack weights, the optional local residual correction and its gate (kept in 0 of
20 task-folds), the classification decision thresholds, and the coarse design choices (feature-block set;
for matbench_expt_gap the hurdle experts and the element-fraction extra-trees expert), which are selected per
fold from a small candidate set by the inner-CV score of the stack on the training rows (matbench_steels: 2 candidates; matbench_expt_gap: 12 candidates; matbench_expt_is_metal: 3 candidates; matbench_glass: 2 candidates). Selected per
fold: matbench_steels: fold 0 F_red|hurdle=off|et_frac=in, fold 1 F_red|hurdle=off|et_frac=in, fold 2 F_red|hurdle=off|et_frac=in, fold 3 F_red|hurdle=off|et_frac=in, fold 4 F_red|hurdle=off|et_frac=in; matbench_expt_gap: fold 0 F_full|hurdle=on|et_frac=in, fold 1 F_full|hurdle=on|et_frac=in, fold 2 F_full|hurdle=on|et_frac=out, fold 3 F_full|hurdle=on|et_frac=in, fold 4 F_full|hurdle=on|et_frac=in; matbench_expt_is_metal: fold 0 F_full|hurdle=off|et_frac=in, fold 1 F_full|hurdle=off|et_frac=in, fold 2 F_full|hurdle=off|et_frac=in, fold 3 F_full|hurdle=off|et_frac=in, fold 4 F_full|hurdle=off|et_frac=in; matbench_glass: fold 0 F_full|hurdle=off|et_frac=in, fold 1 F_full|hurdle=off|et_frac=in, fold 2 F_full|hurdle=off|et_frac=in, fold 3 F_full|hurdle=off|et_frac=in, fold 4 F_full|hurdle=off|et_frac=in. Disclosed as not fold-internal: the candidate set itself was constructed from the alternatives
explored on fold 0's training rows (the Optuna budget of 20 trials; the dropped experts (random forest, CrabNet, MODNet); restricting CatBoost to the F1 feature set; the fixed SVC C and gamma; and the composition of the candidate set itself), and a label-free, transductive median imputation of a
few hundred missing descriptor cells (details in info.json).
Earlier runs on the official folds, all reported: Amendment 1 (2026-10-18, no DFT block; matbench_steels 82.80 MAE; matbench_expt_gap 0.3240 MAE; matbench_expt_is_metal 0.9370 balanced accuracy; matbench_glass 0.8841 balanced accuracy) and
Amendment 2 (2026-10-19, the version first submitted in this PR, design choices fixed from fold 0's training rows;
matbench_steels 80.10 MAE; matbench_expt_gap 0.2352 MAE; matbench_expt_is_metal 0.9547 balanced accuracy; matbench_glass 0.8937 balanced accuracy). Classification tasks are recorded as hard labels at the inner-CV balanced-accuracy threshold
(pre-specified before scoring), so the board's ROC-AUC equals balanced accuracy; the probability ROC-AUC of the
same predictions is matbench_expt_is_metal 0.9895 ± 0.0008, matbench_glass 0.9606 ± 0.0080. The configuration was frozen by SHA-256 hash before the test folds were predicted
with it, and they were predicted and scored once.

Results (mean ± std over the 5 official folds, as computed by matbench, i.e. population std)

Task	Metric	Score (this entry, Amendment 3)	Amendment 2 (first submitted version)
matbench_steels	MAE	80.3217 ± 9.1998	80.10
matbench_expt_gap	MAE	0.2352 ± 0.0149	0.2352
matbench_expt_is_metal	ROC-AUC as computed by matbench from hard labels (= balanced accuracy)	0.9547 ± 0.0030	0.9547
matbench_glass	ROC-AUC as computed by matbench from hard labels (= balanced accuracy)	0.8937 ± 0.0141	0.8937

Included files (benchmarks/matbench_v0.1_DFTStack/)

• results.json.gz, info.json: required files (MatbenchBenchmark.from_file(...): is_valid=True,
  is_recorded=True for all 4 tasks).
• run_submission.py: source that reproduces the run (per-fold candidate selection included) with task.record
  calls.
• dftstack/: the pipeline, experts, feature blocks, DFT feature builder, frozen configurations and candidate
  selection (small .py files) imported by run_submission.py.
• README.md, requirements.txt.

Compute

CPU-only, no GPU. Selection and final predictions on one Intel(R) Core(TM) i7-14650HX workstation (16 cores / 24 threads, 5 workers x
4 threads); new expert cross-fits for the smaller feature-block candidates on the same workstation and on two cloud
CPU pods (AMD EPYC 9575F 64-Core Processor; 32 vCPU, 8 workers x 4 threads). Per-fold wall time, hosts, threads, CPU
models and per-expert fit seconds are recorded in the parameters of every fold in results.json.gz.
DETERMINISM CHECK: fold 0 of matbench_steels and of matbench_expt_gap were re-fitted from scratch on a Linux cloud CPU (AMD EPYC 9575F 64-Core Processor) with the same pinned package versions and compared with the Windows-workstation run (per-expert max |difference| of the test predictions): matbench_expt_gap: cat_F1 0.0e+00, et_F1 8.9e-16, et_F2 8.9e-16, hurdle_et_F1_exp 8.9e-16, hurdle_lgbm_F2_exp 1.7e-21, hurdle_lgbm_F2_median 0.0e+00, knn_frac 0.0e+00, krr_frac 3.9e-13, krr_magpie 5.2e-13, lgbm_F2_tuned 0.0e+00, mlp_F1 1.1e-14, xgb_F2 8.8e-01; re-tuned LightGBM parameters identical; matbench_steels: cat_F1 0.0e+00, et_F1 2.3e-13, et_F2 4.5e-13, et_frac 4.5e-13, knn_frac 0.0e+00, krr_frac 5.4e-11, krr_magpie 1.5e-11, lgbm_F2_tuned 7.8e-02, mlp_F1 1.8e-12, xgb_F2 6.6e+01; re-tuned LightGBM parameters different. Same-machine reruns are exactly reproducible (the reused caches and the from-scratch selection agree to 0.0 wherever the same candidate is selected); across CPU architectures XGBoost (tree_method=hist) and, on the small steels task, the Optuna-tuned LightGBM parameters are not bitwise reproducible, while every other expert agrees to 1e-9 or better.

Paper and code

Manuscript under review at Computational Materials Science (no DOI or preprint yet). Research repository: https://github.com/bhaskargurram-ai/matbench-composition-stack

[ardunn]

@bhaskargurram-ai Great! Would you be able to provide the compute used for each fold as per-fold metadata?

[ardunn]

@claude sweep

[claude]

Mild leakage (the fold-0 design is reused on folds 1–4) and DFT target proxies. The submitter must answer the questions; the maintainer must decide.

@ardunn, please review. I read all files in benchmarks/matbench_v0.1_DFTStack/ at 71935cc, except results.json.gz.

Important

• The frozen configuration came from fold 0's training rows. Those rows are the test rows of folds 1–4.
• The DFT gap and metal flag are close proxies for the matbench_expt_gap and matbench_expt_is_metal targets. The leaderboard must flag this.
• The per-fold compute you asked for on 2026-10-24 is not added yet. No commits came after the request.

CI: all checks pass. No merge conflicts.

Data leakage

• Fold use in the final run: OK. run_submission.py L45–47 uses get_train_and_val_data(fold) and get_test_data(fold, include_target=False). L49 asserts that train and test don't overlap. run_split2 gets only y_tr.
• Tuning, stacking, thresholds: OK. Optuna (pipeline2.py L45–71), the simplex stack weights (L120–126), the gated correction (L23–42), and the classification threshold (pipeline.py L282–291) all use inner folds of the training fold.
• New model per fold: OK. Expert2.fit builds from a factory (experts2.py L59–70). Each fold has its own cache folder.
• Design phase reused on folds 1–4: problem. frozen_config2.py L1–2 and design.py L28–31 say all design choices used "fold 0's TRAINING rows only". Fold 0's training set is the union of the test sets of folds 1–4, so about 80% of all test labels helped make choices that are then used on folds 1–4. Affected: BLOCKS2, hurdle=True and the et_frac drop for expt_gap, n_trials=20, the RF removal (experts2.py L147), CatBoost only on F1, and the fixed SVC C/gamma (L178–180). These choices are coarse, so the effect is probably small. But the rules don't allow test-fold-informed hyperparameters.
• Earlier rounds: can't tell. The code mentions "Amendment-1" and "Amendment-2". If Amendment 1 was scored on the official test folds before Amendment 2 (which adds the DFT block), then the method was chosen after seeing test results.
• Features from outside data: flag. external_dft.py L19–46 adds MP PBE dft_gap_stable, dft_metal_stable, gap statistics, dft_metal_frac, and dft_eform_stable. For expt_gap, the DFT gap is the proxy the rules name. For expt_is_metal, $\text{gap}_{\text{PBE}} \le 0$ is a direct proxy. These are also the targets of matbench_mp_gap and matbench_mp_e_form. Coverage is 73% on the two gap tasks and about 0–1% on steels and glass. info.json and the PR disclose this.
• Preprocessing on all rows: minor. data.py L90 does X.fillna(X.median()) on all rows, test rows included. data.py L89 and features_more.py L63 drop columns based on all rows. No labels are used, so the effect is very small. Can't tell whether CBFV generate_features (features_more.py L44) scales on all rows.
• No pretrained models or LLMs. CrabNet is off.

Protocol and validity

• Official task.folds, no re-split, all 5 folds of 4 tasks recorded (L43–60).
• Composition-only tasks: flag. The DFT lookup picks the "most stable polymorph" by MP structures (external_dft.py L27–39). This adds information that comes from structures. The disclosure says "DFT" but not "structure-derived".
• Classification records hard bool labels at the inner-CV threshold, so $\mathrm{AUC} = \tfrac{1}{2}(\mathrm{TPR} + \mathrm{TNR})$. This matches current v0.1 scoring.
• Algorithm name "DFT-Stack" is unique.

Reproducibility

• run_submission.py plus dftstack/ trains, predicts, and records every fold. No local paths.
• The design step can't be reproduced from this folder: design.py refers to src.design and the Amendment-1 run_split.
• All requirements are pinned, including matbench==0.6, which needs --no-deps on Python 3.12.
• Seeds are set (SEED=0, TPE seed, deterministic LightGBM). Variation between runs is not given.
• Compute: the notes give hardware, 14.3 worker-hours in total, and the median wall time per fold. But task.record (L59) records only pipeline and seed, with no per-fold compute. The code already measures t0 (L44) and meta["expert_seconds"], so these can go into params.
• results.json.gz user_metadata says XStack-A2, but L61 writes DFT-Stack.

Scientific soundness

• Method: unlabeled composition descriptors (Magpie and other presets, Meredig, Miedema, Yang, APE, Wen, TMetal, CBFV Oliynyk) plus the MP DFT block. Experts: LightGBM, XGBoost, CatBoost, ExtraTrees, KRR/SVC, k-NN, MLP, and hurdle experts for expt_gap. They are cross-fitted on 5 inner folds and combined with a simplex stack ($w_i \ge 0$, $\sum_i w_i = 1$).

  Task	DFT-Stack	Best now	Rank
  steels (MAE)	80.10 ± 9.12	TPOT-Mat 79.95	2
  expt_gap (MAE)	0.2352 ± 0.0149	Darwin 0.2865	1, 18% better
  expt_is_metal (ROC-AUC)	0.9547	Darwin 0.9598	2
  glass (ROC-AUC)	0.8937	MODNet 0.9603	2

• The large expt_gap gain is probably from the DFT-gap feature. Steels and glass have almost no DFT coverage, so they show the stack alone.

• Not supported: "The official test folds are never used for fitting or selection". This is true inside each fold, but not for the fold-0 design reused on folds 1–4.

Scope

• Only benchmarks/matbench_v0.1_DFTStack/ changes. No shell commands or unpinned downloads. pd.read_pickle reads only caches the code writes itself.

Questions for @bhaskargurram-ai

1. Can you make the frozen choices again inside each fold's training data? Or show that the results for folds 1–4 don't change?
2. Did you score Amendment 1, or any earlier version, on the official test folds before you decided on Amendment 2?
3. The notes call this "the labels variant". Which other variants exist? Did you choose this one before you saw test scores?
4. Please add per-fold compute (wall time, threads, expert seconds) to params in task.record, as @ardunn asked.
5. Is results_rerun.json.gz identical to results.json.gz?
6. Does CBFV generate_features fill or scale values using all rows?
7. Please state in info.json that the DFT values come from structures (MP polymorphs picked by formation energy).

[ardunn]

Hi @bhaskargurram-ai, thanks for the submission! Please work through the points raised in the Claude review above. If you think any part of that review is wrong, reply here and tag @ardunn specifically so I see it.

Once the review points are addressed (or we agree they don't apply) and CI is passing, we will merge it.

[bhaskargurram-ai]

@ardunn thank you for the detailed review; it was correct on every point. Rather than ask you to choose, I re-ran the entry with the fixed design choices made fold-internally ("Amendment 3": pre-registered, frozen by hash before any test prediction, scored once) and pushed the result.

1. Frozen choices reused on folds 1-4. Fixed. The choices frozen_config2.py had fixed from fold 0's training rows (feature-block set; for expt_gap the hurdle experts and the et_frac expert) are now a small candidate set, and the choice is made inside each training fold by the inner-CV score of the stack on the fold's training rows (the rule of the fold-internal Amendment 1). Block sets F_min (Magpie + fractions + extra descriptors), F_red (+ DFT block), F_full (+ remaining blocks); expt_gap also hurdle on/off and et_frac in/out (12 candidates); steels and glass (empty DFT block): F_red vs F_full. Selected: steels F_red in all 5 folds; glass, expt_is_metal and expt_gap F_full in all 5 (expt_gap with hurdle experts; et_frac in except fold 2, zero weight either way). So glass, expt_is_metal and expt_gap predictions are identical to the previous file (max |diff| 0.0); steels moves from 80.10 to 80.32 MAE (paired bootstrap +0.22, 95% CI [-1.26, +1.83], p = 0.80). The file was replaced although steels got marginally worse; that was the pre-registered rule. Still not fold-internal, disclosed in info.json: the candidate set itself comes from the fold-0 design alternatives (also Optuna budget, dropped experts, CatBoost on F1, SVC C/gamma) and the imputation in point 6.

2. Earlier scoring on the official folds. Yes, two prior runs, each pre-registered and scored once: Amendment 1 (2026-10-18, no DFT block: 82.80 / 0.3240 / 0.9370 / 0.8841) and Amendment 2 (2026-10-19, the version first submitted here: 80.10 / 0.2352 / 0.9547 / 0.8937). All three are in info.json.

3. Variants. Two, labels (submitted) and probabilities, written at the same scoring moment from the same predictions; recording labels was fixed before scoring. Probability ROC-AUC: expt_is_metal 0.9895, glass 0.9606.

4. Per-fold compute. In params of all 20 folds: wall_time_s, n_workers, threads_per_worker, cpu_threads_used, cpu_model, gpu, fit_hosts, expert_fit_seconds per expert, selected_candidate, candidate_inner_cv_scores. Amendment 3 took 2.2 h of job wall time on an i7-14650HX (5 x 4 threads) plus two 32-vCPU AMD EPYC 9575F cloud pods (8 x 4 threads, about 15 min each) for the new expert fits. F_full expert fits were reused from the previous run where the expert reads exactly the same columns. A determinism check backs that: fold 0 of steels and expt_gap re-fitted from scratch on the Linux pod reproduce the Windows run to <= 3e-9 for every expert except XGBoost (hist; max |diff| 65.9 MPa / 0.88 eV) and, on steels only, the Optuna-tuned LightGBM parameters (different best trial). Same-machine reruns are bit-identical.

5. results_rerun.json.gz. Only the output name of run_submission.py, which now reproduces Amendment 3 including the per-fold selection (dftstack/run_final3.py, frozen_config3.py). Not shipped; no complete independent rerun yet.

6. CBFV. Unchanged and disclosed: CBFV and the extra block fill missing cells with label-free column medians over all task rows (counts per task in info.json; steels none). Kept so the reused caches stay valid.

7. Structure-derived. info.json states that the MP values are DFT calculations on crystal structures (most stable polymorph per formula plus polymorph aggregates), "DFT, structure-derived, matched by formula", and that the DFT gap / metal flag are close proxies of the targets. Please flag the entry accordingly.

In this push (90316ea): new results.json.gz, dftstack/run_final3.py + frozen_config3.py, run_submission.py, info.json, README, PR description; algorithm is DFT-Stack. Plan Section 13, freeze hashes, candidate scores and the determinism report are in the research repository.