[liuxiaotong15]

Benchmark submissions

Brief description of your algorithm

We test BERT on all tasks of Matbench.

Included files

-- benchmarks
---- matbench_v0.1_AlchemBERT
------ results.json.gz
------ readme.py              # please visit our repo to get all codes
------ info.json                 

[liuxiaotong15]

A note on the scope of this PR:

This PR was originally opened as an AlchemBERT benchmark submission. While
bringing the branch up to date and getting the submission checks to pass, it
also became necessary to include several compatibility and scoring fixes:

1. score_array() now preserves the original probability predictions for
   ROC-AUC while using separately thresholded labels for accuracy, balanced
   accuracy, and F1. This fixes the issue described in [MUST FIX] How does ROCAUC work in score_array()? #137 and follows up on
   the classification probability discussion in Error when recording group probability results to classification tasks #40.

2. The classification helpers and tests now support NumPy scalar types and
   NumPy 2. The removed np.float_ alias was replaced by dtype-category checks
   using np.issubdtype.

3. The coverage artifact workflow now uses
   actions/upload-artifact@v4 and actions/download-artifact@v4, because
   GitHub rejects the deprecated v3 actions.

I considered splitting the benchmark submission and core compatibility fixes
into separate PRs. However, the original standalone AlchemBERT branch no
longer passes against the repository's current dependency and GitHub Actions
environment. I have therefore kept the changes together in this PR so the
submission remains reproducible and all required checks can pass.

Current checks:

• test (3.9): passed
• benchmark_submission: passed
• code_coverage: passed

The ROC-AUC change affects future scoring and does not rewrite historical
leaderboard artifacts.

[claude]

I can't check for leakage because the training code is not in the PR. The submitter must add it and remove the core changes.

@ardunn, please review. This benchmark submission also changes core scoring code and CI workflows.

Important

• The PR has merge conflicts with main, and main already has its core changes.
• There is no training or prediction code. readme.py only prints a link to an outside repository.

Data leakage

• Training on get_train_and_val_data, prediction on include_target=False, fresh model per fold, preprocessing per fold, hyperparameter choice: can't tell for any of these. benchmarks/matbench_v0.1_AlchemBERT/readme.py L1-12 only prints the gitee URL.
• Text descriptions. info.json L4 describes "natural language to describe a material structure or chemical formula". If the text comes from Robocrystallographer or database text, it can contain the target or a proxy (e.g. "metallic", band-gap words). Please show the template for each task.
• LLM pretraining: can't tell. The BERT checkpoint is not named. The risk is low for bert-base (BooksCorpus/Wikipedia) and medium for SciBERT/MatSciBERT/MatBERT, whose papers report experimental band gaps, metallicity, glass formation, and yield strengths (the expt_*, glass, and steels targets).

Protocol and validity

• Folds, completeness, composition-only inputs, classification form: can't tell. CI checks is_valid, not is_complete. The PR says "all tasks".
• The stored scores come from the submitter's record call, and the leaderboard reads them. So this PR's scoring change has no effect on the AlchemBERT values.
• CI: all 4 checks pass, but they ran on the PR's own modified workflows.

Reproducibility

• Fails. No training, prediction, or mb.record(...) code. The gitee link has no commit hash. The models (25 GB) are "available on contact".
• requirements doesn't pin matbench, pymatgen, or the text-generation libraries. No seeds and no compute info. notes only says "refer to the README".

Scientific soundness

• Method: a fine-tuned BERT regressor/classifier on a simple text description. What is new: a small encoder-only LM. gptchem and Darwin use larger LLMs.
• No scores in the PR, and I can't open results.json.gz. The maintainer should compare per-task means with the leaderboard, especially for expt_*, glass, and steels.
• Can't check the claims against the ChemRxiv preprint (10.26434/chemrxiv-2024-r4dnl).

Scope

• Fails. It changes matbench/data_ops.py, matbench/tests/test_data_ops.py, .github/workflows/python-package.yml, .github/workflows/rebuild_website.yml, and requirements-dev.txt.
• The score_array and NumPy-type fix is correct, but main already has it: Fix ROC-AUC computed on thresholded predictions (#137, #181) #387 (88f4e80, fixes score_array computes roc-auc values on discretized predictions #181/[MUST FIX] How does ROCAUC work in score_array()? #137), cd06e5e, b73c1cf, and 3567fe8. main also has newer actions and pins. So "the standalone branch no longer passes CI" is no longer true.

Merge conflicts: yes. Fix: rebase on main, then git checkout origin/main -- matbench .github requirements-dev.txt. Keep only benchmarks/matbench_v0.1_AlchemBERT/.

Questions for @liuxiaotong15

• Please remove all changes outside the benchmark folder and rebase on main.
• Please add the full training and prediction code, including mb.record(...). Can you pin the gitee repo to a commit?
• Which BERT checkpoint did you use, and on what corpus was it pretrained?
• What is the text template for each task? Does any text come from a database or contain property words?
• For the expt_*, glass, and steels tasks, did the model get only the formula?
• How did you choose the hyperparameters?
• Which matbench version recorded the results? Are the classification predictions probabilities or bools?
• Which seeds, hardware, and training time did you use?
• Please post the mean score for each task.

[ardunn]

A note on the scope of this PR:

This PR was originally opened as an AlchemBERT benchmark submission. While bringing the branch up to date and getting the submission checks to pass, it also became necessary to include several compatibility and scoring fixes:

1. score_array() now preserves the original probability predictions for
   ROC-AUC while using separately thresholded labels for accuracy, balanced
   accuracy, and F1. This fixes the issue described in [MUST FIX] How does ROCAUC work in score_array()? #137 and follows up on
   the classification probability discussion in Error when recording group probability results to classification tasks #40.
2. The classification helpers and tests now support NumPy scalar types and
   NumPy 2. The removed np.float_ alias was replaced by dtype-category checks
   using np.issubdtype.
3. The coverage artifact workflow now uses
   actions/upload-artifact@v4 and actions/download-artifact@v4, because
   GitHub rejects the deprecated v3 actions.

I considered splitting the benchmark submission and core compatibility fixes into separate PRs. However, the original standalone AlchemBERT branch no longer passes against the repository's current dependency and GitHub Actions environment. I have therefore kept the changes together in this PR so the submission remains reproducible and all required checks can pass.

Current checks:

• test (3.9): passed
• benchmark_submission: passed
• code_coverage: passed

The ROC-AUC change affects future scoring and does not rewrite historical leaderboard artifacts.

these have been fixed separately. please change to only include your new benchmark

[ardunn]

Hi @liuxiaotong15, thanks for the submission! Please work through the points raised in the Claude review above. If you think any part of that review is wrong, reply here and tag @ardunn specifically so I see it.

Once the review points are addressed (or we agree they don't apply) and CI is passing, we will merge it.