WBM filtering fails assert on entries_old_corr

[jackwebersdgr]

I'm attempting to compile the filtered WBM dataset in order to test a new model, but ran into this assert:

matbench-discovery/data/wbm/compile_wbm_test_set.py

Line 531 in c1f34da

assert len(entries_old_corr) == 76_390, f"{len(entries_old_corr)=}, expected 76,390"

assert len(entries_old_corr) == 76_390, f"{len(entries_old_corr)=}, expected 76,390"
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: len(entries_old_corr)=256963, expected 76,390

Is the fully filtered WBM dataset stored anywhere else or must it be computed on the fly using this script? Thanks!

[janosh]

strange, i reran that whole script just last month following #121 and it was back to a working state after #122. either way, no need for you to run that file yourself. the data files it creates are listed here, also up on figshare and will be auto-downloaded if you access the corresponding DataFiles attributes. e.g.

import pandas as pd

from matbench_discovery.data import DataFiles

df_summary = pd.read_csv(DataFiles.wbm_summary.path)
df_wbm_init_structs = pd.read_json(DataFiles.wbm_cses_plus_init_structs.path)

[jackwebersdgr]

Ah, I see. I was under the impression that this script would perform the filtration based on matching structure prototypes in MP, but it seems like this also results in ~257k datapoints. Is there a simple way to obtain the filtered 215.5k set, perhaps via some set of material_ids?

Additionally, it seems the documentation in the site is out of date, and can be updated with the above code https://matbench-discovery.materialsproject.org/contribute#--direct-download

[janosh]

have a look at the subset kwarg in load_df_wbm_with_preds

matbench-discovery/matbench_discovery/preds.py

Lines 90 to 111 in c1f34da

	def load_df_wbm_with_preds(
	*,
	models: Sequence[str] = (),
	pbar: bool = True,
	id_col: str = Key.mat_id,
	subset: pd.Index | Sequence[str] | Literal["uniq_protos"] | None = None,
	max_error_threshold: float | None = 5.0,
	**kwargs: Any,
	) -> pd.DataFrame:
	"""Load WBM summary dataframe with model predictions from disk.
	Args:
	models (Sequence[str], optional): Model names must be keys of
	matbench_discovery.data.Model. Defaults to all models.
	pbar (bool, optional): Whether to show progress bar. Defaults to True.
	id_col (str, optional): Column to set as df.index. Defaults to "material_id".
	subset (pd.Index | Sequence[str] | 'uniq_protos' | None, optional):
	Subset of material IDs to keep. Defaults to None, which loads all materials.
	'uniq_protos' drops WBM structures with matching prototype in MP
	training set and duplicate prototypes in WBM test set (keeping only the most
	stable structure per prototype). This increases the 'OOD-ness' of WBM.
	max_error_threshold (float, optional): Maximum absolute error between predicted

matbench-discovery/matbench_discovery/preds.py

Lines 180 to 182 in c1f34da

	if subset == "uniq_protos":
	df_out = df_out.query(Key.uniq_proto)
	elif subset is not None:

and

matbench-discovery/matbench_discovery/enums.py

Lines 141 to 148 in c1f34da

	@unique
	class TestSubset(LabelEnum):
	"""Which subset of the test data to use for evaluation."""
	uniq_protos = "uniq_protos", "Unique Structure Prototypes"
	ten_k_most_stable = "10k_most_stable", "10k Most Stable"
	full = "full", "Full Test Set"

[rydeveraumn]

@jackwebersdgr so this script does not produce the ~215k datapoints? The script crashes as the same point for me. Additionally, to clarify it is this reduced dataset that is being shown on the main page of matbench discovery, correct?

It also seems that the link for the WBM summary is not working. What I would like to do is download the summary and filter out by unique prototype so I can play around with the validation set locally.

Update
The figshare link worked and I can see all of the data from the summary ❤️.

@janosh

[janosh]

@jackwebersdgr so this script does not produce the ~215k datapoints?

the compile_wbm_test_set.py script calculates the structure prototypes using get_protostructure_label_from_spglib

matbench-discovery/data/wbm/compile_wbm_test_set.py

Line 611 in 93d1d5c

from aviary.wren.utils import get_protostructure_label_from_spglib

and stores them in df_summary[Key.wyckoff]. this is used to calculate the subset of unique prototypes that don't overlap with MP proptypes:

matbench-discovery/data/wbm/compile_wbm_test_set.py

Line 673 in 93d1d5c

df_summary[Key.uniq_proto] = ~(mask_proto_in_mp | mask_dupe_protos)

df_summary[Key.uniq_proto] is a boolean Series that is used across the code base to query for the 215k subset of WBM structures that are expected to relax to a novel and unique structure

[janosh]

closing this as completed. remaining issues in compile_wbm_test_set.py are tracked in #179