Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add predictions of open sourced MatterSim-V1 models #13

Merged
merged 4 commits into from
Dec 10, 2024

Conversation

yanghan234
Copy link

Dear developers,

Thank you for creating this benchmark set. It has been incredibly useful for both the computational materials science and AI for materials science communities. We appreciate the effort that has gone into its development.

I am submitting the predictions of the MatterSim-V1 models. Below are the key metrics obtained:

==> metrics-MatterSim-1M.txt <==
MODEL: MatterSim-V1
        mean SRME: 0.5546669270590349
        mean SRE: 0.34250088409755425

==> metrics-MatterSim-5M.txt <==
MODEL: MatterSim-V1
        mean SRME: 0.5745027442558724
        mean SRE: 0.41261004159193315

We noticed that a third-party pull request (#12) also submitted predictions for MatterSim-V1. After reviewing their contributions, we found consistent results between their submission and ours. We commend the authors of PR #12 for their valuable efforts and shared interest in MatterSim-V1.

We hope this contribution will further enhance the benchmark set, and we look forward to future opportunities for collaboration.

Best regards,
Han

@MSimoncelli
Copy link
Contributor

Dear Han,

Thank you for your kind words -- we are honoured that you and your team at Microsoft have found this benchmark useful.
We are happy to approve this PR, especially after seeing that your results are perfectly consistent which those provided by the third-party PR (#12).

Definitely this contribution enhances the benchmark set, and we are very interested in discussing possible collaborations.
Thank you again for your submission and your positive feedback. Please do not hesitate to reach out at michele.simoncelli@columbia.edu if you have any comment or questions.

Best wishes,
Michele

P.S. For completeness, let me mention that if you would like to have your results featured on Matbench Discovery, you should open a pull request including the energy-based discovery metrics here: https://github.com/janosh/matbench-discovery/pulls

@MSimoncelli MSimoncelli merged commit b97a32d into MPA2suite:main Dec 10, 2024
@yanghan234
Copy link
Author

Thank you very much Michele @MSimoncelli !

I truly appreciate your efforts in maintaining this benchmark, and it was a pleasure to contribute to it. Thank you also for the reminder to update matbench-discovery—we’ve just submitted a PR there as well.

Looking forward to future collaborations!

Best regards,
Han

@MSimoncelli
Copy link
Contributor

Thank you for your kind words and contributions @yanghan234.
Regarding the PR on matbench-discovery, it is currently under review and hopefully it will be approved soon.

Very much looking forward to future collaborations,
Michele

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants