NCrystal release 4.0.0

[tkittel]

• This milestone release primarily brings new supported user platforms and a huge restructuring of the environment in which NCrystal is developed and tested. In addition, it also brings many minor fixes, and an updated and improved infrastructure for NCrystal plugins.
• As the perhaps biggest user-visible feature, NCrystal now also supports usage on Windows! This support is NOT considered experimental, and Windows users can now install the "ncrystal" package via tools like pip and conda and expect the same features on both the command-line and in the Python/C/C++ APIs as Unix users have had in the past. Of course, there might be bugs or issues to iron out, so please help us test out the new release on Windows and file issues at https://github.com/mctools/ncrystal/issues as appropriate!
• As another user-visible feature, it is now possible to install optional NCrystal plugins via a simple "pip install", and have NCrystal automatically and seamlessly pick up and load the plugin. We are currently putting together a curated list of known plugins, and in the near future we will provide an overview of these on the Wiki. As a quick preview teaser, try to do "pip install ncrystal-plugin-WaterData" and notice how "nctool --browse" suddenly shows a large list of NCMAT data files for light and heavy water at different temperatures. These files were in the past deemed too large for the NCrystal standard data library, but can now be easily made, without the need for manual download.
• In addition to Windows support, the NCrystal code-base has also been updated to support a variety of other compilers and other dependencies. This notably includes support for the Intel OneAPI compilers, but also latest gcc and clang, C++20, Python 3.13, Numpy 2, and of course Microsoft VisualStudio. Drop support for Python 3.6 and 3.7.
• As part of the efforts to make NCrystal command-line tools easily work on Windows, the various commandline tools of NCrystal (nctool, ncrystal_cif2ncmat, ncrystal_hfg2ncmat, etc.), are now better integrated into the NCrystal Python API. As a result they can be invoked directly in the Python API (i.e. without the need for subprocess calls invoking a shell command), via the new NCrystal.cli Python module. For instance, NCrystal.cli.run('nctool','--dump','Al_sg225.ncmat') will have the same effect as invoking "nctool --dump Al_sg225.ncmat" at the command line. One notable difference is that the code will run in the same process, meaning that any virtual files registed in memory will also be available to these tools.
• NCrystal packaging was updated. Users should still simply install a package from PyPI or conda-forge named "ncrystal", but behind the scenes, the formerly monolithic package named "ncrystal" has been split into three packages: "ncrystal-core", "ncrystal-python", and a metapackage called "ncrystal". The ncrystal-core package contains everything needed for those only needing C/C++ APIs: The binary file library (named libNCrystal.so, libNCrystal.dylib, or NCrystal.dll+NCrystal.lib depending on your platform), the NCrystal C++/C header files, the ncrystal-config command, and the CMake configuration files. Included in ncrystal-core (actually embedded in the binary library) is also the standard data library of NCMAT files. The "ncrystal-python" package contains the NCrystal python module, and all the many command-line tools (except for ncrystal-config). Finally, the package named "ncrystal" is simply a meta-package which depends on both "ncrystal-core" and "ncrystal-python", and ensures that they will be installed in compatible versions.
• The ncrystal repository at https://github.com/mctools/ncrystal is now finally maintained like any other open source project, with good CI and testing coverage, ready to accept PRs, and so on. This actually marks a VERY large change in the NCrystal project since the code has until now, and for historical reasons, actually been developed and tested in an in-house repository at the European Spallation Source's DMSC, nested in a Unix-only build and testing framework (https://mctools.github.io/simplebuild/). The code was then exported to the ncrystal repository at github when ready for a new release, through the usage automated scripting which reformatted the code as appropriate. Although this was a useful mode of development in the early days of the project, it is needless to say that this has long been a burdensome way of working, precluding meaningful usage of pull requests and other open source workflows at GitHub. From now on, the ncrystal repo at GitHub is the true upstream source of NCrystal code, and it also includes a large number of tests which can be invoked via CTest or the simple-build-system.
• The Python and cmdline API should be mostly unchanged by this release, and likewise the C++ and C APIs are mostly the same, although a few container classes were modified to support Microsoft and Intel compilers. Notably, many header file locations were modified, but code sticking to including "NCrystal/NCrystal.hh" and "NCrystal/ncrystal.h" should be safe (it is henceforth recommended to only include those files, since the location of other files might change. The C++ interface of NCrystal4 is NOT ABI compatible with earlier versions of NCrystal, and will require recompilation of downstream code. Code only using the C-API in "NCrystal/ncrystal.h" would on the other hand be ABI compatible.
• The new "ncrystal" PyPI package now comes with optional dependencies listed. At the time of writing this is: "plot" which installs dependencies required by various plotting functions in the NCrystal cmdline tools or Python API (currently this is just matplotlib), "composer" which installs dependencies required for usage of the NCMATComposer (currently this is just spglib), "cif" which installs dependencies required for reading and processing CIF files (currently this is gemmi, spglib and ase), Finally, the "all" dependencies combines all of the above, so doing pip install "ncrystal[all]" should ensure that users will not encounter missing dependencies when using the NCrystal Python or cmdline APIs. Another category, "devel", is intended for NCrystal developers and CI pipelines, and adds additional packages needed for such work.
• Keeping in line with general trends at github, the default branch of the ncrystal repository is named main rather than master, from this point on. For a little while we will try to keep the master branch alive, in case of any hardwired usage of the branch name "master" somewhere, but this will not be the case for long. This change does not, however, change the fact that we will in general try to keep default branch (now: main), synchronised with the main release (apart possibly from minor documentation and CI fixes), and keep the "develop" branch as the main branch to target with pull requests between releases. NOTE: Any local clones of the ncrystal code repository should be updated to reflect this change. If there is no local interesting content in the clone, one option is to remove the local clone and get a fresh one from github. Otherwise, GitHub advices the following four commands to "repair" your local clones (experts only):

    git branch -m master main
    git fetch origin
    git branch -u origin/main main
    git remote set-head origin -a

• Added a new ncrystal_mcstasunion command-line script which is to be used instead of calling 'python -mNCrystal.mcstasutils --union'.
• The NCrystal-Geant4 bindings are moved out of the NCrystal repository and into https://github.com/mctools/ncrystal-geant4 . Note that at the time of this release, this migration is still work in progress and not complete yet. Anyone needing NCrystal and Geant4 are recommended to use NCrystal release 3.9.7 as a short term workaround.
• For more details, refer to the CHANGELOG.

---

This discussion was created from the release NCrystal release 4.0.0.