Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test nxdl hdf5 validator #280

Open
wants to merge 6 commits into
base: fairmat
Choose a base branch
from
Open

Test nxdl hdf5 validator #280

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
cbff1e1
Test nxdl.xml file for hdf5 validator.
RubelMozumder Aug 12, 2024
df93d00
nxdls for hdf5 validation test and include test nxdl path in nxdl_uti…
RubelMozumder Aug 13, 2024
40a2e8a
update nxdl files.
RubelMozumder Aug 13, 2024
6c98493
update tests regarding test nxdl files.
RubelMozumder Aug 14, 2024
3edba61
black.
RubelMozumder Aug 14, 2024
5ff1937
remove the print statement.
RubelMozumder Aug 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions dev_tools/tests/data/NXhdf5_validator_1.nxdl.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="nxdlformat.xsl" ?>
<definition xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://definition.nexusformat.org/nxdl/3.1 ../nxdl.xsd"
xmlns="http://definition.nexusformat.org/nxdl/3.1"
name="NXhdf5_validator_1"
extends="NXobject"
type="group"
category="application"
>
<doc>This is a dummy NXDL to test out the dataconverter.</doc>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should not mention the dataconverter, which is a pynxtools concept. Just say it is a dummy for testing validation.

<group type="NXentry">
<field name="hdf5_validator_1_program_name"/>
<field name="definition">
<doc>This is a dummy NXDL to test out the dataconverter.</doc>
<attribute name="version"/>
<enumeration>
<item value="hdf5_validator_1"/>
<item value="hdf5_VALIDATOR_1"/>
<item value="HDF5_VALIDATOR_1"/>
</enumeration>
</field>
<group type="NXdata" name="hdf5_validator_1_data" optional="true">
<attribute name="signal"/>
<attribute name="auxiliary_signals"/>
<attribute name="AXISNAME_indices" type="NX_NUMBER"/>
<attribute name="axes"/>
<field name="AXISNAME" units="NX_ANY"/>
<attribute name="long_name"/>
<field name="DATA" type="NX_NUMBER" units="NX_ANY"/>
</group>
<group type="NXnote" name="hdf5_validator_1_required" optional="true">
<doc>This is a required yet empty group.</doc>
<field name="required_field" type="NX_CHAR"/>
</group>
</group>
</definition>
39 changes: 39 additions & 0 deletions dev_tools/tests/data/NXhdf5_validator_2.nxdl.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="nxdlformat.xsl" ?>
<definition xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://definition.nexusformat.org/nxdl/3.1 ../nxdl.xsd"
xmlns="http://definition.nexusformat.org/nxdl/3.1"
name="NXhdf5_validator_2"
extends="NXhdf5_validator_1"
type="group"
category="application"
>
<doc>This is a dummy NXDL to test out the dataconverter.</doc>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see above

<group type="NXentry">
<field name="hdf5_validator_2_program_name"/>

<group type="NXnote" name="hdf5_validator_2_required">
<doc>This is a required yet empty group.</doc>
<field name="required_field" type="NX_CHAR"/>
</group>
<group type="NXuser" name="hdf5_validator_2_users_req">
<doc>This is a required yet empty group.</doc>
<field name="required_field" type="NX_CHAR"/>
</group>
<group name="experiment_result" type="NXdata">
<doc>
All experiment results data such as scattering angle (2theta),
intensity, incident angle, scattering vector, etc will be stored here.
</doc>
<field name="hdf5_validator_2_intensity" type="NX_FLOAT">
<doc>
Number of scattered electrons per unit time.
</doc>
<dimensions rank="2">
<dim index="1" value="nX"/>
<dim index="1" value="nY"/>
</dimensions>
</field>
</group>
</group>
</definition>
0 dev_tools/tests/NXtest.nxdl.xml → dev_tools/tests/data/NXtest.nxdl.xml
View file
Open in desktop
File renamed without changes.
11 changes: 11 additions & 0 deletions dev_tools/tests/test_docs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,11 @@
from ..globals.directories import get_xsd_file
from ..nxdl import iter_definitions

SKIP_FILES_FROM_TESTS = [
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get why you skipped these files here, but should the doc generation not just work on them if they are valid files?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, my mistake, I just skip the doc generation not all the tests, where some tests validate the nexus data structure.

The files are not presentable to the outer users. I think Doc generation would make this NXDL accessible to the public. As the files are reference app defs for nexus file validator, later I will add further nxdl to make the validator fail.

And the doc generating does not give validity to the files. NXDL validation performed test in dev_tools and validation function while writing a nexus file from the NeXus template.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"The files are not presentable to the outer users. I think Doc generation would make this NXDL accessible to the public." I don't understand this, but this is probably related to my comment above. The user should never be able to use an NXDL in dev_tools/tests/data, independent of the doc generation.

"NXhdf5_validator_2.nxdl.xml",
"NXhdf5_validator_1.nxdl.xml",
]


@pytest.fixture(scope="module")
def doc_generator():
Expand All @@ -28,16 +33,22 @@ def anchor_registry_write(tmpdir_factory):

@pytest.mark.parametrize("nxdl_file", list(iter_definitions()))
def test_nxdl_generate_doc(nxdl_file, doc_generator):
if nxdl_file.name in SKIP_FILES_FROM_TESTS:
pytest.skip(f"Skip {nxdl_file.name}")
assert doc_generator(nxdl_file)


@pytest.mark.parametrize("nxdl_file", list(iter_definitions()))
def test_nxdl_anchor_list(nxdl_file, doc_generator, anchor_registry):
if nxdl_file.name in SKIP_FILES_FROM_TESTS:
pytest.skip(f"Skip {nxdl_file.name}")
assert doc_generator(nxdl_file, anchor_registry=anchor_registry)


@pytest.mark.parametrize("nxdl_file", list(iter_definitions()))
def test_nxdl_anchor_write_list(nxdl_file, doc_generator, anchor_registry_write):
if nxdl_file.name in SKIP_FILES_FROM_TESTS:
pytest.skip(f"Skip {nxdl_file.name}")
assert doc_generator(nxdl_file, anchor_registry=anchor_registry_write)


Expand Down
2 changes: 1 addition & 1 deletion dev_tools/tests/test_nxdl_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ def test_get_nexus_classes_units_attributes():
def test_get_node_at_nxdl_path():
"""Test to verify if we receive the right XML element for a given NXDL path"""
local_dir = os.path.abspath(os.path.dirname(__file__))
nxdl_file_path = os.path.join(local_dir, "./NXtest.nxdl.xml")
nxdl_file_path = os.path.join(local_dir, "./data/NXtest.nxdl.xml")
elem = ET.parse(nxdl_file_path).getroot()
node = nexus.get_node_at_nxdl_path("/ENTRY/NXODD_name", elem=elem)
assert node.attrib["type"] == "NXdata"
Expand Down
23 changes: 15 additions & 8 deletions dev_tools/utils/nxdl_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,15 @@ def get_app_defs_names():
app_def_path_glob = nexus_def_path / "applications" / "*.nxdl*"

contrib_def_path_glob = Path(nexus_def_path) / "contributed_definitions" / "*.nxdl*"
test_def_path_glob = Path(nexus_def_path) / "dev_tools/tests/data" / "*.nxdl*"

files = sorted(glob(str(app_def_path_glob)))
for nexus_file in sorted(glob(str(contrib_def_path_glob))):
delte_files = sorted(glob(str(contrib_def_path_glob))) + sorted(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is not used at all

glob(str(test_def_path_glob))
)
for nexus_file in sorted(glob(str(contrib_def_path_glob))) + sorted(
glob(str(test_def_path_glob))
):
root = get_xml_root(nexus_file)
if root.attrib["category"] == "application":
files.append(nexus_file)
Expand Down Expand Up @@ -363,7 +369,12 @@ def find_definition_file(bc_name):
Note that it first checks in contributed and goes beyond only if no contributed found
"""
bc_filename = None
for nxdl_folder in ["contributed_definitions", "base_classes", "applications"]:
for nxdl_folder in [
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems a bit awkward to also search the "dev_tools/tests/data" folder here outside of the tests. Is it somehow possible to add this only in the test stage?

"contributed_definitions",
"base_classes",
"applications",
"dev_tools/tests/data",
]:
nxdl_file = nexus_def_path / nxdl_folder / f"{bc_name}.nxdl.xml"
if nxdl_file.exists():
bc_filename = nexus_def_path / nxdl_folder / f"{bc_name}.nxdl.xml"
Expand Down Expand Up @@ -517,9 +528,7 @@ def check_attr_name_nxdl(param):
return logger, elem, nxdl_path, doc, attr, req_str


def try_find_default(
logger, orig_elem, elem, nxdl_path, doc, attr
): # pylint: disable=too-many-arguments
def try_find_default(logger, orig_elem, elem, nxdl_path, doc, attr): # pylint: disable=too-many-arguments
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what changed here?

"""Try to find if default is defined as a child of the NXDL element"""
if elem is not None:
if doc:
Expand All @@ -539,9 +548,7 @@ def try_find_default(
return logger, elem, nxdl_path, doc, attr


def other_attrs(
logger, orig_elem, elem, nxdl_path, doc, attr
): # pylint: disable=too-many-arguments
def other_attrs(logger, orig_elem, elem, nxdl_path, doc, attr): # pylint: disable=too-many-arguments
"""Handle remaining attributes"""
if elem is not None:
if doc:
Expand Down
Loading