From be6cbd78db5eaec0cba8bb5ef57f1e9ee8271870 Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Sat, 9 Apr 2022 19:08:56 +0200 Subject: [PATCH 1/7] [INFRA] fix spec-internal links from schema entries add test try new method REPLACEME -> PATH_TO_SRC remove first attempt add relpaths make_columns_table use PATH_TO_SRC in columns.yaml html -> md add relpath arg for glossary fix tests add relpath for suffix tables add relpath for metadata add tests run black PATH_TO_SRC -> SPEC_ROOT fix param ordering consistency '..'|'.' -> page.file relpath -> page_file detect relpath from page.file fix docstrings fix tests run black fix PDF build MNT: Restore sources Revert test_render changes --- macros_doc.md | 2 +- pdf_build_src/process_markdowns.py | 16 +++++++ src/schema/objects/columns.yaml | 10 +++-- src/schema/objects/metadata.yaml | 62 +++++++++++++-------------- src/schema/objects/suffixes.yaml | 6 +-- tools/mkdocs_macros_bids/macros.py | 28 ++++++++---- tools/schemacode/schemacode/render.py | 36 ++++++++++++++-- 7 files changed, 109 insertions(+), 51 deletions(-) diff --git a/macros_doc.md b/macros_doc.md index c202891717..cc238c6781 100644 --- a/macros_doc.md +++ b/macros_doc.md @@ -78,7 +78,7 @@ consistency. ## What macros are available? All the macros we use are in listed in this -[python file](https://github.com/bids-standard/bids-specification/blob/master/tools/mkdocs_macros_bids/macros.py). +[Python file](https://github.com/bids-standard/bids-specification/blob/master/tools/mkdocs_macros_bids/macros.py). | Name | Purpose | Uses schema | Example | | ----------------------- | -------------------------------------------------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | diff --git a/pdf_build_src/process_markdowns.py b/pdf_build_src/process_markdowns.py index 31bf6eb746..4f594b2927 100644 --- a/pdf_build_src/process_markdowns.py +++ b/pdf_build_src/process_markdowns.py @@ -562,6 +562,13 @@ def edit_titlepage(): file.writelines(data) +class MockPage: + pass + +class MockFile: + pass + + def process_macros(duplicated_src_dir_path): """Search for mkdocs macros in the specification, run the embedded functions, and replace the macros with their outputs. @@ -594,6 +601,15 @@ def process_macros(duplicated_src_dir_path): with open(filename, "r") as fo: contents = fo.read() + # Create a mock MkDocs Page object that has a "file" attribute, + # which is a mock MkDocs File object with a str "src_path" attribute + # The src_path + mock_file = MockFile() + mock_file.src_path = os.sep.join(filename.split(os.sep)[1::]) + + page = MockPage() + page.file = mock_file + # Replace code snippets in the text with their outputs matches = re.findall(re_code_snippets, contents) for m in matches: diff --git a/src/schema/objects/columns.yaml b/src/schema/objects/columns.yaml index 36e93fe308..350451e851 100644 --- a/src/schema/objects/columns.yaml +++ b/src/schema/objects/columns.yaml @@ -3,7 +3,7 @@ HED: name: HED description: | Hierarchical Event Descriptor (HED) Tag. - See [Appendix III](/99-appendices/03-hed.html) for details. + See [Appendix III](SPEC_ROOT/99-appendices/03-hed.md) for details. type: string abbreviation: name: abbreviation @@ -16,14 +16,16 @@ acq_time__scans: Acquisition time refers to when the first data point in each run was acquired. Furthermore, if this header is provided, the acquisition times of all files from the same recording MUST be identical. - Datetime format and their anonymization are described in [Units](/02-common-principles.html#units). + Datetime format and their anonymization are described in + [Units](SPEC_ROOT/02-common-principles.md#units). type: string format: datetime acq_time__sessions: name: acq_time description: | Acquisition time refers to when the first data point of the first run was acquired. - Datetime format and their anonymization are described in [Units](/02-common-principles.html#units). + Datetime format and their anonymization are described in + [Units](SPEC_ROOT/02-common-principles.md#units). type: string format: datetime age: @@ -512,7 +514,7 @@ units: description: | Physical unit of the value represented in this channel, for example, `V` for Volt, or `fT/cm` for femto Tesla per centimeter - (see [Units](/02-common-principles.html#units)). + (see [Units](SPEC_ROOT/02-common-principles.md#units)). type: string format: unit value: diff --git a/src/schema/objects/metadata.yaml b/src/schema/objects/metadata.yaml index 7f584f9760..e9f49e649f 100644 --- a/src/schema/objects/metadata.yaml +++ b/src/schema/objects/metadata.yaml @@ -61,7 +61,7 @@ AnatomicalLandmarkCoordinateSystem: name: AnatomicalLandmarkCoordinateSystem description: | Defines the coordinate system for the anatomical landmarks. - See [Appendix VIII](/99-appendices/08-coordinate-systems.html) + See [Appendix VIII](SPEC_ROOT/99-appendices/08-coordinate-systems.md) for a list of restricted keywords for coordinate systems. If `"Other"`, provide definition of the coordinate system in `"AnatomicalLandmarkCoordinateSystemDescription"`. @@ -262,7 +262,7 @@ BodyPartDetails: BodyPartDetailsOntology: name: BodyPartDetailsOntology description: | - [URI](/02-common-principles.html#uniform-resource-indicator) of ontology used for + [URI](SPEC_ROOT/02-common-principles.md#uniform-resource-indicator) of ontology used for BodyPartDetails (for example: `"https://www.ebi.ac.uk/ols/ontologies/uberon"`). type: string format: uri @@ -378,7 +378,7 @@ ChunkTransformationMatrixAxis: Code: name: Code description: | - [URI](/02-common-principles.html#uniform-resource-indicator) + [URI](SPEC_ROOT/02-common-principles.md#uniform-resource-indicator) of the code used to present the stimuli. Persistent identifiers such as DOIs are preferred. If multiple versions of code may be hosted at the same location, @@ -388,7 +388,7 @@ Code: CogAtlasID: name: CogAtlasID description: | - [URI](/02-common-principles.html#uniform-resource-indicator) + [URI](SPEC_ROOT/02-common-principles.md#uniform-resource-indicator) of the corresponding [Cognitive Atlas](https://www.cognitiveatlas.org/) Task term. type: string @@ -396,7 +396,7 @@ CogAtlasID: CogPOID: name: CogPOID description: | - [URI](/02-common-principles.html#uniform-resource-indicator) + [URI](SPEC_ROOT/02-common-principles.md#uniform-resource-indicator) of the corresponding [CogPO](http://www.cogpo.org/) term. type: string format: uri @@ -446,9 +446,9 @@ DatasetDOI: description: | The Digital Object Identifier of the dataset (not the corresponding paper). DOIs SHOULD be expressed as a valid - [URI](/02-common-principles.html#uniform-resource-indicator); + [URI](SPEC_ROOT/02-common-principles.md#uniform-resource-indicator); bare DOIs such as `10.0.2.3/dfjj.10` are - [DEPRECATED](/02-common-principles.html#definitions). + [DEPRECATED](SPEC_ROOT/02-common-principles.md#definitions). type: string format: uri DatasetType: @@ -547,7 +547,7 @@ DigitizedHeadPointsCoordinateSystem: description: | Defines the coordinate system for the digitized head points. See - [Appendix VIII](/99-appendices/08-coordinate-systems.html) + [Appendix VIII](SPEC_ROOT/99-appendices/08-coordinate-systems.md) for a list of restricted keywords for coordinate systems. If `"Other"`, provide definition of the coordinate system in `"DigitizedHeadPointsCoordinateSystemDescription"`. @@ -637,7 +637,7 @@ EEGCoordinateSystem: Defines the coordinate system for the EEG sensors. See - [Appendix VIII](/99-appendices/08-coordinate-systems.html) + [Appendix VIII](SPEC_ROOT/99-appendices/08-coordinate-systems.md) for a list of restricted keywords for coordinate systems. If `"Other"`, provide definition of the coordinate system in `EEGCoordinateSystemDescription`. @@ -706,13 +706,13 @@ EchoTime: (please note that the DICOM term is in milliseconds not seconds). The data type number may apply to files from any MRI modality concerned with a single value for this field, or to the files in a - [file collection](/99-appendices/10-file-collections.html) + [file collection](SPEC_ROOT/99-appendices/10-file-collections.md) where the value of this field is iterated using the - [echo entity](/99-appendices/09-entities.html#echo). + [echo entity](SPEC_ROOT/99-appendices/09-entities.md#echo). The data type array provides a value for each volume in a 4D dataset and should only be used when the volume timing is critical for interpretation of the data, such as in - [ASL](/04-modality-specific-files/01-magnetic-resonance-imaging-data.html#\ + [ASL](SPEC_ROOT/04-modality-specific-files/01-magnetic-resonance-imaging-data.md#\ arterial-spin-labeling-perfusion-data) or variable echo time fMRI sequences. anyOf: @@ -823,7 +823,7 @@ FiducialsCoordinateSystem: Defines the coordinate system for the fiducials. Preferably the same as the `"EEGCoordinateSystem"`. See - [Appendix VIII](/99-appendices/08-coordinate-systems.html) + [Appendix VIII](SPEC_ROOT/99-appendices/08-coordinate-systems.md) for a list of restricted keywords for coordinate systems. If `"Other"`, provide definition of the coordinate system in `"FiducialsCoordinateSystemDescription"`. @@ -881,13 +881,13 @@ FlipAngle: Corresponds to: DICOM Tag 0018, 1314 `Flip Angle`. The data type number may apply to files from any MRI modality concerned with a single value for this field, or to the files in a - [file collection](/99-appendices/10-file-collections.html) + [file collection](SPEC_ROOT/99-appendices/10-file-collections.md) where the value of this field is iterated using the - [flip entity](/99-appendices/09-entities.html#flip). + [flip entity](SPEC_ROOT/99-appendices/09-entities.md#flip). The data type array provides a value for each volume in a 4D dataset and should only be used when the volume timing is critical for interpretation of the data, such as in - [ASL](/04-modality-specific-files/01-magnetic-resonance-imaging-data.html#\ + [ASL](SPEC_ROOT/04-modality-specific-files/01-magnetic-resonance-imaging-data.md#\ arterial-spin-labeling-perfusion-data) or variable flip angle fMRI sequences. anyOf: @@ -968,14 +968,14 @@ GeneticLevel: Genetics.Database: name: Genetics.Database description: | - [URI](/02-common-principles.html#uniform-resource-indicator) + [URI](SPEC_ROOT/02-common-principles.md#uniform-resource-indicator) of database where the dataset is hosted. type: string format: uri Genetics.Dataset: name: Genetics.Dataset description: | - [URI](/02-common-principles.html#uniform-resource-indicator) + [URI](SPEC_ROOT/02-common-principles.md#uniform-resource-indicator) where data can be retrieved. type: string format: uri @@ -984,7 +984,7 @@ Genetics.Descriptors: description: | List of relevant descriptors (for example, journal articles) for dataset using a valid - [URI](/02-common-principles.html#uniform-resource-indicator) + [URI](SPEC_ROOT/02-common-principles.md#uniform-resource-indicator) when possible. anyOf: - type: string @@ -1015,7 +1015,7 @@ HED: name: HED description: | Hierarchical Event Descriptor (HED) information, - see: [Appendix III](/99-appendices/03-hed.html) for details. + see: [Appendix III](SPEC_ROOT/99-appendices/03-hed.md) for details. anyOf: - type: string - type: object @@ -1067,7 +1067,7 @@ HeadCoilCoordinateSystem: description: | Defines the coordinate system for the head coils. See - [Appendix VIII](/99-appendices/08-coordinate-systems.html) + [Appendix VIII](SPEC_ROOT/99-appendices/08-coordinate-systems.md) for a list of restricted keywords for coordinate systems. If `"Other"`, provide definition of the coordinate system in `HeadCoilCoordinateSystemDescription`. @@ -1139,7 +1139,7 @@ ImageAcquisitionProtocol: name: ImageAcquisitionProtocol description: | Description of the image acquisition protocol or - [URI](/02-common-principles.html#uniform-resource-indicator) + [URI](SPEC_ROOT/02-common-principles.md#uniform-resource-indicator) (for example from [protocols.io](https://www.protocols.io/)). type: string ImageDecayCorrected: @@ -1440,7 +1440,7 @@ License: description: | The license for the dataset. The use of license name abbreviations is RECOMMENDED for specifying a license - (see [Appendix II](/99-appendices/02-licenses.html)). + (see [Appendix II](SPEC_ROOT/99-appendices/02-licenses.md)). The corresponding full license text MAY be specified in an additional `LICENSE` file. type: string @@ -1487,7 +1487,7 @@ MEGCoordinateSystem: name: MEGCoordinateSystem description: | Defines the coordinate system for the MEG sensors. - See [Appendix VIII](/99-appendices/08-coordinate-systems.html) + See [Appendix VIII](SPEC_ROOT/99-appendices/08-coordinate-systems.md) for a list of restricted keywords for coordinate systems. If `"Other"`, provide definition of the coordinate system in `"MEGCoordinateSystemDescription"`. @@ -2119,7 +2119,7 @@ ReferencesAndLinks: description: | List of references to publications that contain information on the dataset. A reference may be textual or a - [URI](/02-common-principles.html#uniform-resource-indicator). + [URI](SPEC_ROOT/02-common-principles.md#uniform-resource-indicator). items: type: string type: array @@ -2175,7 +2175,7 @@ RepetitionTimePreparation: The data type array provides a value for each volume in a 4D dataset and should only be used when the volume timing is critical for interpretation of the data, such as in - [ASL](/04-modality-specific-files/01-magnetic-resonance-imaging-data.html\ + [ASL](SPEC_ROOT/04-modality-specific-files/01-magnetic-resonance-imaging-data.md\ #arterial-spin-labeling-perfusion-data). anyOf: - type: number @@ -2228,7 +2228,7 @@ SampleExtractionProtocol: name: SampleExtractionProtocol description: | Description of the sample extraction protocol or - [URI](/02-common-principles.html#uniform-resource-indicator) + [URI](SPEC_ROOT/02-common-principles.md#uniform-resource-indicator) (for example from [protocols.io](https://www.protocols.io/)). type: string SampleFixation: @@ -2313,7 +2313,7 @@ ScanDate: description: | Date of scan in the format `"YYYY-MM-DD[Z]"`. This field is DEPRECATED, and this metadata SHOULD be recorded in the `acq_time` column of the - corresponding [Scans file](/03-modality-agnostic-files.html#scans-file). + corresponding [Scans file](SPEC_ROOT/03-modality-agnostic-files.md#scans-file). type: string format: date ScanOptions: @@ -2477,7 +2477,7 @@ SourceDatasets: description: | Used to specify the locations and relevant attributes of all source datasets. Valid keys in each object include `"URL"`, `"DOI"` (see - [URI](/02-common-principles.html#uniform-resource-indicator)), and + [URI](SPEC_ROOT/02-common-principles.md#uniform-resource-indicator)), and `"Version"` with [string](https://www.w3schools.com/js/js_json_datatypes.asp) values. @@ -2775,7 +2775,7 @@ Units: description: | Measurement units for the associated file. SI units in CMIXF formatting are RECOMMENDED - (see [Units](/02-common-principles.html#units)). + (see [Units](SPEC_ROOT/02-common-principles.md#units)). type: string format: unit VascularCrushing: @@ -2934,7 +2934,7 @@ iEEGCoordinateSystem: description: | Defines the coordinate system for the iEEG sensors. See - [Appendix VIII](/99-appendices/08-coordinate-systems.html) + [Appendix VIII](SPEC_ROOT/99-appendices/08-coordinate-systems.md) for a list of restricted keywords for coordinate systems. If `"Other"`, provide definition of the coordinate system in `iEEGCoordinateSystemDescription`. diff --git a/src/schema/objects/suffixes.yaml b/src/schema/objects/suffixes.yaml index 25ed8a5b8c..d9547aa795 100644 --- a/src/schema/objects/suffixes.yaml +++ b/src/schema/objects/suffixes.yaml @@ -463,7 +463,7 @@ blood: name: Blood recording data description: | Blood measurements of radioactivity stored in - [tabular files](/02-common-principles.html#tabular-files) + [tabular files](SPEC_ROOT/02-common-principles.md#tabular-files) and located in the `pet/` directory along with the corresponding PET data. bold: name: Blood-Oxygen-Level Dependent image @@ -579,11 +579,11 @@ pet: phase: name: Phase image description: | - [DEPRECATED](/02-common-principles.html#definitions). + [DEPRECATED](SPEC_ROOT/02-common-principles.md#definitions). Phase information associated with magnitude information stored in BOLD contrast. This suffix should be replaced by the - [`part-phase`](/99-appendices/09-entities.html#part) + [`part-phase`](SPEC_ROOT/99-appendices/09-entities.md#part) in conjunction with the `bold` suffix. anyOf: - unit: arbitrary diff --git a/tools/mkdocs_macros_bids/macros.py b/tools/mkdocs_macros_bids/macros.py index d27aef4f6d..dba9bc03ce 100644 --- a/tools/mkdocs_macros_bids/macros.py +++ b/tools/mkdocs_macros_bids/macros.py @@ -71,9 +71,14 @@ def make_entity_definitions(): return text -def make_glossary(): +def make_glossary(page_file=None): """Generate glossary. + Parameters + ---------- + page_file : MkDocs File object | None + The file where this macro is called, provided by the "page.file" variable. + Returns ------- text : str @@ -82,17 +87,20 @@ def make_glossary(): """ schemapath = utils.get_schema_path() schema_obj = schema.load_schema(schemapath) - text = render.make_glossary(schema_obj) + text = render.make_glossary(schema_obj, page_file=page_file) return text -def make_suffix_table(suffixes): +def make_suffix_table(suffixes, page_file=None): """Generate a markdown table of suffix information. Parameters ---------- suffixes : list of str A list of the suffixes to include in the table. + page_file : MkDocs File object | None + The file where this macro is called, provided by the "page.file" variable. + Returns ------- @@ -102,11 +110,11 @@ def make_suffix_table(suffixes): """ schemapath = utils.get_schema_path() schema_obj = schema.load_schema(schemapath) - table = render.make_suffix_table(schema_obj, suffixes) + table = render.make_suffix_table(schema_obj, suffixes, page_file=page_file) return table -def make_metadata_table(field_info): +def make_metadata_table(field_info, page_file=None): """Generate a markdown table of metadata field information. Parameters @@ -118,6 +126,8 @@ def make_metadata_table(field_info): Until requirement levels can be codified in the schema, this argument will be dictionary, with the field names as keys and the requirement levels as values. + page_file : MkDocs File object | None + The file where this macro is called, provided by the "page.file" variable. Returns ------- @@ -127,11 +137,11 @@ def make_metadata_table(field_info): """ schemapath = utils.get_schema_path() schema_obj = schema.load_schema(schemapath) - table = render.make_metadata_table(schema_obj, field_info) + table = render.make_metadata_table(schema_obj, field_info, page_file=page_file) return table -def make_columns_table(column_info): +def make_columns_table(column_info, page_file=None): """Generate a markdown table of TSV column information. Parameters @@ -143,6 +153,8 @@ def make_columns_table(column_info): Until requirement levels can be codified in the schema, this argument will be a dictionary, with the column names as keys and the requirement levels as values. + page_file : MkDocs File object | None + The file where this macro is called, provided by the "page.file" variable. Returns ------- @@ -152,7 +164,7 @@ def make_columns_table(column_info): """ schemapath = utils.get_schema_path() schema_obj = schema.load_schema(schemapath) - table = render.make_columns_table(schema_obj, column_info) + table = render.make_columns_table(schema_obj, column_info, page_file=page_file) return table diff --git a/tools/schemacode/schemacode/render.py b/tools/schemacode/schemacode/render.py index c1b8872142..72d1e0ee0a 100644 --- a/tools/schemacode/schemacode/render.py +++ b/tools/schemacode/schemacode/render.py @@ -56,7 +56,7 @@ def make_entity_definitions(schema): return text -def make_glossary(schema): +def make_glossary(schema, page_file=None): """Generate glossary. Parameters @@ -64,6 +64,8 @@ def make_glossary(schema): schema : dict The schema object, which is a dictionary with nested dictionaries and lists stored within it. + page_file : MkDocs File object | None + The file where this macro is called, provided by the "page.file" variable. Returns ------- @@ -119,6 +121,11 @@ def make_glossary(schema): obj_desc = obj_desc.replace("\n\n", "
") # Otherwise a newline corresponds to a space obj_desc = obj_desc.replace("\n", " ") + # Spec internal links need to be replaced + if page_file is not None: + relpath = os.sep.join([".."] * page_file.src_path.count(os.sep)) + relpath = "." if not relpath else relpath + obj_desc = obj_desc.replace("SPEC_ROOT", relpath) text += f'\n' text += f"\n## {obj_key}\n\n" @@ -403,13 +410,15 @@ def _remove_numeric_suffixes(string): return table_str -def make_suffix_table(schema, suffixes, tablefmt="github"): +def make_suffix_table(schema, suffixes, page_file=None, tablefmt="github"): """Produce suffix table (markdown) based on requested suffixes. Parameters ---------- schema : dict suffixes : list of str + page_file : MkDocs File object | None + The file where this macro is called, provided by the "page.file" variable. tablefmt : str Returns @@ -440,6 +449,11 @@ def make_suffix_table(schema, suffixes, tablefmt="github"): description = description.replace("\n\n", "
") # Otherwise a newline corresponds to a space description = description.replace("\n", " ") + # Spec internal links need to be replaced + if page_file is not None: + relpath = os.sep.join([".."] * page_file.src_path.count(os.sep)) + relpath = "." if not relpath else relpath + description = description.replace("SPEC_ROOT", relpath) df.loc[suffix] = [suffix_info["name"], description] @@ -452,7 +466,7 @@ def make_suffix_table(schema, suffixes, tablefmt="github"): return table_str -def make_metadata_table(schema, field_info, tablefmt="github"): +def make_metadata_table(schema, field_info, page_file=None, tablefmt="github"): """Produce metadata table (markdown) based on requested fields. Parameters @@ -468,6 +482,8 @@ def make_metadata_table(schema, field_info, tablefmt="github"): and the second string is additional table-specific information about the metadata field that will be appended to the field's base definition from the schema. + page_file : MkDocs File object | None + The file where this macro is called, provided by the "page.file" variable. tablefmt : string, optional The target table format. The default is "github" (GitHub format). @@ -519,6 +535,11 @@ def make_metadata_table(schema, field_info, tablefmt="github"): description = description.replace("\n\n", "
") # Otherwise a newline corresponds to a space description = description.replace("\n", " ") + # Spec internal links need to be replaced + if page_file is not None: + relpath = os.sep.join([".."] * page_file.src_path.count(os.sep)) + relpath = "." if not relpath else relpath + description = description.replace("SPEC_ROOT", relpath) df.loc[field_name] = [requirement_info, type_string, description] @@ -527,7 +548,7 @@ def make_metadata_table(schema, field_info, tablefmt="github"): return table_str -def make_columns_table(schema, column_info, tablefmt="github"): +def make_columns_table(schema, column_info, page_file=None, tablefmt="github"): """Produce columns table (markdown) based on requested fields. Parameters @@ -543,6 +564,8 @@ def make_columns_table(schema, column_info, tablefmt="github"): and the second string is additional table-specific information about the column that will be appended to the column's base definition from the schema. + page_file : MkDocs File object | None + The file where this macro is called, provided by the "page.file" variable. tablefmt : string, optional The target table format. The default is "github" (GitHub format). @@ -583,6 +606,11 @@ def make_columns_table(schema, column_info, tablefmt="github"): description = column_schema[field]["description"] + " " + description_addendum + if page_file is not None: + relpath = os.sep.join([".."] * page_file.src_path.count(os.sep)) + relpath = "." if not relpath else relpath + description = description.replace("SPEC_ROOT", relpath) + # Try to add info about valid values valid_values_str = utils.describe_valid_values(column_schema[field]) if valid_values_str: From a605539bfe94217ace9cc393a86e9218c78bbf26 Mon Sep 17 00:00:00 2001 From: "Christopher J. Markiewicz" Date: Wed, 27 Apr 2022 13:28:16 -0400 Subject: [PATCH 2/7] RF: Accept source paths as strings, render unconditionally --- tools/schemacode/schemacode/render.py | 69 ++++++++++++++++----------- 1 file changed, 41 insertions(+), 28 deletions(-) diff --git a/tools/schemacode/schemacode/render.py b/tools/schemacode/schemacode/render.py index 72d1e0ee0a..71e4c38279 100644 --- a/tools/schemacode/schemacode/render.py +++ b/tools/schemacode/schemacode/render.py @@ -1,6 +1,7 @@ """Functions for rendering portions of the schema as text.""" import logging import os +import posixpath import pandas as pd from tabulate import tabulate @@ -14,6 +15,26 @@ logging.basicConfig(format="%(asctime)-15s [%(levelname)8s] %(message)s") +def get_relpath(src_path): + """Retrieve relative path to the source root from the perspective of a Markdown file. + + As a convenience, ``None`` is interpreted as the empty string, and a value of ``'.'`` + is returned. + + Examples + -------- + >>> get_relpath("02-common-principles.md") + '.' + >>> get_relpath("04-modality-specific-files/01-magnetic-resonance-imaging-data.md") + '..' + >>> get_relpath("we/lack/third_levels.md") + '../..' + >>> get_relpath(None) + '.' + """ + return posixpath.relpath(".", posixpath.dirname(src_path or "")) + + def make_entity_definitions(schema): """Generate definitions and other relevant information for entities in the specification. @@ -56,7 +77,7 @@ def make_entity_definitions(schema): return text -def make_glossary(schema, page_file=None): +def make_glossary(schema, src_path=None): """Generate glossary. Parameters @@ -64,8 +85,9 @@ def make_glossary(schema, page_file=None): schema : dict The schema object, which is a dictionary with nested dictionaries and lists stored within it. - page_file : MkDocs File object | None - The file where this macro is called, provided by the "page.file" variable. + src_path : str | None + The file where this macro is called, which may be explicitly provided + by the "page.file.src_path" variable. Returns ------- @@ -122,10 +144,7 @@ def make_glossary(schema, page_file=None): # Otherwise a newline corresponds to a space obj_desc = obj_desc.replace("\n", " ") # Spec internal links need to be replaced - if page_file is not None: - relpath = os.sep.join([".."] * page_file.src_path.count(os.sep)) - relpath = "." if not relpath else relpath - obj_desc = obj_desc.replace("SPEC_ROOT", relpath) + obj_desc = obj_desc.replace("SPEC_ROOT", get_relpath(src_path)) text += f'\n' text += f"\n## {obj_key}\n\n" @@ -410,15 +429,16 @@ def _remove_numeric_suffixes(string): return table_str -def make_suffix_table(schema, suffixes, page_file=None, tablefmt="github"): +def make_suffix_table(schema, suffixes, src_path=None, tablefmt="github"): """Produce suffix table (markdown) based on requested suffixes. Parameters ---------- schema : dict suffixes : list of str - page_file : MkDocs File object | None - The file where this macro is called, provided by the "page.file" variable. + src_path : str | None + The file where this macro is called, which may be explicitly provided + by the "page.file.src_path" variable. tablefmt : str Returns @@ -450,10 +470,7 @@ def make_suffix_table(schema, suffixes, page_file=None, tablefmt="github"): # Otherwise a newline corresponds to a space description = description.replace("\n", " ") # Spec internal links need to be replaced - if page_file is not None: - relpath = os.sep.join([".."] * page_file.src_path.count(os.sep)) - relpath = "." if not relpath else relpath - description = description.replace("SPEC_ROOT", relpath) + description = description.replace("SPEC_ROOT", get_relpath(src_path)) df.loc[suffix] = [suffix_info["name"], description] @@ -466,7 +483,7 @@ def make_suffix_table(schema, suffixes, page_file=None, tablefmt="github"): return table_str -def make_metadata_table(schema, field_info, page_file=None, tablefmt="github"): +def make_metadata_table(schema, field_info, src_path=None, tablefmt="github"): """Produce metadata table (markdown) based on requested fields. Parameters @@ -482,8 +499,9 @@ def make_metadata_table(schema, field_info, page_file=None, tablefmt="github"): and the second string is additional table-specific information about the metadata field that will be appended to the field's base definition from the schema. - page_file : MkDocs File object | None - The file where this macro is called, provided by the "page.file" variable. + src_path : str | None + The file where this macro is called, which may be explicitly provided + by the "page.file.src_path" variable. tablefmt : string, optional The target table format. The default is "github" (GitHub format). @@ -536,10 +554,7 @@ def make_metadata_table(schema, field_info, page_file=None, tablefmt="github"): # Otherwise a newline corresponds to a space description = description.replace("\n", " ") # Spec internal links need to be replaced - if page_file is not None: - relpath = os.sep.join([".."] * page_file.src_path.count(os.sep)) - relpath = "." if not relpath else relpath - description = description.replace("SPEC_ROOT", relpath) + description = description.replace("SPEC_ROOT", get_relpath(src_path)) df.loc[field_name] = [requirement_info, type_string, description] @@ -548,7 +563,7 @@ def make_metadata_table(schema, field_info, page_file=None, tablefmt="github"): return table_str -def make_columns_table(schema, column_info, page_file=None, tablefmt="github"): +def make_columns_table(schema, column_info, src_path=None, tablefmt="github"): """Produce columns table (markdown) based on requested fields. Parameters @@ -564,8 +579,9 @@ def make_columns_table(schema, column_info, page_file=None, tablefmt="github"): and the second string is additional table-specific information about the column that will be appended to the column's base definition from the schema. - page_file : MkDocs File object | None - The file where this macro is called, provided by the "page.file" variable. + src_path : str | None + The file where this macro is called, which may be explicitly provided + by the "page.file.src_path" variable. tablefmt : string, optional The target table format. The default is "github" (GitHub format). @@ -606,10 +622,7 @@ def make_columns_table(schema, column_info, page_file=None, tablefmt="github"): description = column_schema[field]["description"] + " " + description_addendum - if page_file is not None: - relpath = os.sep.join([".."] * page_file.src_path.count(os.sep)) - relpath = "." if not relpath else relpath - description = description.replace("SPEC_ROOT", relpath) + description = description.replace("SPEC_ROOT", get_relpath(src_path)) # Try to add info about valid values valid_values_str = utils.describe_valid_values(column_schema[field]) From 14dce72f79a51b32a82b4f1dd34a4c5692c1f87d Mon Sep 17 00:00:00 2001 From: "Christopher J. Markiewicz" Date: Wed, 27 Apr 2022 13:29:02 -0400 Subject: [PATCH 3/7] RF: Retrieve src_path implicitly --- tools/mkdocs_macros_bids/macros.py | 52 ++++++++++++++++++++---------- 1 file changed, 35 insertions(+), 17 deletions(-) diff --git a/tools/mkdocs_macros_bids/macros.py b/tools/mkdocs_macros_bids/macros.py index dba9bc03ce..da0303540d 100644 --- a/tools/mkdocs_macros_bids/macros.py +++ b/tools/mkdocs_macros_bids/macros.py @@ -9,6 +9,13 @@ from examplecode import example +def _get_source_path(level=1): + import inspect + caller = inspect.currentframe().f_back + for _ in range(level): + caller = caller.f_back + return caller.f_locals["_Context__self"]["page"].file.src_path + def make_filename_template(**kwargs): """Generate a filename template snippet from the schema, based on specific @@ -71,13 +78,14 @@ def make_entity_definitions(): return text -def make_glossary(page_file=None): +def make_glossary(src_path=None): """Generate glossary. Parameters ---------- - page_file : MkDocs File object | None - The file where this macro is called, provided by the "page.file" variable. + src_path : str | None + The file where this macro is called, which may be explicitly provided + by the "page.file.src_path" variable. Returns ------- @@ -85,22 +93,24 @@ def make_glossary(page_file=None): A multiline string containing descriptions and some formatting information about the entities in the schema. """ + if src_path is None: + src_path = _get_source_path() schemapath = utils.get_schema_path() schema_obj = schema.load_schema(schemapath) - text = render.make_glossary(schema_obj, page_file=page_file) + text = render.make_glossary(schema_obj, src_path=src_path) return text -def make_suffix_table(suffixes, page_file=None): +def make_suffix_table(suffixes, src_path=None): """Generate a markdown table of suffix information. Parameters ---------- suffixes : list of str A list of the suffixes to include in the table. - page_file : MkDocs File object | None - The file where this macro is called, provided by the "page.file" variable. - + src_path : str | None + The file where this macro is called, which may be explicitly provided + by the "page.file.src_path" variable. Returns ------- @@ -108,13 +118,15 @@ def make_suffix_table(suffixes, page_file=None): A Markdown-format table containing the corresponding table for the requested fields. """ + if src_path is None: + src_path = _get_source_path() schemapath = utils.get_schema_path() schema_obj = schema.load_schema(schemapath) - table = render.make_suffix_table(schema_obj, suffixes, page_file=page_file) + table = render.make_suffix_table(schema_obj, suffixes, src_path=src_path) return table -def make_metadata_table(field_info, page_file=None): +def make_metadata_table(field_info, src_path=None): """Generate a markdown table of metadata field information. Parameters @@ -126,8 +138,9 @@ def make_metadata_table(field_info, page_file=None): Until requirement levels can be codified in the schema, this argument will be dictionary, with the field names as keys and the requirement levels as values. - page_file : MkDocs File object | None - The file where this macro is called, provided by the "page.file" variable. + src_path : str | None + The file where this macro is called, which may be explicitly provided + by the "page.file.src_path" variable. Returns ------- @@ -135,13 +148,15 @@ def make_metadata_table(field_info, page_file=None): A Markdown-format table containing the corresponding table for the requested fields. """ + if src_path is None: + src_path = _get_source_path() schemapath = utils.get_schema_path() schema_obj = schema.load_schema(schemapath) - table = render.make_metadata_table(schema_obj, field_info, page_file=page_file) + table = render.make_metadata_table(schema_obj, field_info, src_path=src_path) return table -def make_columns_table(column_info, page_file=None): +def make_columns_table(column_info, src_path=None): """Generate a markdown table of TSV column information. Parameters @@ -153,8 +168,9 @@ def make_columns_table(column_info, page_file=None): Until requirement levels can be codified in the schema, this argument will be a dictionary, with the column names as keys and the requirement levels as values. - page_file : MkDocs File object | None - The file where this macro is called, provided by the "page.file" variable. + src_path : str | None + The file where this macro is called, which may be explicitly provided + by the "page.file.src_path" variable. Returns ------- @@ -162,9 +178,11 @@ def make_columns_table(column_info, page_file=None): A Markdown-format table containing the corresponding table for the requested columns. """ + if src_path is None: + src_path = _get_source_path() schemapath = utils.get_schema_path() schema_obj = schema.load_schema(schemapath) - table = render.make_columns_table(schema_obj, column_info, page_file=page_file) + table = render.make_columns_table(schema_obj, column_info, src_path=src_path) return table From 6ddd71a73d15ae4ee6b87f75a9e1731e36364f24 Mon Sep 17 00:00:00 2001 From: "Christopher J. Markiewicz" Date: Wed, 27 Apr 2022 13:30:08 -0400 Subject: [PATCH 4/7] RF: Mock _Context__self object in PDF build --- pdf_build_src/process_markdowns.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/pdf_build_src/process_markdowns.py b/pdf_build_src/process_markdowns.py index 4f594b2927..4175ad228f 100644 --- a/pdf_build_src/process_markdowns.py +++ b/pdf_build_src/process_markdowns.py @@ -10,6 +10,7 @@ from datetime import datetime import json import os +import posixpath import re import subprocess import sys @@ -605,11 +606,13 @@ def process_macros(duplicated_src_dir_path): # which is a mock MkDocs File object with a str "src_path" attribute # The src_path mock_file = MockFile() - mock_file.src_path = os.sep.join(filename.split(os.sep)[1::]) + mock_file.src_path = posixpath.sep.join(filename.split(os.sep)[1:]) page = MockPage() page.file = mock_file + _Context__self = {"page": page} + # Replace code snippets in the text with their outputs matches = re.findall(re_code_snippets, contents) for m in matches: From 5e07fe3e646947ea72c7fee81e8ef8133d7cc36d Mon Sep 17 00:00:00 2001 From: "Christopher J. Markiewicz" Date: Wed, 27 Apr 2022 15:29:32 -0400 Subject: [PATCH 5/7] DOC: (Over)documenting the _get_source_path() magic --- tools/mkdocs_macros_bids/macros.py | 43 ++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/tools/mkdocs_macros_bids/macros.py b/tools/mkdocs_macros_bids/macros.py index da0303540d..355ddbf091 100644 --- a/tools/mkdocs_macros_bids/macros.py +++ b/tools/mkdocs_macros_bids/macros.py @@ -9,11 +9,54 @@ from examplecode import example + def _get_source_path(level=1): + """ Detect the path of the file we are rendering a macro in. + + This (ab)uses the Python call stack to find its way to the Jinja2 function + that is calling the macro. From there, it looks at Jinja2's Context object, + which contains all the variables available to the Markdown snippet that is + calling the macro. + + One variable provided by mkdocs-macros is called ``page``, which includes a + ``file`` attribute that would allow us to insert the page name into the text + of the page, or in this case, pass it as a variable. The ``file`` attribute + has a ``src_path`` attribute of its own that is a path relative to the ``src/`` + directory. + + The level parameter indicates how many steps above the calling function Jinja2 + is. Currently it's always 1, but refactors may justify passing a larger number. + + This allows us to use + + ```{markdown} + {{ MACRO__make_glossary() }} + ``` + + instead of: + + ```{markdown} + {{ MACRO__make_glossary(page.file.src_path) }} + ``` + + Why are we doing all this? We need to render links that are defined in the schema + relative to the source tree as paths relative to the Markdown file they're being + rendered in. So [SPEC_ROOT/02-common-principles.md](Common principles) becomes + [./02-common-principles.md](Common principles) or + [../02-common-principles.md](Common principles), depending on which file it + appears in. + + If a future maintainer decides that this is terrible, or a bug can't be fixed, + just go back to explicitly using the ``page.file`` variable throughout the macros. + """ import inspect + # currentframe = _get_source_path() + # caller = the macro calling this function, e.g. make_glossary() caller = inspect.currentframe().f_back + # We need to go one level higher to find Jinja2 for _ in range(level): caller = caller.f_back + # Jinja2 equivalent: {{ page.file.src_path }} return caller.f_locals["_Context__self"]["page"].file.src_path From 9678b8211a71687092b7c803d7a08f09a9f0dcd5 Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Fri, 29 Apr 2022 08:39:24 +0200 Subject: [PATCH 6/7] [MISC] make dataset_description.Authors RECOMMENDED (#1092) --- src/03-modality-agnostic-files.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/03-modality-agnostic-files.md b/src/03-modality-agnostic-files.md index 82bdc20af0..c08afc70c7 100644 --- a/src/03-modality-agnostic-files.md +++ b/src/03-modality-agnostic-files.md @@ -27,7 +27,7 @@ and a guide for using macros can be found at "HEDVersion": "RECOMMENDED", "DatasetType": "RECOMMENDED", "License": "RECOMMENDED", - "Authors": "OPTIONAL", + "Authors": "RECOMMENDED", "Acknowledgements": "OPTIONAL", "HowToAcknowledge": "OPTIONAL", "Funding": "OPTIONAL", From 9a146fa09c7ca00a184ead345a2c6ce527d5de48 Mon Sep 17 00:00:00 2001 From: bids-maintenance Date: Fri, 29 Apr 2022 06:41:32 +0000 Subject: [PATCH 7/7] [DOC] Auto-generate changelog entry for PR #1096 --- src/CHANGES.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/CHANGES.md b/src/CHANGES.md index b6bf16c8eb..3161bcb4a2 100644 --- a/src/CHANGES.md +++ b/src/CHANGES.md @@ -2,9 +2,11 @@ ## [Unreleased](https://github.com/bids-standard/bids-specification/tree/HEAD) +- \[INFRA] Fix internal links implicitly [#1096](https://github.com/bids-standard/bids-specification/pull/1096) ([effigies](https://github.com/effigies)) +- \[MISC] make dataset_description.Authors RECOMMENDED [#1092](https://github.com/bids-standard/bids-specification/pull/1092) ([sappelhoff](https://github.com/sappelhoff)) - ENH: use \__file\_\_ instead of getcwd to determine HERE, provide informative message on # of files [#1091](https://github.com/bids-standard/bids-specification/pull/1091) ([yarikoptic](https://github.com/yarikoptic)) - \[INFRA] Set light and dark mode image for logo in README [#1085](https://github.com/bids-standard/bids-specification/pull/1085) ([anibalsolon](https://github.com/anibalsolon)) -- State of the schema sprint [#1075](https://github.com/bids-standard/bids-specification/pull/1075) ([effigies](https://github.com/effigies)) +- \[ENH] State of the schema sprint [#1075](https://github.com/bids-standard/bids-specification/pull/1075) ([effigies](https://github.com/effigies)) - \[DOC] Preface each macro call with comment [#1052](https://github.com/bids-standard/bids-specification/pull/1052) ([Remi-Gau](https://github.com/Remi-Gau)) - ENH: allow for .png and .tif in eeg/ieeg/meg as allowed for micr [#1049](https://github.com/bids-standard/bids-specification/pull/1049) ([yarikoptic](https://github.com/yarikoptic)) - \[SCHEMA] Add a new `file\_relative` format [#1046](https://github.com/bids-standard/bids-specification/pull/1046) ([tsalo](https://github.com/tsalo))