Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Testing Script #13

Merged
merged 43 commits into from
Jun 5, 2024
Merged

Add Testing Script #13

merged 43 commits into from
Jun 5, 2024

Conversation

nguyenank
Copy link
Collaborator

Add a testing script (mapper-test.yml) that adds integration with the ontology-mapper-tests repo [OMT repo] to automatically evaluate the mapper performance against a gold standard.

The script runs for each push and pull-request for any branches in the list for the appropriate event under on:; add the name of the branch if you want the script to run for a given action.

The tool currently runs on terms from the UK Biobank (OMT repo: data/ukb-efo/terms.txt), using EFO 2.106 released on 2019-03-18 (OMT repo: data/ukb-efo/efo-v2019-03-18.owl) as referenced in the paper on hand-mapping UK Biobank terms to EFO. The output is saved in the OMT repo in mapper-outputs/ukb-efo/{commit SHA}.csv(.json).

A script then calculates the coverage error and one error (OMT repo: compare-scripts/coverage_one_error.py) between the mapper output and the mappings done by hand (OMT repo: data/ukb-efo/gold-standard.csv). Results are saved in a spreadsheet (OMT repo: summary.csv), with each row being a different run of the script. The spreadsheet has the following columns:

  • branch - the branch on which either a push/pull-request was done that initiated the script run
  • commit-sha - the identifier of the most recent commit that is used as an ID for the particular script run
  • github-comment - the comment of the most recent commit
  • ukb-efo_mapper_output_filename - the filename of the mapper output from running on the UK Biobank terms with the EFO ontology; should always be of the form:ukb-efo/{commit SHA}.csv
  • coverage_error - the coverage error
  • one_error - the one error

@rsgoncalves rsgoncalves merged commit a97c99c into main Jun 5, 2024
1 of 2 checks passed
@rsgoncalves rsgoncalves deleted the task5-workflow branch July 8, 2024 16:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants