Sanitize metadata in chunks

[huddlej]

Description of proposed changes

Instead of loading all metadata into memory to perform transformation and filtering associated with sanitizing logic, process metadata in chunks that easily fit in memory.

To support this low-memory implementation, this PR changes the approach to resolving duplicate records. We now make a first pass through the metadata to store strain ids and corresponding database ids for each record in a temporary file on disk. We make a second pass to filter duplicate records, transform the remaining records, and stream the results to disk.

One minor interface change (improvement) from this refactoring is the creation of a separate file containing the list of all duplicate strains when the user has requested an error on duplicates. This change allows the user to bioinformatically address duplicates after the fact in a way that printing the list of strains to stderr did not easily allow.

Note that one side effect of the two-pass approach to sanitizing metadata is that reading metadata from tarballs no longer worked as expected. This functionality was fixed by writing the requested single file from a given tarball to a temporary file on disk instead of streaming the contents of that file as an io.BufferedReader.

Fixes #697

Development plan

• Move hard-coded strain field names to defaults for a user argument to parallel augur filter’s --metadata-id-columns.
• Define a function to check for and return all available accession fields.
• Move logic for renaming fields into a function and call it from earlier in the script
• Move logic for stripping prefixes into a function with tests
• Use augur.io.read_metadata to read metadata in an iterator.
• Loop through the metadata in the first pass, writing out strain name (i.e., the index column of the data frame) and all available accession fields to a database.
• Loop through the metadata in the second pass,
  • Select for the latest accession for each strain in the current batch from the name/id mapping database.
  • Filter the current batch data frame to the latest accessions
  • Apply column and value changes to the filtered batch data frame
  • Write the filtered and transformed batch data frame to disk.
• Delete the name/id mapping database.

Testing

This PR refactors top-level code into its own functions with doctests. It also adds cram-based functional tests that were used for test-driven development of the new low-memory functionality and to confirm that the user interface remains stable.

Run doctests:

python3 -m doctest scripts/sanitize_metadata.py

Run functional tests:

cram --shell=/bin/bash tests/sanitize-metadata.t

Run sanitize script on full GISAID metadata.

python3 scripts/sanitize_metadata.py \
  --metadata gisaid_metadata.tsv.gz \
  --metadata-id-columns 'Virus name' \
  --database-id-columns 'Accession ID' \
  --parse-location-field Location \
  --rename-fields 'Virus name=strain' Type=type 'Accession ID=gisaid_epi_isl' 'Collection date=date' 'Additional location information=additional_location_information' 'Sequence length=length' Host=host 'Patient age=patient_age' Gender=sex Clade=GISAID_clade 'Pango lineage=pango_lineage' Lineage=pango_lineage 'Pangolin version=pangolin_version' Variant=variant 'AA Substitutions=aa_substitutions' aaSubstitutions=aa_substitutions 'Submission date=date_submitted' 'Is reference?=is_reference' 'Is complete?=is_complete' 'Is high coverage?=is_high_coverage' 'Is low coverage?=is_low_coverage' N-Content=n_content GC-Content=gc_content \
  --strip-prefixes hCoV-19/ \
  --output results/sanitized_metadata_gisaid.tsv.gz

Running this script on the 3,492,105 records from a recent full GISAID download took ~10 minutes (~1 minute for the first pass to build the id map, ~9 minutes for the second pass) and used ~330 MB of memory at peak use. The current implementation could be sped up substantially by tuning the duplicate filter logic (e.g., querying a sqlite database indexed by strain name instead of searching a plain text file for each chunk in the second pass) and by using the C-based pandas parser for TSV instead of the Python-based parser. This latter change requires a breaking change to the Augur API.

Release checklist

This pull request introduces the following features that should be noted in the change log:

• Enable configuration of metadata strain id and database id columns in the build config
• Create a separate text file named like <input>.duplicates.txt containing a list of duplicate strains (one per line) when the user requests --error-on-duplicate-strains.

Remaining tasks are:

• Resolve issues reading from tarball file handles with augur.io.read_metadata
• Add config parameter to allow toggling --error-on-duplicate-strains
• Document new config parameters
• Update change log to reflect new features