export v2: Add --metadata-columns option

[joverlee521]

Description of proposed changes

Allows users to specify additional metadata columns to export as
node attributes that are not specified as coloring options via
the existing --color-by-metadata option or the Auspice config JSON.
This allows the metadata columns to be visible in the tree in Auspice
without polluting the color-by options.

Related issue(s)

Resolves #1383

Checklist

• Checks pass
• If making user-facing changes, add a message in CHANGES.md summarizing the changes in this PR

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (abc86a8) 66.71% compared to head (32c9740) 66.96%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1384      +/-   ##
==========================================
+ Coverage   66.71%   66.96%   +0.24%     
==========================================
  Files          69       69              
  Lines        7322     7341      +19     
  Branches     1799     1804       +5     
==========================================
+ Hits         4885     4916      +31     
+ Misses       2169     2154      -15     
- Partials      268      271       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jameshadfield]

Nice one Jover - the overall aim of this work is something I've wanted for a long time. I haven't tested the code, just skimmed through.

These columns will not be used as coloring options in Auspice but will be visible in the tree

Here do you mean visible in the modal when (shift-)clicking on a branch/tip? It'd be good to surface them as sidebar filtering options too, but that'll take an Auspice change I think. My understanding of the linked issue (and from discussions with others) is that datasets are adding unnecessary colorings in order to expose metadata as filtering options.

[joverlee521]

Here do you mean visible in the modal when (shift-)clicking on a branch/tip?

Yes, I mean they will only be visible on the modal on click.

It'd be good to surface them as sidebar filtering options too, but that'll take an Auspice change I think. My understanding of the linked issue (and from discussions with others) is that datasets are adding unnecessary colorings in order to expose metadata as filtering options.

Yup, I think this will require changes in Auspice. My understanding is that currently filters must be defined in the Auspice JSON under meta.filters. Even colorings are not automatically available as filters (nextstrain/auspice#1251).

[victorlin]

Functionality is great! Suggestions below are non-blocking (except changelog request).

[jameshadfield]

Allows users to specify additional metadata columns to export as node attributes that are not specified as coloring options via the existing --color-by-metadata option or the Auspice config JSON. This allows the metadata columns to be visible in the tree in Auspice without polluting the color-by options.

This behaviour is already possible by specifying the (TSV metadata column names) in the auspice config's filters array. There are side effects beyond those implemented in this PR, as each key in filters will be shown as a (collapsed) section in the footer, but the (big) upside is that the metadata is searchable.

I want to change the general behaviour here a bit, but I think we are better to sketch the big direction out first before merging any PRs. I'm envisioning something like the following, but it's not been completely thought through:

1. Auspice uses all node_attrs as sidebar filtering options (and maybe make branch labels searchable too, but that's complicated).
2. The filters array (in the dataset JSON) is reserved for the items listed in the footer. I see this functionality as less important now that we have sidebar filtering.
3. We have some way of exporting metadata (TSV) fields without them being colourings. Maybe that's this PR?

But maybe just doing (1) is enough? Thoughts?

[jameshadfield]

Any functionality we support via command line arguments should be available via the auspice-config JSON as well, as the former is a subset of the latter's capabilities.

[joverlee521]

Hmm, this flag relates to the metadata input/export but not really Auspice control settings. Not sure this fits in the auspice-config JSON?

[jameshadfield]

I see it as controlling what data is exported, similar to filters & colours. The (original) intention when we started adding command line arguments for options in the config JSON was for auspice-config to be all you need to control what data the JSON contains.

[joverlee521]

Okay, that makes sense. I'll move up the flag to be part of the config args group too then.

[joverlee521]

Auspice config schema update and changes to support the new property added in bdb2b2c

[joverlee521]

I want to change the general behaviour here a bit, but I think we are better to sketch the big direction out first before merging any PRs.

1. Auspice uses all node_attrs as sidebar filtering options (and maybe make branch labels searchable too, but that's complicated).
2. The filters array (in the dataset JSON) is reserved for the items listed in the footer. I see this functionality as less important now that we have sidebar filtering.
3. We have some way of exporting metadata (TSV) fields without them being colourings. Maybe that's this PR?

But maybe just doing (1) is enough? Thoughts?

I think (1) is definitely needed. I think this will require some design on filtering for continuous value fields?

On (2), I think we can even deprecate the footer filters. I definitely only use the sidebar filtering and have not looked at footer filters in a while.

I think even if we implement (1), we still need some way to support (3). I think it would be nice to fully detangle exported metadata from coloring options. I would want to be more explicit about what metadata fields are exported instead of letting it stay a side effect of other Auspice configurations.

[joverlee521]

Rebased to fix merge conflicts in changelog.

[joverlee521]

@jameshadfield since (1) was implemented in nextstrain/auspice#1743, do you still have feedback here?

[jameshadfield]

@jameshadfield since (1) was implemented in nextstrain/auspice#1743, do you still have feedback here?

On (2), I think we can even deprecate the footer filters. I definitely only use the sidebar filtering and have not looked at footer filters in a while.

My interpretation of the deprecation here was to deprecate the auspice UI, but keep the augur export filters setting and use this as the way to add metadata that won't be a coloring. As in "what data would you like to include for filtering in auspice". But would you prefer to add --metadata-columns and remove --filters entirely?

(If we do add --metadata-columns, I do think it needs to be a setting in the auspice-config as well.)

[joverlee521]

My interpretation of the deprecation here was to deprecate the auspice UI, but keep the augur export filters setting and use this as the way to add metadata that won't be a coloring. As in "what data would you like to include for filtering in auspice". But would you prefer to add --metadata-columns and remove --filters entirely?

(If we do add --metadata-columns, I do think it needs to be a setting in the auspice-config as well.)

I'd prefer to add --metadata-columns. The --filters flag does not currently exist, so this would just be deprecating the filters key in the auspice config JSON.

For me,--filters does not tell me if the data is coming from the metadata TSV. I'd be confused about whether or not I need to include node-data JSON fields to use them as filters as well.

[huddlej]

On (2), I think we can even deprecate the footer filters. I definitely only use the sidebar filtering and have not looked at footer filters in a while.

I'd vote to please keep the footer filters, if possible. I still use them regularly and find them helpful for quickly picking from a longer list of options (e.g., year-month values for flu, clades, regions, etc.). It's generally been helpful to have a way to see in a single view what the filter fields are and the specific values within each field.

[jameshadfield]

I haven't re-reviewed everything but the changes in bdb2b2c look good by inspection

[joverlee521]

I'd vote to please keep the footer filters, if possible. I still use them regularly and find them helpful for quickly picking from a longer list of options (e.g., year-month values for flu, clades, regions, etc.). It's generally been helpful to have a way to see in a single view what the filter fields are and the specific values within each field.

@huddlej No worries, this PR doesn't touch the footer filters! Although this reminded me that the default footer filters will likely only be a subset of available filter fields since it currently does not include the additional metadata columns!

augur/augur/export_v2.py

Lines 539 to 542 in 4bf72f8

	# if not specified in the config, include all boolean and categorical colorbys
	potentials = {coloring["key"] for coloring in data_json['meta']["colorings"]
	if coloring["type"] != "continuous" and coloring["key"] != 'gt'}
	data_json['meta']['filters'] = list(potentials)

[joverlee521]

Merging this PR as-is, will update the footer filters separately (if desired).