Filtering of Files on Dataset Page, Search using Solr on Dataset Page

[djbrooke]

On the dataset page, users should be able to search for AND filter files as depicted in the screenshot. This may involve some work with Solr on the page (or some representation of the Solr index that can be easily tapped into). Assigning to @scolapasta so that he can use tech hours to generate some ideas.

[scolapasta]

As we only index the latest published and draft versions ("present versions"), we'd only be able to use Solr for those versions. It had previously been decided that this was ok (at least, as a first batch).

We discussed today at tech hours - we will generate a list of ids from either the solr (for present versions) or the db (for past versions). That list could then be used the same way to get the details to display on the cards. The facets themselves would only be rendered on present versions.

[djbrooke]

• We should check on the implications for the Solr load as part of this. Does Solr load with the page (to get the facets), or only if the user attempts to filter/search on the page?

[mheppler]

Changes to the files table source code on the dataset pg is being revised as part of Enable the display of file hierarchy metadata on the dataset page #5572 which will impact the same HTML files this issue touches. This will need some development coordination and manual merging to resolve the expected conflicts.

[mheppler]

Added static placeholders for file table facets on dataset pg in a new branch 5584-dataset-solr-facets.

[mheppler]

Updated the latest from develop in order to get the new file hierarchy view toggle UI component (#5572) merged into this branch.

[landreev]

(just made the PR above; just to make it easier to look at the commits. it should stay in dev. for now!)

[mheppler]

• add render logic to hide filter facets and Sort btn when there is 1 or less files
• add selected state bold styling to facet values in btn label, and dropdown menu
• move dynamic file counter into file table header to make space for filter facets
• move file thumbnails from table column into the file metadata table column to make room for the dynamic file counter in the file table header
• move Upload Files and Download Rsync Script btns to left of search input to make space for Sort btn
• change Sort btn style to match dv pg
• wire up Sort btn
• fix various responsive layout issues with file table (related to new display:flex; container around file metadata)
• add word-break: break-all; inline style to checksum + UNF containers to fix responsive layout issues
• move text to bundle

[landreev]

@mheppler I changed your rendering rules, to show the Sort button for non-indexed versions too. But that really breaks your styling of the fragment, because without the facets, the button is now hanging in the middle of all that white space:

I'm sure you can fix it... but I would seriously consider living without it, for non-indexed versions (since they will be rarely used; and that's how the page is looking now anyway). But up to you.

I'm still working on the back end.

[landreev]

sort button should be working now.
(for indexed and non-indexed versions both)

[mheppler]

Fixed the Sort btn layout issue @landreev discovered in old versions with no facets. Fixed other various layout issues including the checksums and UNF's getting off in small browser windows due to the new flexbox layout used with the file thumbnail and metadata layout in each row. (See attached.)

These fixes were added to the to-do list above which outlines all the moving pieces.

[landreev]

Notes for the reviewer(s):

In the process of working on this we realized it was impossible to accurately search for files in draft versions using our solr index. This is because we do NOT index files that have not changed between the latest published version and the draft. (this is to avoid having duplicate search cards for these files)
So for any dataset we index a) all the files in the latest published version and b) all the files that have been added in the draft (if present), as well as the files for which any metadata have changed in the draft.
What this means is we can search files in the latest published version by supplying the filter query "datasetVersionId: N". However, if you run a search with "datasetVersionId: M", where M is the version number of the draft, you're only going to find the files added or changed since the last publication. You can use "parentId: N" where N is the id of the dataset. And then you will find all the files in the published and draft versions. But that means we are still getting the files that have been deleted from the draft.
This does not cause a problem with the list of the files shown on the dataset page. Because the search results are filtered against the files in the version. BUT it can mess up the numbers in the facets, if there are any deleted files.

I solved this by adding another boolean to the solr schema - "fileDeleted". It's set to true in a solr document for a published file that's no longer found in the draft (if exists). This way we can find all the files in the draft by adding 2 filter queries: "parentId: N" and "fileDeleted: false"

The disadvantage of this solution is having to force the solr schema update (and the reindex). The search that serves the page is built with backward compatibility - so that it doesn't completely fail if it's talking to the solr server that doesn't have the new schema yet (in this case it reverts to searching without the new flag; potentially showing higher numbers in the facets).

Is it worth it? - We could alternatively just say that this is a known limitation - that the facet numbers for draft versions may show higher counts, on account of the deleted files. After all, only the dataset owners will be seeing the draft versions.

(that said, we will have to force a reindex of all everything sometime soon anyway - to get all the improved file types indexed!)

[landreev]

Notes for QA:

This PR has a new solr schema.
The new schema, and a reindex, are required - although it's still going to work; the only problem is the numbers in the facets may be inaccurate (for drafts ONLY! - and only if any previously published files have been deleted from the draft; this is explained in the long comment above)

If solr is down, the page should still be working - the search box should still be there, just without the facets.
Similarly, this is how the page should behave for published versions other than the latest published, or the draft.

Aside from testing the searching and the facets for basic accuracy, the page should probably be tested some more with larger numbers of files.

[djbrooke]

I'm going to do an experiment for a possible workflow change, that is, once there's a PR for something, lock the discussion in the issue and move the discussion to the PR. I think this will work better with the flow on our new project board https://github.com/orgs/IQSS/projects/2.

So, I'm going to "lock" this conversation and move the conversation to #5820. Like I said, an experiment.