Search API results payload extension

[GPortas]

What this PR does / why we need it:

Currently, the search API does not return all the attributes needed for files and collections, required to display the file and collection item cards.

I have extended the Search API result payloads to include:

• For the Dataverse entity: "affiliation", "parentDataverseName", "parentDataverseIdentifier" and "image_url" (optional).
• For the DataFile entity: "releaseOrCreateDate" and "image_url" (optional).
• For the Dataset entity: "image_url" (optional).

The schema.xml file for Solr has been updated to include a new field called dvParentAlias for the parentDataverseIdentifier. This field is now indexed in the IndexServiceBean.

The image_url field was already included in the SolrSearchResult JSON, but it wasn’t returned by the API because it was appended only after the Solr query was processed in the SearchIncludeFragment of JSF. Now, the field is set in SearchServiceBean, ensuring it is always returned by the API when an image is available.

However, in the search.rst documentation, the field was incorrectly listed as being returned by the API, even though it never was since it was only set in JSF.

I have reused the same URL generation logic from JSF. But currently, URL generation differs between datasets and dataverses&files. For datasets, an absolute URL is generated, while for files and dataverses, a Base64 URL is generated. In a future iteration, it would be beneficial to standardize the URL format so that the same type of URL is returned for all entities. I have mentioned this in the docs.

Which issue(s) this PR closes:

• Closes Extend Search API object type payloads to include necessary information for collection item cards #10810

Special notes for your reviewer:

For the next released version, a Solr reindex will be necessary to apply the new schema.xml version. (Mentioned in the release notes)

Suggestions on how to test this:

Create dataverses, datasets and files and test the search endpoint to verify that new params are correctly retrieved.

curl "http://localhost:8080/api/search?q=*&sort=date&order=desc" -H "X-Dataverse-Key: <YOUR_API_KEY>"

Does this PR introduce a user interface change? If mockups are available, please link/include them here:
No

Is there a release notes update needed for this change?:
Yes

Additional documentation:
No

Preview docs at https://dataverse-guide--10811.org.readthedocs.build/en/10811/api/search.html

Related issues:

• image_url from Search API results no longer yields a downloadable image #3616

[johannes-darms]

Does this PR also resolved/includes the requested changes of #9788?

[GPortas]

Does this PR also resolved/includes the requested changes of #9788?

@johannes-darms No. This PR adds new fields to the Search API result, without changing existing ones. The scopes are different.

I think what is proposed in #9788 requires further discussion, in particular for maintaining backward compatibility in the endpoint, since existing fields would be redefined / renamed.

[johannes-darms]

Does this PR also resolved/includes the requested changes of #9788?

@johannes-darms No. This PR adds new fields to the Search API result, without changing existing ones. The scopes are different.

Partly the issue list difference between list/search and get, but also the missing items needed to recreate the dataverse/collection cards based on the search response like logo, affiliation and description. Some of these are already implemented in this PR. Could we add the rest in this PR as well?

I think what is proposed in #9788 requires further discussion, in particular for maintaining backward compatibility in the endpoint, since existing fields would be redefined / renamed.

Only suggestion two of the problem changes a property, the rest are just additions. Phil has already proposed a solution that would work. I think this is a great PR to fix the problems listed in #9788 and #9713.

[GPortas]

@johannes-darms

I am working on adding your requested properties. imageUrl was straightforward to expose, as you describe.

I see that the linked icon depends on the isInTree boolean flag property from the Solr result. Actually this property is not indexed; instead, it is post-calculated in the UI logic and appended to the SolrSearchResult, specifically in SearchIncludeFragment, as can be seen here: https://github.com/IQSS/dataverse/blob/develop/src/main/java/edu/harvard/iq/dataverse/search/SearchIncludeFragment.java#L1475.

If we want to expose the flag as it is used by the JSF UI, we need to move the calculation logic out of SearchIncludeFragment, potentially to SearchServiceBean or IndexServiceBean.

What is already indexed and therefore straightforward to expose is subtreePaths, but for now I'm not sure if this information is enough to calculate the linked status, as in my tests I can't get the complete subtree starting from the root dataverse, which I believe is what we would need to determine the linked status. This field only returns the first dataverse below root. For instance, if I have this hierarchy: root > dv1 > dv2 > dv3, subtreePaths for entity dv2 returns "/<dv1_id>", instead of "/<dv1_id>/<dv2_id>" which is what I expect (the entire subtree).

I'm still looking into this, but I think determining the linked status may require exploring a solution that would go beyond the scope of this PR / issue.

[johannes-darms]

I am working on adding your requested properties. imageUrl was straightforward to expose, as you describe.

Many thanks!

I see that the linked icon depends on the isInTree boolean flag property from the Solr result. Actually this property is not indexed; instead, it is post-calculated in the UI logic and appended to the SolrSearchResult, specifically in SearchIncludeFragment, as can be seen here: https://github.com/IQSS/dataverse/blob/develop/src/main/java/edu/harvard/iq/dataverse/search/SearchIncludeFragment.java#L1475. [...]
I'm still looking into this, but I think determining the linked status may require exploring a solution that would go beyond the scope of this PR / issue.

Bummer, I totally agree with you that this refactoring should be a separated PR. Especially as there are two for loops to determine the harvested and linked status of an item, refactoring could improve the performance.

[coveralls]

coverage: 20.74% (+0.005%) from 20.735%
when pulling e517183 on 10810-search-api-payload-extensions
into 4143031 on develop.

[pdurbin]

Overall, this looks great. I'm asking some questions in this review.

[pdurbin]

This feels really weird. image_url is sometimes a URL (good!) and sometimes a base64 value (weird!)? Can we pick one?

[GPortas]

Just for keeping track in the PR. I repost the same info I sent to you on Slack.

For the SPA, using an absolute URL is more suitable than Base64, as it allows the browser to automatically cache the resources. However, I believe there might be performance-related reasons for using Base64. For reference, see: https://github.com/IQSS/dataverse/blob/develop/src/main/java/edu/harvard/iq/dataverse/search/SolrSearchResult.java#L42.

[qqmyers]

Browser caching should work for the current UI as well, so I'm doubtful that there's a big performance gain embedding base64.

[pdurbin]

Datasets can have identifier_of_dataverse but not parentDataverseIdentifier, right?

[GPortas]

Yes. parentDataverseIdentifier is only for Dataverses.

[pdurbin]

Files do not get parentDataverseIdentifier, right?

[GPortas]

Right! Just for Dataverses

[github-actions]

📦 Pushed preview images as

ghcr.io/gdcc/dataverse:10810-search-api-payload-extensions

ghcr.io/gdcc/configbaker:10810-search-api-payload-extensions

🚢 See on GHCR. Use by referencing with full name as printed above, mind the registry name.

[pdurbin]

I didn't run the code but it seems reasonable and @GPortas addressed my concern about image_url by explaining in the guides that it is expected to change in #10831. Tests are passing. Approved!