Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support deletion of saved searches #10198

Merged
merged 16 commits into from
Aug 27, 2024
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions doc/release-notes/9317-delete-saved-search.md
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm going to leave comments below about backward incompatible changes. I'm pretty sure my preference is to NOT introduce any backward-incompatible changes but if we do the should be listed in doc/sphinx-guides/source/api/changelog.rst and the release notes should give an overview of backward-incompatible changes and tell people to check that changelog.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, I removed every authentication requirement for /api/admin/savedsearches

Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
### Saved search deletion

Saved search can now be removed using API `/api/admin/savedsearches/$id`. See PR #10198.
This is reflected in the [Saved Search Native API section](https://guides.dataverse.org/en/latest/api/native-api.html#saved-search) of the Guide.
7 changes: 5 additions & 2 deletions doc/sphinx-guides/source/api/native-api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5745,8 +5745,7 @@ The ``$identifier`` should start with an ``@`` if it's a user. Groups start with
Saved Search
~~~~~~~~~~~~

The Saved Search, Linked Dataverses, and Linked Datasets features shipped with Dataverse 4.0, but as a "`superuser-only <https://github.com/IQSS/dataverse/issues/90#issuecomment-86094663>`_" because they are **experimental** (see `#1364 <https://github.com/IQSS/dataverse/issues/1364>`_, `#1813 <https://github.com/IQSS/dataverse/issues/1813>`_, `#1840 <https://github.com/IQSS/dataverse/issues/1840>`_, `#1890 <https://github.com/IQSS/dataverse/issues/1890>`_, `#1939 <https://github.com/IQSS/dataverse/issues/1939>`_, `#2167 <https://github.com/IQSS/dataverse/issues/2167>`_, `#2186 <https://github.com/IQSS/dataverse/issues/2186>`_, `#2053 <https://github.com/IQSS/dataverse/issues/2053>`_, and `#2543 <https://github.com/IQSS/dataverse/issues/2543>`_). The following API endpoints were added to help people with access to the "admin" API make use of these features in their current form. There is a known issue (`#1364 <https://github.com/IQSS/dataverse/issues/1364>`_) that once a link to a Dataverse collection or dataset is created, it cannot be removed (apart from database manipulation and reindexing) which is why a ``DELETE`` endpoint for saved searches is neither documented nor functional. The Linked Dataverse collections feature is `powered by Saved Search <https://github.com/IQSS/dataverse/issues/1852>`_ and therefore requires that the "makelinks" endpoint be executed on a periodic basis as well.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a conversation going on in #10628 about how these docs used to explain to people that they might have to wait up to 24 hours for their saved search to show results. Should we add that back as part of this pull request?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated with The update of all saved search is run by a timer once a week (See :ref:saved-search-timer) so if you just created a saved search, you can run manually makelinks endpoint that will find new dataverses and datasets that match the saved search and then link the search results to the dataverse in which the saved search is defined.


The Saved Search, Linked Dataverses, and Linked Datasets features are only accessible to superusers except for Linking a dataset. The following API endpoints were added to help people with access to the “admin” API make use of these features in their current form, keep in mind that they are partially experimental.
List all saved searches. ::

GET http://$SERVER/api/admin/savedsearches/list
Expand All @@ -5755,6 +5754,10 @@ List a saved search by database id. ::

GET http://$SERVER/api/admin/savedsearches/$id

Delete a saved search by database id. The ``unlink=true`` query parameter unlink links (Linked dataset or Dataverse collection) related to the deleted saved search. This parameter should be well considered as you cannot know if the saved search created the links or if someone else did via other API. Also, it may be followed ``/makelinks/all`` depending on the need if other saved searches could recreate some deleted links or by reindexing some Dataverse or Dataset. ::

DELETE http://$SERVER/api/admin/savedsearches/$id?unlink=true

Execute a saved search by database id and make links to Dataverse collections and datasets that are found. The JSON response indicates which Dataverse collections and datasets were newly linked versus already linked. The ``debug=true`` query parameter adds to the JSON response extra information about the saved search being executed (which you could also get by listing the saved search). ::

PUT http://$SERVER/api/admin/savedsearches/makelinks/$id?debug=true
Expand Down
100 changes: 88 additions & 12 deletions src/main/java/edu/harvard/iq/dataverse/api/SavedSearches.java
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
package edu.harvard.iq.dataverse.api;

import edu.harvard.iq.dataverse.Dataverse;
import edu.harvard.iq.dataverse.api.auth.AuthRequired;
import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser;
import edu.harvard.iq.dataverse.engine.command.DataverseRequest;
import edu.harvard.iq.dataverse.engine.command.exception.CommandException;
Expand All @@ -24,6 +25,8 @@
import jakarta.ws.rs.Path;
import jakarta.ws.rs.PathParam;
import jakarta.ws.rs.QueryParam;
import jakarta.ws.rs.container.ContainerRequestContext;
import jakarta.ws.rs.core.Context;
import jakarta.ws.rs.core.Response;
import static jakarta.ws.rs.core.Response.Status.BAD_REQUEST;
import static jakarta.ws.rs.core.Response.Status.INTERNAL_SERVER_ERROR;
Expand All @@ -46,22 +49,46 @@ public Response meta() {
}

@GET
@AuthRequired
@Path("list")
public Response list() {
public Response list(@Context ContainerRequestContext crc) {

AuthenticatedUser superuser = null;
try {
superuser = getRequestAuthenticatedUserOrDie(crc);
} catch (WrappedResponse wr) {
return wr.getResponse();
}
if (superuser == null || !superuser.isSuperuser()) {
return error(Response.Status.UNAUTHORIZED, "Superusers only.");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this backward-incompatible change? I'm only going to leave this comment once but it applies to all endpoints that formerly were not superuser-only.

@luddaniel I know we talked about this a bit at https://dataverse.zulipchat.com/#narrow/stream/379673-dev/topic/admin.20API.20and.20superuser.20authorization/near/443079852 but my understanding was that we were talking theoretically about the future. I didn't realize you were thinking about making backward-incompatible changes now, in the name of consistency. We should probably pause and get rough consensus from more of the core team and decide on the timing for introducing these changes, if at all.

}

JsonArrayBuilder savedSearchesBuilder = Json.createArrayBuilder();
List<SavedSearch> savedSearches = savedSearchSvc.findAll();
for (SavedSearch savedSearch : savedSearches) {
JsonObjectBuilder thisSavedSearch = toJson(savedSearch);
savedSearchesBuilder.add(thisSavedSearch);
}
JsonObjectBuilder response = Json.createObjectBuilder();
response.add("saved searches", savedSearchesBuilder);
response.add("savedSearches", savedSearchesBuilder);
return ok(response);
}

@GET
@AuthRequired
@Path("{id}")
public Response show(@PathParam("id") long id) {
public Response show(@Context ContainerRequestContext crc, @PathParam("id") long id) {

AuthenticatedUser superuser = null;
try {
superuser = getRequestAuthenticatedUserOrDie(crc);
} catch (WrappedResponse wr) {
return wr.getResponse();
}
if (superuser == null || !superuser.isSuperuser()) {
return error(Response.Status.UNAUTHORIZED, "Superusers only.");
}

SavedSearch savedSearch = savedSearchSvc.find(id);
if (savedSearch != null) {
JsonObjectBuilder response = toJson(savedSearch);
Expand Down Expand Up @@ -89,7 +116,18 @@ private JsonObjectBuilder toJson(SavedSearch savedSearch) {
}

@POST
public Response add(JsonObject body) {
@AuthRequired
public Response add(@Context ContainerRequestContext crc, JsonObject body) {

AuthenticatedUser superuser = null;
try {
superuser = getRequestAuthenticatedUserOrDie(crc);
} catch (WrappedResponse wr) {
return wr.getResponse();
}
if (superuser == null || !superuser.isSuperuser()) {
return error(Response.Status.UNAUTHORIZED, "Superusers only.");
}

if (body == null) {
return error(BAD_REQUEST, "JSON is expected.");
Expand Down Expand Up @@ -159,7 +197,7 @@ public Response add(JsonObject body) {

try {
SavedSearch persistedSavedSearch = savedSearchSvc.add(toPersist);
return ok("Added: " + persistedSavedSearch);
return ok("Added: " + persistedSavedSearch, Json.createObjectBuilder().add("id", persistedSavedSearch.getId()));
} catch (EJBException ex) {
StringBuilder errors = new StringBuilder();
Throwable throwable = ex.getCause();
Expand All @@ -172,17 +210,31 @@ public Response add(JsonObject body) {
}

@DELETE
@AuthRequired
@Path("{id}")
public Response delete(@PathParam("id") long doomedId) {
boolean disabled = true;
if (disabled) {
return error(BAD_REQUEST, "Saved Searches can not safely be deleted because links can not safely be deleted. See https://github.com/IQSS/dataverse/issues/1364 for details.");
public Response delete(@Context ContainerRequestContext crc, @PathParam("id") long doomedId, @QueryParam("unlink") boolean unlink) {

AuthenticatedUser superuser = null;
try {
superuser = getRequestAuthenticatedUserOrDie(crc);
} catch (WrappedResponse wr) {
return wr.getResponse();
}
if (superuser == null || !superuser.isSuperuser()) {
return error(Response.Status.UNAUTHORIZED, "Superusers only.");
}

SavedSearch doomed = savedSearchSvc.find(doomedId);
if (doomed == null) {
return error(NOT_FOUND, "Could not find saved search id " + doomedId);
}
boolean wasDeleted = savedSearchSvc.delete(doomedId);
boolean wasDeleted;
try {
wasDeleted = savedSearchSvc.delete(doomedId, unlink);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to put this in a command? There's a "CreateSavedSearchCommand"

Copy link
Contributor Author

@luddaniel luddaniel Aug 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @sekmiller, if my understanding is correct Command are used to share the same code between UI and API endpoints.

I note that CreateSavedSearchCommand is not used anymore (it was used in DataversePage.java for UI purposes in the past).
I note that adding Saved Search using POST API is not using CreateSavedSearchCommand.
I note that the code has been initiated back in 2015 and has been for a long time labelled experimental.

It could be prematured to create a Command that could be called from interface. What do you think ?
Sorry, but I'm more "quick win and no refactor while it's not needed" kind of mind.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One reason that I suggested the command was because commands also check for permissions. I didn't notice that this is a superuser only feature - so that permission check wouldn't be available in the command.
We can leave it be for now, but if we ever open it up to other users we might want to revisit the command issue.

} catch (Exception e) {
return error(INTERNAL_SERVER_ERROR, "Problem while trying to unlink links of saved search id " + doomedId);
}

if (wasDeleted) {
return ok(Json.createObjectBuilder().add("Deleted", doomedId));
} else {
Expand All @@ -191,8 +243,20 @@ public Response delete(@PathParam("id") long doomedId) {
}

@PUT
@AuthRequired
@Path("makelinks/all")
public Response makeLinksForAllSavedSearches(@QueryParam("debug") boolean debug) {
public Response makeLinksForAllSavedSearches(@Context ContainerRequestContext crc, @QueryParam("debug") boolean debug) {

AuthenticatedUser superuser = null;
try {
superuser = getRequestAuthenticatedUserOrDie(crc);
} catch (WrappedResponse wr) {
return wr.getResponse();
}
if (superuser == null || !superuser.isSuperuser()) {
return error(Response.Status.UNAUTHORIZED, "Superusers only.");
}

JsonObjectBuilder makeLinksResponse;
try {
makeLinksResponse = savedSearchSvc.makeLinksForAllSavedSearches(debug);
Expand All @@ -205,8 +269,20 @@ public Response makeLinksForAllSavedSearches(@QueryParam("debug") boolean debug)
}

@PUT
@AuthRequired
@Path("makelinks/{id}")
public Response makeLinksForSingleSavedSearch(@PathParam("id") long savedSearchIdToLookUp, @QueryParam("debug") boolean debug) {
public Response makeLinksForSingleSavedSearch(@Context ContainerRequestContext crc, @PathParam("id") long savedSearchIdToLookUp, @QueryParam("debug") boolean debug) {

AuthenticatedUser superuser = null;
try {
superuser = getRequestAuthenticatedUserOrDie(crc);
} catch (WrappedResponse wr) {
return wr.getResponse();
}
if (superuser == null || !superuser.isSuperuser()) {
return error(Response.Status.UNAUTHORIZED, "Superusers only.");
}

SavedSearch savedSearchToMakeLinksFor = savedSearchSvc.find(savedSearchIdToLookUp);
if (savedSearchToMakeLinksFor == null) {
return error(BAD_REQUEST, "Count not find saved search id " + savedSearchIdToLookUp);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,29 +2,28 @@

import edu.harvard.iq.dataverse.Dataset;
import edu.harvard.iq.dataverse.DatasetLinkingDataverse;
import edu.harvard.iq.dataverse.DatasetLinkingServiceBean;
import edu.harvard.iq.dataverse.Dataverse;
import edu.harvard.iq.dataverse.DataverseLinkingDataverse;
import edu.harvard.iq.dataverse.DataverseLinkingServiceBean;
import edu.harvard.iq.dataverse.DvObject;
import edu.harvard.iq.dataverse.DvObjectServiceBean;
import edu.harvard.iq.dataverse.EjbDataverseEngine;
import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser;
import edu.harvard.iq.dataverse.authorization.users.GuestUser;
import edu.harvard.iq.dataverse.engine.command.DataverseRequest;
import edu.harvard.iq.dataverse.search.SearchServiceBean;
import edu.harvard.iq.dataverse.search.SolrQueryResponse;
import edu.harvard.iq.dataverse.search.SolrSearchResult;
import edu.harvard.iq.dataverse.engine.command.exception.CommandException;
import edu.harvard.iq.dataverse.engine.command.impl.DeleteDatasetLinkingDataverseCommand;
import edu.harvard.iq.dataverse.engine.command.impl.DeleteDataverseLinkingDataverseCommand;
import edu.harvard.iq.dataverse.engine.command.impl.LinkDatasetCommand;
import edu.harvard.iq.dataverse.engine.command.impl.LinkDataverseCommand;
import edu.harvard.iq.dataverse.search.SearchException;
import edu.harvard.iq.dataverse.search.SearchFields;
import edu.harvard.iq.dataverse.search.SearchServiceBean;
import edu.harvard.iq.dataverse.search.SolrQueryResponse;
import edu.harvard.iq.dataverse.search.SolrSearchResult;
import edu.harvard.iq.dataverse.search.SortBy;
import edu.harvard.iq.dataverse.util.SystemConfig;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import java.util.logging.Level;
import java.util.logging.Logger;
import jakarta.ejb.EJB;
import jakarta.ejb.Schedule;
import jakarta.ejb.Stateless;
Expand All @@ -39,6 +38,12 @@
import jakarta.persistence.TypedQuery;
import jakarta.servlet.http.HttpServletRequest;

import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import java.util.logging.Level;
import java.util.logging.Logger;

@Stateless
@Named
public class SavedSearchServiceBean {
Expand All @@ -50,6 +55,10 @@ public class SavedSearchServiceBean {
@EJB
DvObjectServiceBean dvObjectService;
@EJB
protected DatasetLinkingServiceBean dsLinkingService;
@EJB
protected DataverseLinkingServiceBean dvLinkingService;
@EJB
EjbDataverseEngine commandEngine;
@EJB
SystemConfig systemConfig;
Expand Down Expand Up @@ -101,11 +110,15 @@ public SavedSearch add(SavedSearch toPersist) {
return persisted;
}

public boolean delete(long id) {
public boolean delete(long id, boolean unlink) throws SearchException, CommandException {
SavedSearch doomed = find(id);
boolean wasDeleted = false;
if (doomed != null) {
System.out.println("deleting saved search id " + doomed.getId());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While we're in this part of the code, should we delete this println? Or maybe convert it to our regular logger and set the level to "fine"?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

if(unlink) {
DataverseRequest dataverseRequest = new DataverseRequest(doomed.getCreator(), getHttpServletRequest());
unLinksForSingleSavedSearch(dataverseRequest, doomed);
}
em.remove(doomed);
em.flush();
wasDeleted = true;
Expand Down Expand Up @@ -240,6 +253,37 @@ public JsonObjectBuilder makeLinksForSingleSavedSearch(DataverseRequest dvReq, S
return response;
}

public void unLinksForSingleSavedSearch(DataverseRequest dvReq, SavedSearch savedSearch) throws SearchException, CommandException {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't love this method name. unLinkForSingleSavedSearch might be better. Or removeLinkForSingleSavedSearch?

I'm not even sure we need "single" in there. I probably added it back in the day for a reason but now it seems verbose. 🤷

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was too focused on mimicry the makeLinksForSingleSavedSearch method name, I renamed it removeLinks and added a java doc.

logger.info("UNLINK SAVED SEARCH (" + savedSearch.getId() + ") START search and unlink process");
Date start = new Date();
Dataverse linkingDataverse = savedSearch.getDefinitionPoint();

SolrQueryResponse queryResponse = findHits(savedSearch);
for (SolrSearchResult solrSearchResult : queryResponse.getSolrSearchResults()) {

DvObject dvObjectThatDefinitionPointWillLinkTo = dvObjectService.findDvObject(solrSearchResult.getEntityId());
if (dvObjectThatDefinitionPointWillLinkTo == null) {
continue;
}

if (dvObjectThatDefinitionPointWillLinkTo.isInstanceofDataverse()) {
Dataverse linkedDataverse = (Dataverse) dvObjectThatDefinitionPointWillLinkTo;
DataverseLinkingDataverse dvld = dvLinkingService.findDataverseLinkingDataverse(linkedDataverse.getId(), linkingDataverse.getId());
if(dvld != null) {
Dataverse dv = commandEngine.submitInNewTransaction(new DeleteDataverseLinkingDataverseCommand(dvReq, linkingDataverse, dvld, true));
}
} else if (dvObjectThatDefinitionPointWillLinkTo.isInstanceofDataset()) {
Dataset linkedDataset = (Dataset) dvObjectThatDefinitionPointWillLinkTo;
DatasetLinkingDataverse dsld = dsLinkingService.findDatasetLinkingDataverse(linkedDataset.getId(), linkingDataverse.getId());
if(dsld != null) {
Dataset ds = commandEngine.submitInNewTransaction(new DeleteDatasetLinkingDataverseCommand(dvReq, linkedDataset, dsld, true));
}
}
}

logger.info("UNLINK SAVED SEARCH (" + savedSearch.getId() + ") total time in ms: " + (new Date().getTime() - start.getTime()));
}

private SolrQueryResponse findHits(SavedSearch savedSearch) throws SearchException {
String sortField = SearchFields.TYPE; // first return dataverses, then datasets
String sortOrder = SortBy.DESCENDING;
Expand Down
Loading
Loading