Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add testing interface, including a set of capabilities to tests for #6687

Merged
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
a60bdec
Add testing interface, including a set of capabilities to tests for
DominikVoigt Jul 13, 2020
4881f7b
Merge branch 'feat/add-capability-tests-for-fetchers' of https://gith…
koppor Jul 13, 2020
7c13275
Remove all interfaces
koppor Jul 13, 2020
c7795e9
Squashed 'src/main/resources/csl-styles/' changes from a995c63a0a..bf…
Jul 15, 2020
a89db08
Merge commit 'c7795e9124d55908b8f074c475d62229972a4edc'
Jul 15, 2020
5ec7ef6
Merge branch 'feat/add-capability-tests-for-fetchers' of https://gith…
DominikVoigt Jul 15, 2020
ea4b67f
Remove empty lines in tests
DominikVoigt Jul 15, 2020
0950cb6
Correct SpringerFetcherTest
DominikVoigt Jul 15, 2020
6bffc68
Rename AdvancedFetcher and AdvancedSearchConfig.java accordingly to c…
DominikVoigt Jul 15, 2020
f5f1b5b
Add documentation to the default field
DominikVoigt Jul 15, 2020
7e81cca
Modify AdvancedSearchBasedParserFetcher interface to be in line with …
DominikVoigt Jul 15, 2020
db19e60
Adapt test of getAdvancedURL
DominikVoigt Jul 15, 2020
4065fa5
Integrate change requests.
DominikVoigt Jul 15, 2020
7b9e85f
Integrate change requests.
DominikVoigt Jul 17, 2020
d2a79fd
Merge remote-tracking branch 'upstream/master' into feat/add-capabili…
DominikVoigt Jul 19, 2020
9673b6c
Start Working on format conversion
DominikVoigt Jul 19, 2020
e4b1ad1
Continue integrate format conversion
DominikVoigt Jul 20, 2020
de62312
Continue integrate format conversion.
DominikVoigt Jul 20, 2020
bb089f4
Merge remote-tracking branch 'upstream/master' into feat/add-capabili…
DominikVoigt Jul 20, 2020
988a275
Get it to build. Start correcting tests.
DominikVoigt Jul 21, 2020
a2a26e2
Work on correcting tests.
DominikVoigt Jul 21, 2020
6f482a4
Finalize format conversion changes.
DominikVoigt Jul 24, 2020
a746197
Merge remote-tracking branch 'upstream/master' into feat/add-capabili…
DominikVoigt Jul 24, 2020
f675e13
Add ADR.
DominikVoigt Jul 24, 2020
9054011
Remove comment.
DominikVoigt Jul 24, 2020
c8ead6b
Revert "Squashed 'src/main/resources/csl-styles/' changes from a995c6…
DominikVoigt Jul 24, 2020
1eddd44
Revert "Revert "Squashed 'src/main/resources/csl-styles/' changes fro…
DominikVoigt Jul 24, 2020
92c05be
Update ADRs
DominikVoigt Jul 26, 2020
84a9b53
Update ADR. Remove import.
DominikVoigt Jul 27, 2020
8fa03d5
Remove unused import.
DominikVoigt Jul 27, 2020
b86f34e
Merge remote-tracking branch 'upstream/master' into feat/add-capabili…
DominikVoigt Jul 27, 2020
c694bc1
Remove whitespaces and redundant paragraph from ADR.
DominikVoigt Jul 27, 2020
e1b0bf8
Move format conversion from Fetchers into ImportCleanup.
DominikVoigt Jul 28, 2020
d424f88
Revert incorrect change.
DominikVoigt Jul 28, 2020
d6eb086
Remove debugging print statement.
DominikVoigt Jul 29, 2020
f08184e
Integrate requested changes.
DominikVoigt Aug 2, 2020
8f3fe0f
Integrate requested changes.
DominikVoigt Aug 2, 2020
056cc93
Change test to use ImportCleanup.
DominikVoigt Aug 2, 2020
e0fc10a
Merge remote-tracking branch 'upstream/master' into feat/add-capabili…
DominikVoigt Aug 2, 2020
1c87c1d
Remove unused bibentry format
DominikVoigt Aug 2, 2020
190414f
Integrate requested changes
DominikVoigt Aug 2, 2020
a37d29e
Inline variable
DominikVoigt Aug 2, 2020
4b6b54b
Modify ADR
DominikVoigt Aug 2, 2020
b5d9369
Fix compile error
DominikVoigt Aug 2, 2020
cf2ff45
Change ADR title
DominikVoigt Aug 2, 2020
37293c6
Inline some variables
DominikVoigt Aug 2, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion config/checkstyle/checkstyle.xml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,9 @@

<!-- BeforeExecutionFileFilters is required for sources that are based on java14 -->
<module name="BeforeExecutionExclusionFileFilter">
<property name="fileNamePattern" value="AuthorAndsReplacer.java|EntryTypeView.java|MainTableFieldValueFormatter.java|Ordinal.java" />
<property
name="fileNamePattern"
value="AuthorAndsReplacer.java|EntryTypeView.java|IEEE.java|MainTableFieldValueFormatter.java|Ordinal.java"/>
</module>

<module name="SuppressionFilter">
Expand Down
38 changes: 38 additions & 0 deletions src/main/java/org/jabref/logic/importer/AdvancedFetcher.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
package org.jabref.logic.importer;

import java.io.IOException;
import java.io.InputStream;
import java.util.List;

import org.jabref.logic.importer.fetcher.AdvancedSearchConfig;
import org.jabref.logic.net.URLDownload;
import org.jabref.model.entry.BibEntry;

/**
* This interface allows SearchBasedParserFetcher fetchers to test their corresponding
* library APIs for their advanced search options, e.g. search in the "title" field.
*/
public interface AdvancedFetcher extends SearchBasedParserFetcher {
DominikVoigt marked this conversation as resolved.
Show resolved Hide resolved

/**
* This method is used to send queries with advanced URL parameters.
* This method is necessary as the performSearch method does not support certain URL parameters that are used for
* fielded search, such as a title, author, or year parameter.
*
* @param advancedSearchConfig the search config defining all fielded search parameters
*/
default List<BibEntry> performAdvancedSearch(AdvancedSearchConfig advancedSearchConfig) throws FetcherException {
DominikVoigt marked this conversation as resolved.
Show resolved Hide resolved
try (InputStream stream = getAdvancedURLDownload(advancedSearchConfig).asInputStream()) {
List<BibEntry> fetchedEntries = getParser().parseEntries(stream);
fetchedEntries.forEach(this::doPostCleanup);
return fetchedEntries;
} catch (IOException e) {
// TODO: Catch HTTP Response 401/403 errors and report that user has no rights to access resource
throw new FetcherException("A network error occurred", e);
} catch (ParseException e) {
throw new FetcherException("An internal parser error occurred", e);
}
}

URLDownload getAdvancedURLDownload(AdvancedSearchConfig advancedSearchConfig);
DominikVoigt marked this conversation as resolved.
Show resolved Hide resolved
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
package org.jabref.logic.importer.fetcher;

import java.util.Objects;
import java.util.Optional;

import org.jabref.model.strings.StringUtil;

public class AdvancedSearchConfig {
private final String defaultField;
private final String author;
private final String title;
private final Integer fromYear;
private final Integer toYear;
private final String journal;

private AdvancedSearchConfig(String defaultField, String author, String title, Integer fromYear, Integer toYear, String journal) {
this.defaultField = defaultField;
this.author = author;
this.title = title;
this.fromYear = fromYear;
this.toYear = toYear;
this.journal = journal;
}

public Optional<String> getDefaultField() {
DominikVoigt marked this conversation as resolved.
Show resolved Hide resolved
return Optional.ofNullable(defaultField);
}

public Optional<String> getAuthor() {
return Optional.ofNullable(author);
}

public Optional<String> getTitle() {
return Optional.ofNullable(title);
}

public Optional<Integer> getFromYear() {
return Optional.ofNullable(fromYear);
}

public Optional<Integer> getToYear() {
return Optional.ofNullable(toYear);
}

public Optional<String> getJournal() {
return Optional.ofNullable(journal);
}

public static AdvancedSearchConfigBuilder builder() {
return new AdvancedSearchConfigBuilder();
}

public static class AdvancedSearchConfigBuilder {
private String defaultField;
private String author;
private String title;
private String journal;
private Integer fromYear;
private Integer toYear;

public AdvancedSearchConfigBuilder() {
}

public AdvancedSearchConfigBuilder defaultField(String defaultField) {
if (Objects.requireNonNull(defaultField).isBlank()) {
throw new IllegalArgumentException("Parameter must not be blank");
}
this.defaultField = defaultField;
return this;
}

public AdvancedSearchConfigBuilder author(String author) {
if (Objects.requireNonNull(author).isBlank()) {
throw new IllegalArgumentException("Parameter must not be blank");
}
this.author = author;
return this;
}

public AdvancedSearchConfigBuilder title(String title) {
if (Objects.requireNonNull(title).isBlank()) {
throw new IllegalArgumentException("Parameter must not be blank");
}
this.title = title;
return this;
}

public AdvancedSearchConfigBuilder fromYear(Integer fromYear) {
this.fromYear = Objects.requireNonNull(fromYear);
return this;
}

public AdvancedSearchConfigBuilder toYear(Integer toYear) {
this.toYear = Objects.requireNonNull(toYear);
return this;
}

public AdvancedSearchConfigBuilder journal(String journal) {
if (Objects.requireNonNull(journal).isBlank()) {
throw new IllegalArgumentException("Parameter must not be blank");
}
this.journal = journal;
return this;
}

/**
* Instantiates the AdvancesSearchConfig from the provided Builder parameters
* If all text fields are empty an empty optional is returned
*
* @return AdvancedSearchConfig instance with the fields set to the values defined in the building instance.
* @throws IllegalStateException An IllegalStateException is thrown in case all text search fields are empty.
* See: https://softwareengineering.stackexchange.com/questions/241309/builder-pattern-when-to-fail/241320#241320
*/
public AdvancedSearchConfig build() throws IllegalStateException {
if (textSearchFieldsAreEmpty()) {
throw new IllegalStateException("At least one text field has to be set");
}
return new AdvancedSearchConfig(defaultField, author, title, fromYear, toYear, journal);
}

private boolean textSearchFieldsAreEmpty() {
return StringUtil.isBlank(defaultField) && StringUtil.isBlank(title) && StringUtil.isBlank(author) && StringUtil.isBlank(journal);
}
}
}
2 changes: 1 addition & 1 deletion src/main/java/org/jabref/logic/importer/fetcher/ArXiv.java
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,7 @@ private List<ArXivEntry> searchForEntries(String searchQuery) throws FetcherExce
}

private List<ArXivEntry> queryApi(String searchQuery, List<ArXivIdentifier> ids, int start, int maxResults)
throws FetcherException {
throws FetcherException {
Document result = callApi(searchQuery, ids, start, maxResults);
List<Node> entries = XMLUtil.asList(result.getElementsByTagName("entry"));

Expand Down
41 changes: 28 additions & 13 deletions src/main/java/org/jabref/logic/importer/fetcher/IEEE.java
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
import java.util.stream.Collectors;

import org.jabref.logic.help.HelpFile;
import org.jabref.logic.importer.AdvancedFetcher;
import org.jabref.logic.importer.FulltextFetcher;
import org.jabref.logic.importer.ImportFormatPreferences;
import org.jabref.logic.importer.Parser;
Expand All @@ -42,7 +43,7 @@
*
* @implNote <a href="https://developer.ieee.org/docs">API documentation</a>
*/
public class IEEE implements FulltextFetcher, SearchBasedParserFetcher {
public class IEEE implements FulltextFetcher, SearchBasedParserFetcher, AdvancedFetcher {

private static final Logger LOGGER = LoggerFactory.getLogger(IEEE.class);
private static final String STAMP_BASE_STRING_DOCUMENT = "/stamp/stamp.jsp?tp=&arnumber=";
Expand All @@ -67,18 +68,10 @@ private static BibEntry parseJsonRespone(JSONObject jsonEntry, Character keyword
BibEntry entry = new BibEntry();

switch (jsonEntry.optString("content_type")) {
case "Books":
entry.setType(StandardEntryType.Book);
break;
case "Conferences":
entry.setType(StandardEntryType.InProceedings);
break;
case "Courses":
entry.setType(StandardEntryType.Misc);
break;
default:
entry.setType(StandardEntryType.Article);
break;
case "Books" -> entry.setType(StandardEntryType.Book);
case "Conferences" -> entry.setType(StandardEntryType.InProceedings);
case "Courses" -> entry.setType(StandardEntryType.Misc);
default -> entry.setType(StandardEntryType.Article);
}

entry.setField(StandardField.ABSTRACT, jsonEntry.optString("abstract"));
Expand Down Expand Up @@ -236,4 +229,26 @@ public String getName() {
public Optional<HelpFile> getHelpPage() {
return Optional.of(HelpFile.FETCHER_IEEEXPLORE);
}

@Override
public URLDownload getAdvancedURLDownload(AdvancedSearchConfig advancedSearchConfig) {
URLDownload urlDownload;
try {
URIBuilder uriBuilder = new URIBuilder("https://ieeexploreapi.ieee.org/api/v1/search/articles");
uriBuilder.addParameter("apikey", API_KEY);
advancedSearchConfig.getDefaultField().ifPresent(defaultField -> uriBuilder.addParameter("querytext", defaultField));
advancedSearchConfig.getAuthor().ifPresent(author -> uriBuilder.addParameter("author", author));
advancedSearchConfig.getTitle().ifPresent(articleTitle -> uriBuilder.addParameter("article_title", articleTitle));
advancedSearchConfig.getJournal().ifPresent(journalTitle -> uriBuilder.addParameter("publication_title", journalTitle));
advancedSearchConfig.getFromYear().map(String::valueOf).ifPresent(year -> uriBuilder.addParameter("start_year", year));
advancedSearchConfig.getToYear().map(String::valueOf).ifPresent(year -> uriBuilder.addParameter("end_year", year));

URLDownload.bypassSSLVerification();
urlDownload = new URLDownload(uriBuilder.build().toURL());
} catch (URISyntaxException | MalformedURLException ex) {
LOGGER.error("Error creating URL.", ex);
throw new IllegalStateException("Error during creation of URL.", ex);
DominikVoigt marked this conversation as resolved.
Show resolved Hide resolved
}
return urlDownload;
}
}
Loading