Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add validations from appsec #562

Merged
merged 10 commits into from
Jan 31, 2024
Merged

Add validations from appsec #562

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
ed8e581
add validations from appsec
HenryL27 Jan 26, 2024
9a6fe41
update changelog
HenryL27 Jan 26, 2024
66d2efd
add validation on number of characters in query context path (1024)
HenryL27 Jan 29, 2024
a8c4c9e
drop max strlen to 1000
HenryL27 Jan 29, 2024
e53c0a1
separate query validation errors; also log them
HenryL27 Jan 29, 2024
4b17c4a
remove a superfluous return lol
HenryL27 Jan 29, 2024
838755f
Apply suggestions from code review
HenryL27 Jan 30, 2024
c03d0b0
use clusterService to get settings rather than environment
HenryL27 Jan 30, 2024
dba691d
import sneakythrows
HenryL27 Jan 30, 2024
75adcac
apply spotless and fix embarassing syntax error
HenryL27 Jan 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 30 additions & 13 deletions .../java/org/opensearch/neuralsearch/processor/rerank/context/QueryContextSourceFetcher.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,17 +27,21 @@
import org.opensearch.search.SearchExtBuilder;

import lombok.AllArgsConstructor;
import lombok.extern.log4j.Log4j2;

/**
* Context Source Fetcher that gets context from the rerank query ext.
*/
@Log4j2
@AllArgsConstructor
public class QueryContextSourceFetcher implements ContextSourceFetcher {

public static final String NAME = "query_context";
public static final String QUERY_TEXT_FIELD = "query_text";
public static final String QUERY_TEXT_PATH_FIELD = "query_text_path";

public static final Integer MAX_QUERY_PATH_STRLEN = 1000;

private final Environment environment;

@Override
Expand Down Expand Up @@ -72,16 +76,7 @@
} else if (ctxMap.containsKey(QUERY_TEXT_PATH_FIELD)) {
// Case "query_text_path": ser/de the query into a map and then find the text at the path specified
String path = (String) ctxMap.get(QUERY_TEXT_PATH_FIELD);
if (!validatePath(path)) {
throw new IllegalArgumentException(
String.format(
Locale.ROOT,
"%s exceeded the maximum path length of %d",
QUERY_TEXT_PATH_FIELD,
MapperService.INDEX_MAPPING_DEPTH_LIMIT_SETTING.get(environment.settings())
)
);
}
validatePath(path);
Map<String, Object> map = requestToMap(searchRequest);
// Get the text at the path
Object queryText = ObjectPath.eval(path, map);
Expand Down Expand Up @@ -125,10 +120,32 @@
return map;
}

private boolean validatePath(final String path) {
private void validatePath(final String path) throws IllegalArgumentException {
if (path == null || path.isEmpty()) {
return true;
return;

Check warning on line 125 in src/main/java/org/opensearch/neuralsearch/processor/rerank/context/QueryContextSourceFetcher.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/neuralsearch/processor/rerank/context/QueryContextSourceFetcher.java#L125 

Added line #L125 was not covered by tests
}
if (path.length() > MAX_QUERY_PATH_STRLEN) {
log.error(String.format(Locale.ROOT, "invalid %s due to too many characters: %s", QUERY_TEXT_PATH_FIELD, path));
throw new IllegalArgumentException(
String.format(
Locale.ROOT,
"%s exceeded the maximum path length of %d characters",
QUERY_TEXT_PATH_FIELD,
MAX_QUERY_PATH_STRLEN
)
);
}
if (path.split("\\.").length > MapperService.INDEX_MAPPING_DEPTH_LIMIT_SETTING.get(environment.settings())) {
log.error(String.format(Locale.ROOT, "invalid %s due to too many nested fields: %s", QUERY_TEXT_PATH_FIELD, path));
throw new IllegalArgumentException(
String.format(

Check warning on line 141 in src/main/java/org/opensearch/neuralsearch/processor/rerank/context/QueryContextSourceFetcher.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/neuralsearch/processor/rerank/context/QueryContextSourceFetcher.java#L139-L141 

Added lines #L139 - L141 were not covered by tests
Locale.ROOT,
"%s exceeded the maximum path length of %d nested fields",
QUERY_TEXT_PATH_FIELD,
MapperService.INDEX_MAPPING_DEPTH_LIMIT_SETTING.get(environment.settings())

Check warning on line 145 in src/main/java/org/opensearch/neuralsearch/processor/rerank/context/QueryContextSourceFetcher.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/neuralsearch/processor/rerank/context/QueryContextSourceFetcher.java#L145 

Added line #L145 was not covered by tests
)
);
}
return path.split("\\.").length <= MapperService.INDEX_MAPPING_DEPTH_LIMIT_SETTING.get(environment.settings());
return;
HenryL27 marked this conversation as resolved.
Show resolved Hide resolved
}
}
26 changes: 24 additions & 2 deletions ...t/java/org/opensearch/neuralsearch/processor/rerank/MLOpenSearchRerankProcessorTests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -230,7 +230,29 @@ public void testRerankContext_whenQueryTextPathIsBadPointer_thenFail() throws IO
.equals(QueryContextSourceFetcher.QUERY_TEXT_PATH_FIELD + " must point to a string field"));
}

public void testRerankContext_whenQueryTextPathIsExceeedinglyLong_thenFail() throws IOException {
public void testRerankContext_whenQueryTextPathIsExceeedinglyManyCharacters_thenFail() throws IOException {
HenryL27 marked this conversation as resolved.
Show resolved Hide resolved
// "eighteencharacters" * 60 = 1080 character string > max len of 1024
setupParams(Map.of(QueryContextSourceFetcher.QUERY_TEXT_PATH_FIELD, "eighteencharacters".repeat(60)));
setupSearchResults();
@SuppressWarnings("unchecked")
ActionListener<Map<String, Object>> listener = mock(ActionListener.class);
processor.generateRerankingContext(request, response, listener);
ArgumentCaptor<Exception> argCaptor = ArgumentCaptor.forClass(Exception.class);
verify(listener, times(1)).onFailure(argCaptor.capture());
assert (argCaptor.getValue() instanceof IllegalArgumentException);
assert (argCaptor.getValue()
.getMessage()
.equals(
String.format(
Locale.ROOT,
"%s exceeded the maximum path length of %d characters",
QueryContextSourceFetcher.QUERY_TEXT_PATH_FIELD,
QueryContextSourceFetcher.MAX_QUERY_PATH_STRLEN
)
));
}

public void textRerankContext_whenQueryTextPathIsExceeedinglyDeeplyNested_ThenFail() throws IOException {
HenryL27 marked this conversation as resolved.
Show resolved Hide resolved
setupParams(Map.of(QueryContextSourceFetcher.QUERY_TEXT_PATH_FIELD, "a.b.c.d.e.f.g.h.i.j.k.l.m.n.o.p.q.r.s.t.w.x.y.z"));
setupSearchResults();
@SuppressWarnings("unchecked")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this annotation? I don't see ActionListener<Map<String, Object>> listener = mock(ActionListener.class); is causing any warning in my local.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

¯_(ツ)_/¯. I get a warning when I remove it. Makes sense since it's a typecast from generic action listener to action listener for that map thingy.

Expand All @@ -244,7 +266,7 @@ public void testRerankContext_whenQueryTextPathIsExceeedinglyLong_thenFail() thr
.equals(
String.format(
Locale.ROOT,
"%s exceeded the maximum path length of %d",
"%s exceeded the maximum path length of %d nested fields",
QueryContextSourceFetcher.QUERY_TEXT_PATH_FIELD,
MapperService.INDEX_MAPPING_DEPTH_LIMIT_SETTING.get(environment.settings())
)
Expand Down
Loading