"Advanced search" returning "Page index out of bounds for available matches" error

[corridordigital]

I'm using Amundsen with neo4j and ES, with a standard docker-compose installation.
The data is ingested in neo4j (visible through browser) & Elasticsearch (visible through API calls), and I can find it and diplay it using the homepage searchbar.

Expected Behavior

Advanced Search must work and return results when there are assets that match the search keywords. This actually used to work with older versions (the one on the master branch, for example).

Using "*" as a keyword in the Source bar should return every asset.

Current Behavior

I get a Page index out of bounds for available matches error everytime.

Possible Solution

The error is raised by the frontend service (here).

Steps to Reproduce

1. Standard docker installation
2. Data ingestion from a PostgreSQL database
3. Using the search bar on the homepage works well
4. Doing an "Advanced Search" returns the error

Screenshots (if appropriate)

Homepage search :

Advanced search :

Context

Issue appeared with newer versions of Amundsen. It happens evey single time.

Your Environment

• Amunsen version used: newest (main branch)
• Data warehouse stores:
• Deployment (k8s or native): native (docker-compose on ubuntu)

[boring-cyborg]

Thanks for opening your first issue here!

[corridordigital]

Reproduced the issue on Debian 10 (Server).
I am using miniconda3 for the Python environment.

The issue happens with Chrome & Firefox as browsers.

[corridordigital]

I've been comparing amundsenfrontend DEBUG logs between the newest version (defective), and an older fully functioning version (3.6.0).
I've been focusing on the logs generated by Advanced Search queries, and more specifically on the data received by the frontend service for displaying. The data is stored in JSON strings, and the keyword_args_json looks like this :

{
    "filters": {
        "database": [
            "*"
        ]
    },
    "search_term": "",
    "page_index": 0,
    "search_type": "update_filter"
}', output='{
    "search_term": "",
    "msg": "Success",
    "tables": {
        "page_index": 0,
        "results": [
            {
               <METADATA ABOUT MY DB>
        ],
        "total_results": null
    },
    "status_code": 200
}

I've noticed something very suprising in this data : total_results": null, which is wrongin my case, as I can see in the results JSON. In y older version, this has a non-null value, equal to the number of search hits.

total_results happens to be used to raise the Page index out of bounds for available matches error, in here :

if (page_index < 0 || startIndex > total_results) {
     return (
       <div className="search-list-container">
         <div className="search-error body-placeholder">
           {PAGE_INDEX_ERROR_MESSAGE}
         </div>
       </div>
     );
   }

[mgorsk1]

which version of ES and Amundsen you are using? Might be some mismatch after introducing support for ES7

cc @verdan

[corridordigital]

which version of ES and Amundsen you are using? Might be some mismatch after introducing support for ES7

cc @verdan

Hi,
It's a fresh install, with the lastest versions :

• elasticsearch:7.13.3
• Amundsen :
  • frontend : 3.12
  • search : 2.10
  • metadata : 3.9
  • builder : 6.0.2
  • common : 0.19.0
  • gremlin : 0.0.12
  • rds : 0.0.5

[corridordigital]

@mgorsk1 I can't identify the exact origin of the problem, but I can provide logs & evidence.

[corridordigital]

I tried using older (6) and newer versions of ES through changing my docker-amundsen.yml file. They don't seem to be compatible with the other services.

[dewes]

Also having same problem, fresh git clone, runing it with docker-compose -f docker-amundsen.yml up, executing sample_data_loader.py. (After facing same problem on #1442).

Data is on Neo4j and in Elasticsearch as results shows up on direct query:

curl -X GET "http://localhost:5001/search?query_term=test"
{"results": [{"key": "hive://gold.test_schema/test_table1", "database": "hive", "display_name": "test_schema.test_table1", "total_usage": 1330, "name": "test_table1", "badges": [{"tag_name": "beta"}], "schema": "test_schema", "column_names": ["col1", "col2", "col3", "col4", "col5"], "column_descriptions": [], "description": "1st test table", "programmatic_descriptions": ["### Quality Report:\n--- \nIpsus enom. Ipsus enom ipsus lorenum.\n---\n", "Size: 50T\n\nMonthly Cost: $5000"], "cluster": "gold", "schema_description": "test schema description", "tags": [{"tag_name": "tag1"}, {"tag_name": "tag2"}, {"tag_name": "low_quality"}, {"tag_name": "expensive"}], "last_updated_timestamp": 1570230473}, {"key": "dynamo://gold.test_schema/test_table2", "database": "dynamo", "display_name": "test_schema.test_table2", "total_usage": 520, "name": "test_table2", "badges": [{"tag_name": "npi"}, {"tag_name": "json"}], "schema": "test_schema", "column_names": ["col4", "col3", "col2", "col1"], "column_descriptions": [], "description": "2nd test table", "programmatic_descriptions": ["Size: 1T\n\nMonthly Cost: $50"], "cluster": "gold", "schema_description": null, "tags": [{"tag_name": "cheap"}, {"tag_name": "recommended"}], "last_updated_timestamp": 1070230473}, {"key": "hive://gold.test_schema/test_table3", "database": "hive", "display_name": "test_schema.test_table3", "total_usage": 0, "name": "test_table3", "badges": [], "schema": "test_schema", "column_names": ["col1"], "column_descriptions": [], "description": "3rd test", "programmatic_descriptions": [], "cluster": "gold", "schema_description": "test schema description", "tags": [{"tag_name": "needs_documentation"}], "last_updated_timestamp": null}, {"key": "hive://gold.test_schema/test_view1", "database": "hive", "display_name": "test_schema.test_view1", "total_usage": 0, "name": "test_view1", "badges": [], "schema": "test_schema", "column_names": ["col1"], "column_descriptions": [], "description": "1st test view", "programmatic_descriptions": [], "cluster": "gold", "schema_description": "test schema description", "tags": [{"tag_name": "tag1"}], "last_updated_timestamp": null}, {"key": "delta://gold.test_schema/delta_test_table", "database": "delta", "display_name": "test_schema.delta_test_table", "total_usage": 0, "name": "delta_test_table", "badges": [], "schema": "test_schema", "column_names": ["col2", "col1"], "column_descriptions": [], "description": "test table for delta", "programmatic_descriptions": [], "cluster": "gold", "schema_description": null, "tags": [{"tag_name": "delta"}], "last_updated_timestamp": null}, {"key": "hive://gold.test_schema/test's_table4", "database": "hive", "display_name": "test_schema.test's_table4", "total_usage": 0, "name": "test's_table4", "badges": [], "schema": "test_schema", "column_names": [], "column_descriptions": [], "description": "4th test", "programmatic_descriptions": [], "cluster": "gold", "schema_description": "test schema description", "tags": [{"tag_name": "needs_documentation"}], "last_updated_timestamp": null}]}

And on Neo4j:

On frontend it shows up while writing search terms, like:

But nothing on results search page, just "Page index out of bounds for available matches"

[tmihordea-gpsw]

I am also getting the same error as #1386 (comment)
It looks related to this breaking change in elasticsearch 7: https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html

[dfall-gh]

Hi everyone,

@corridordigital : same issue for me with same version of Amundsen (Tried on Firefox and Chrome).

Indeed, I noticed that the error was raised here:

if (page_index < 0 || startIndex > total_results) {
     return (
       <div className="search-list-container">
         <div className="search-error body-placeholder">
           {PAGE_INDEX_ERROR_MESSAGE}
         </div>
       </div>
     );
   }

In the Amundsen logs (docker compose launched in debug mode), I saw results for my advanced search.

From the browser Debug Console I got :

• total_results : null (despite the fact we can see results in the JSON)
• page_index : 0
• startIndex : 1

So
if (page_index < 0 || startIndex > total_results) is TRUE.

So this bug seems to appear because of total_results with NULL value.

Thank everyone for your help on this.

[verdan]

I believe this is because of the older version of Amundsen search. have you guys tried this using local docker-compose?, as I can see the version on the search package on docker hub is a couple of months older.

In any way, I will release a new version of the search, that should fix this problem for you all.

[verdan]

#1445

[corridordigital]

Hi @verdan 👋
I tested your last commit from the vm-versions-upgrade branch, and the search works great !
Can't wait to see it merged on main.
Thanks a lot,

[verdan]

sweet! merged....

[dewes]

Also confirm it's working now! Thanks for the very fast patch!

[dfall-gh]

Works also fine for me !

Thank your for the fix , really appreciated.