[Ingest Management]: Error 'Unable to initialize Ingest Manager: Bad Request' is displayed on clicking 'Ingest Manager' link on 8.0.0-SNAPSHOT Kibana #77133
Assignees
Labels
bug
Fixes for quality problems that affect the customer experience
impact:high
Addressing this issue will have a high level of impact on the quality/strength of our product.
Team:Fleet
Team label for Observability Data Collection Fleet team
Comments
ghost
self-assigned this
|
@karanbirsingh-qasource Please review the bug. |
ghost
added
bug
|
Pinging @elastic/ingest-management (Team:Ingest Management) |
ghost
added
the
impact:critical
ghost
added
impact:high
ghost
mentioned this issue
|
Reviewed and assigned to @EricDavisX |
|
looks to me that this will happen every time the user opens Kibana again, the first time they access Ingest Manager the setup api gives a 500, then it returns ok the subsequent times: I agree with High impact therefore. Assigning to PH to see who can take a look |
|
@jfsiii Can you take the lead to reproduce that issue? |
|
Bringing Slack convo here, I can reproduce in Cloud and am looking into this now. I'm taking snapshots & HAR files and will try to reproduce locally. |
|
Update: TL;DR: can repro on Cloud but I see nothing in logs that gives a clue what's happening. Next snapshot should have better detail about the source. Is there a way to determine which commits are in a snapshot/build? I want to see if certain PRs are included. staging instancekibana: https://1aef124d8bc9424ea7f6e2ca049ca7e0.us-central1.gcp.foundit.no:9243/app/home On plugin load in browserPOST /ingest_manager/setup fails w/500 & POST /ingest_manager/setup fails with 502
POST /ingest_manager/setup response
Kibana logs OK And looking at ES all the appropriate "adding *" messages appear to be in The "Stopping all plugins" in the Kibana logs 30 seconds before the "Internal Server Error" from our request seems noteworthy. However, I later triggered the 500 a few times and there were no other "Stopping ..." messages.
There are some "Request error, retrying" messages which seem to point to ES connectivity issues, but they aren't present near all the errors, so ... I can't find any interesting activity on ES around the time of the failures. See 500's at 16:36 & 16:38 here But nothing on ES at that time |
|
I twice tried to create a deploy from an 8.x snapshot this morning but both failed. The ES instance wouldn't start (due to an issue with the plan?). It's also a company-wide holiday today so I'll check back on Monday and see how things look. |
On the Kibana side, the top level |
|
@skh thanks, I should have resolved that question. @EricDavisX gave me the tip in Slack and it worked great. Latest update: I cannot build an 8.x snapshot on staging. ES is failing with a bad plan. I've pinged the Cloud Slack channel with the details and I will update again later with any changes. |
|
We found a bug. The Cloud team is aware and tracking in a ticket. I asked about expediting a fix. Will update more later. |
|
Cloud is rolling back the source of the issue. The expectation is that we'll be able to deploy snapshots again today and I can resume investigating. |
|
The rollback is complete, but not yet available in all zones. AWS eu-central definitely works, though
I deployed an 8.x snapshot (f5a9fecfc854456387d94cdedcfb8a94) and got a more detailed error
This looks to be a forwarded 400 response from Elasticsearch. Pinging @elastic/endpoint-management for visibility and any suggestions. I'll post in Slack as well. |
|
I tried this: Reproduce locally
Reproduce locally by connecting local Kibana to Cloud ES
Comment out transform code locally, connect local Kibana to fresh Cloud ES
The Fleet page remains spinning in the UI Some logs from the console |
|
Sorry for the short message now, but I don't have much time until tomorrow. I added a bunch of logging in the install path (mostly install failure |
|
@jfsiii |
|
I added some more Here's logs from two scenarios I ran local Kibana / Cloud ES (❌ like OP)local Kibana logs [1]local Kibana / local ES ✅From scratch (no ES data): Kibana [2]From scratch (no ES data): ES [3]Restart Kibana (ES remains up w/ packages installed): [4]cc @neptunian & @nnamdifrankie in the hopes these give more insight into what's happening |
|
Update: We have two changes pending which should fix the error on load.
|
|
An additional option to get testing going is to run the following curl command after your 8.0 snapshot build completes and you get your Kibana URL. Do the below BEFORE going to the Ingest page.
Open a terminal, fill in the above, run this command. After that Ingest should load. curl -X POST -H 'kbn-xsrf: xxx' --user <user>:<pass> \
-H "Content-Type: application/json" -d '{"force" : true}' \
<kibana-url>/api/ingest_manager/epm/packages/endpoint-0.15.0The above is only necessary until we get another 8.0 SNAPSHOT build. The root issue should be fixed there. |
|
I just tried tried this on a few deployments, and it seems that it only works on an instance which has never tried to setup before. That makes sense now but just confirming (for myself as much as anyone). Existing deployment -
|
Yes like I mentioned in our chat. This is because of how Ingest Manager currently handles downgrades where it leave the index templates even when you remove the package. I believe there is some marked difference between 0.15.0 to 0.16.0, which I think has to do with changes to datastream that prevents the downgrade after setup. However once we fix the build we should not be looking back after setup. |
|
@rahulgupta-qasource Summary: we believe this has been fixed by #77640. There are some unrelated issues on staging at the moment which require a workaround to deploy. The Kibana 8.x SNAPSHOT builds are failing on the default 1GB memory. However, if you increase that to 2GB, they will behave as expected Select 8.x
Click Customize
Increase to 2GB
|
|
I concur - I see this fixed in latest 8.0 snapshot. shall we close it out? |
|
Hi @jfsiii / @EricDavisX Thank you for sharing the feedback. On increasing default 1GB memory to 2GB under 'Kibana 1 configuration', user is successfully able to deploy 8.0.0-SNAPSHOT Kibana cloud environment on staging cloud platform and no error is displayed on clicking left hand menu 'Ingest Manager' link (Fixed). Screenshot:
Hence, we are closing this bug. |
|
Bug Conversion: Below 01 Testcase already exists for this ticket: |
This issue was closed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Kibana version:
Kibana: 8.0.0-SNAPSHOT
Elasticsearch version:
Elasticsearch: 8.0.0-SNAPSHOT
Host OS and Browser version:
Windows 10 , All
Original install method (e.g. download page, yum, from source, etc.):
From 8.0.0-SNAPSHOT on staging cloud environment
Description
[Ingest Management]: Error 'Unable to initialize Ingest Manager: Bad Request' is displayed on clicking 'Ingest Manager' link on 8.0.0-SNAPSHOT Kibana
Preconditions
Steps to Reproduce
Test data
N/A
Impacted Test case id
N/A
Actual Result
[Ingest Management]: Error 'Unable to initialize Ingest Manager: Bad Request' is displayed on clicking 'Ingest Manager' link on 8.0.0-SNAPSHOT Kibana.
Expected Result
[Ingest Management]: User should be successfully able to access Ingest Manager application on 8.0.0-SNAPSHOT Kibana.
What's working
What's not working
Screenshot
Logs
Error: 'Failed to load resource: the server responded with a status of 500 ()' is displayed in browser console logs.
The text was updated successfully, but these errors were encountered: