Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JWA Customization: Kubecost Response Error #101

Closed
saffaalvi opened this issue Mar 7, 2022 · 5 comments
Closed

JWA Customization: Kubecost Response Error #101

saffaalvi opened this issue Mar 7, 2022 · 5 comments
Labels
size/S ~1 day

Comments

@saffaalvi
Copy link
Contributor

When the kubecost changes were brought into 1.3, on the dev cluster there is a 503 error from the cost response.
upstream connect error or disconnect/reset before headers. reset reason: connection termination
From what I've seen online, this might have something to do with Istio.

Not sure if the kubecost service is even supposed to be working on dev?

cc @sylus

@sylus
Copy link
Member

sylus commented Mar 7, 2022

It seems someone manually deployed a bad image?

I set the SHA back to 813f13a and it works fine now

@Jose-Matsuda
Copy link
Contributor

We manually deployed an image that we built on Saffa's branch to check and fix for a different error that appeared when opening up the console.

@sylus
Copy link
Member

sylus commented Mar 8, 2022

Ok i finally found some time ^_^ so oddly i wasn't seeing it earlier in the day but the container image is the same as was reverted to and i see this problem now.

I put the SHA for the latest commit 708d307

The service is up for cost-analyzer and the end points are correct so this requires some further investigation.

@sylus
Copy link
Member

sylus commented Mar 8, 2022

This was a network policy that needed to be updated it should be fixed now.

@sylus sylus closed this as completed Mar 8, 2022
@zachomedia
Copy link
Contributor

https://github.com/StatCan/aaw-network-policies/commit/af17a3c3c7417cd91acab0cbc1447535c8f406d0 was the change (the labels changed on the jupyter web app pod in Kubeflow 1.3)

@Jose-Matsuda Jose-Matsuda added the size/S ~1 day label Mar 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size/S ~1 day
Projects
None yet
Development

No branches or pull requests

4 participants