Error 429 still exists

[sherifkozman]

We are running off commit : 691f9f5 while the rate limiting error 429 still persists.

[Edit by @sethvargo - removed a screenshot that contained identifiable information]

[sethvargo]

Hi @sherifkozman - can you share the server logs that correspond to this request? Since, the reset time is Jan 1, 1970 (time.At(0)), there's likely an error connecting to Redis which you should see in the server or apiserver logs.

[sethvargo]

Also, I deleted your screenshot. It had your cookie value and domain, which someone could have easily used to login and access the server as you.

[sherifkozman]

@sethvargo this one is actually on the "server" service not apiserver

[sethvargo]

That's the request log. This would be a log a few lines before or after. It should be named ratelimit or similar.

[sherifkozman]

There is nothing before or after, if there a debug env we should enable ?

The only thing before is this set of logs, do we need to update the server code to anything after August 17th ?

[sethvargo]

Would it be possible to give me access to your logs for this project temporarily? github-username at google dot com. It's challenging to debug via screenshots 😄 . Did you deploy with LOG_DEBUG=true and DB_DEBUG=true?

[sethvargo]

Responded on chat with a few suggestions.

[sethvargo]

Just an update on this since @sherifkozman and I are chatting offline. We believe the issue is related to scale up/down events not properly closing connections and then, eventually, exhausting the pool. We're going to investigate some service-level changes and will report back.

[mikehelmick]

#391 should fix this issue.

/close

[google-oss-robot]

@mikehelmick: Closing this issue.

In response to this:

#391 should fix this issue.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.