Kubernetes@V1 - Fails to query GitHub API

[oskarm93]

Environment

• Server - Azure Pipelines

  • rcss-willistowerswatson - PPM - Release ID 2355

• Agent - Private:

  • [AGENT_VERSION] --> [2.148.0] ON WS2016

Issue Description

We are using V1 of Kubernetes task. Some days this task fails when we have too many builds / releases in a time period. The Kubectl command being run does not make a difference.
The command attempts to look for latest release of kubectl. This is despite:

1. Kubectl already being on the PATH.
2. Kubectl already downloaded to _tools directory.
3. Having a separate task to install Helm / Kubernetes tool before that (which we swapped for static install).

2019-03-11T18:40:49.1361118Z ##[debug][GET]https://api.github.com/repos/kubernetes/kubernetes/releases?page=0&per_page=100
2019-03-11T18:40:49.6391169Z ##[error]response.body.forEach is not a function

It does not seem to handle bad output from the Github API. After sending a GET to this URL above right after the failure, it turns out we hit an API request quota for unauthenticated requests. As an organization we have a single outbound IP address, so that contributes to the problem. Too many builds in 1 team project will fail another team's builds / releases and vice-versa.

Task logs

2019-03-11T18:40:48.6816155Z ##[debug]Evaluating condition for step: 'Log out of AKS'
2019-03-11T18:40:48.6817155Z ##[debug]Evaluating: always()
2019-03-11T18:40:48.6817511Z ##[debug]Evaluating always:
2019-03-11T18:40:48.6817819Z ##[debug]=> True
2019-03-11T18:40:48.6818249Z ##[debug]Result: True
2019-03-11T18:40:48.6818677Z ##[section]Starting: Log out of AKS
2019-03-11T18:40:48.6825780Z ==============================================================================
2019-03-11T18:40:48.6826159Z Task         : Deploy to Kubernetes
2019-03-11T18:40:48.6826375Z Description  : Deploy, configure, update your Kubernetes cluster in Azure Container Service by running kubectl commands.
2019-03-11T18:40:48.6826574Z Version      : 1.1.24
2019-03-11T18:40:48.6826733Z Author       : Microsoft Corporation
2019-03-11T18:40:48.6826965Z Help         : [More Information](https://go.microsoft.com/fwlink/?linkid=851275)
2019-03-11T18:40:48.6827206Z ==============================================================================
2019-03-11T18:40:48.9820084Z ##[debug]agent.workFolder=D:\_Builds\E20-IX-VSBD008P.01
2019-03-11T18:40:48.9906390Z ##[debug]loading inputs and endpoints
2019-03-11T18:40:48.9916447Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN
2019-03-11T18:40:48.9931497Z ##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION
2019-03-11T18:40:48.9937174Z ##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION
2019-03-11T18:40:48.9940841Z ##[debug]loading INPUT_CHECKLATEST
2019-03-11T18:40:48.9946190Z ##[debug]loading INPUT_COMMAND
2019-03-11T18:40:48.9949055Z ##[debug]loading INPUT_CONFIGMAPFILE
2019-03-11T18:40:48.9952324Z ##[debug]loading INPUT_CONFIGURATION
2019-03-11T18:40:48.9954799Z ##[debug]loading INPUT_CONFIGURATIONTYPE
2019-03-11T18:40:48.9957246Z ##[debug]loading INPUT_CONNECTIONTYPE
2019-03-11T18:40:48.9960233Z ##[debug]loading INPUT_CONTAINERREGISTRYTYPE
2019-03-11T18:40:48.9962803Z ##[debug]loading INPUT_CWD
2019-03-11T18:40:48.9965301Z ##[debug]loading INPUT_FORCEUPDATE
2019-03-11T18:40:48.9967464Z ##[debug]loading INPUT_FORCEUPDATECONFIGMAP
2019-03-11T18:40:48.9971851Z ##[debug]loading INPUT_OUTPUTFORMAT
2019-03-11T18:40:48.9977526Z ##[debug]loading INPUT_SECRETTYPE
2019-03-11T18:40:48.9982395Z ##[debug]loading INPUT_SPECIFYLOCATION
2019-03-11T18:40:48.9988019Z ##[debug]loading INPUT_USECONFIGMAPFILE
2019-03-11T18:40:48.9991102Z ##[debug]loading INPUT_USECONFIGURATIONFILE
2019-03-11T18:40:48.9994257Z ##[debug]loading INPUT_VERSIONORLOCATION
2019-03-11T18:40:48.9998661Z ##[debug]loading INPUT_VERSIONSPEC
2019-03-11T18:40:49.0012399Z ##[debug]loading SECRET_AKSINGRESSPRIVATEKEY
2019-03-11T18:40:49.0018139Z ##[debug]loading SECRET_HTTPSCERTIFICATEPASSWORD
2019-03-11T18:40:49.0024558Z ##[debug]loading SECRET_SCHEDULEMANAGERSQLUSERPASSWORD
2019-03-11T18:40:49.0025524Z ##[debug]loading SECRET_SETTINGSMANAGERSQLUSERPASSWORD
2019-03-11T18:40:49.0026722Z ##[debug]loading SECRET_SQLADMINPASSWORD
2019-03-11T18:40:49.0029904Z ##[debug]loading SECRET_SQLSCALETESTPASSWORD
2019-03-11T18:40:49.0032935Z ##[debug]loading SECRET_SQLSERVERCONNECTIONSTRINGSCALETEST
2019-03-11T18:40:49.0037353Z ##[debug]loading SECRET_SQLSERVERCONNECTIONSTRINGSCHEDULEMANAGER
2019-03-11T18:40:49.0039660Z ##[debug]loading SECRET_SQLSERVERCONNECTIONSTRINGSETTINGSMANAGER
2019-03-11T18:40:49.0042738Z ##[debug]loading SECRET_SQLSERVERCONNECTIONSTRINGSMF
2019-03-11T18:40:49.0058045Z ##[debug]loading SECRET_SQLSMFUSERPASSWORD
2019-03-11T18:40:49.0068147Z ##[debug]loaded 31
2019-03-11T18:40:49.1114840Z ##[debug]agent.proxyurl=undefined
2019-03-11T18:40:49.1115934Z ##[debug]VSTS_ARM_REST_IGNORE_SSL_ERRORS=undefined
2019-03-11T18:40:49.1116283Z ##[debug]AZURE_HTTP_USER_AGENT=VSTS_5f22a025-b310-4483-bb27-d8b98fb03a0e_release_12_2355_11463_1
2019-03-11T18:40:49.1200114Z ##[debug]check path : D:\_Builds\E20-IX-VSBD008P.01\_tasks\Kubernetes_cbc316a2-586f-4def-be79-488a1f503564\1.1.24\task.json
2019-03-11T18:40:49.1200993Z ##[debug]set resource file to: D:\_Builds\E20-IX-VSBD008P.01\_tasks\Kubernetes_cbc316a2-586f-4def-be79-488a1f503564\1.1.24\task.json
2019-03-11T18:40:49.1202753Z ##[debug]system.culture=en-US
2019-03-11T18:40:49.1237163Z ##[debug]cwd=D:\_Builds\E20-IX-VSBD008P.01\r2\a
2019-03-11T18:40:49.1248539Z ##[debug]containerRegistryType=Azure Container Registry
2019-03-11T18:40:49.1250747Z ##[debug]command=logout
2019-03-11T18:40:49.1251190Z ##[debug]KUBECONFIG=D:\_Builds\E20-IX-VSBD008P.01\_temp\kubectlTask\1552315183701\config
2019-03-11T18:40:49.1255374Z ##[debug]which 'kubectl'
2019-03-11T18:40:49.1275371Z ##[debug]found: 'D:\_Builds\E20-IX-VSBD008P.01\_tool\kubectl\1.7.0\x64\kubectl.exe'
2019-03-11T18:40:49.1276103Z ##[debug]agent.tempDirectory=D:\_Builds\E20-IX-VSBD008P.01\_temp
2019-03-11T18:40:49.1313108Z ##[debug]connectionType=Azure Resource Manager
2019-03-11T18:40:49.1332292Z ##[debug]versionOrLocation=version
2019-03-11T18:40:49.1338557Z ##[debug]versionSpec=1.7.0
2019-03-11T18:40:49.1342422Z ##[debug]checkLatest=false
2019-03-11T18:40:49.1361118Z ##[debug][GET]https://api.github.com/repos/kubernetes/kubernetes/releases?page=0&per_page=100
2019-03-11T18:40:49.1610106Z (node:12116) Warning: Use Cipheriv for counter mode of aes-256-ctr
2019-03-11T18:40:49.1610795Z (node:12116) Warning: Use Cipheriv for counter mode of aes-256-ctr
2019-03-11T18:40:49.1611036Z (node:12116) Warning: Use Cipheriv for counter mode of aes-256-ctr
2019-03-11T18:40:49.1611281Z (node:12116) Warning: Use Cipheriv for counter mode of aes-256-ctr
2019-03-11T18:40:49.1611544Z (node:12116) Warning: Use Cipheriv for counter mode of aes-256-ctr
2019-03-11T18:40:49.1611766Z (node:12116) Warning: Use Cipheriv for counter mode of aes-256-ctr
2019-03-11T18:40:49.1612018Z (node:12116) Warning: Use Cipheriv for counter mode of aes-256-ctr
2019-03-11T18:40:49.1612260Z (node:12116) Warning: Use Cipheriv for counter mode of aes-256-ctr
2019-03-11T18:40:49.1612663Z (node:12116) Warning: Use Cipheriv for counter mode of aes-256-ctr
2019-03-11T18:40:49.1612896Z (node:12116) Warning: Use Cipheriv for counter mode of aes-256-ctr
2019-03-11T18:40:49.1613142Z (node:12116) Warning: Use Cipheriv for counter mode of aes-256-ctr
2019-03-11T18:40:49.1613352Z (node:12116) Warning: Use Cipheriv for counter mode of aes-256-ctr
2019-03-11T18:40:49.1613584Z (node:12116) Warning: Use Cipheriv for counter mode of aes-256-ctr
2019-03-11T18:40:49.1613813Z (node:12116) Warning: Use Cipheriv for counter mode of aes-256-ctr
2019-03-11T18:40:49.1614042Z (node:12116) Warning: Use Cipheriv for counter mode of aes-256-ctr
2019-03-11T18:40:49.1614560Z (node:12116) Warning: Use Cipheriv for counter mode of aes-256-ctr
2019-03-11T18:40:49.1614796Z (node:12116) Warning: Use Cipheriv for counter mode of aes-256-ctr
2019-03-11T18:40:49.1615008Z (node:12116) Warning: Use Cipheriv for counter mode of aes-256-ctr
2019-03-11T18:40:49.1615243Z (node:12116) Warning: Use Cipheriv for counter mode of aes-256-ctr
2019-03-11T18:40:49.1615474Z (node:12116) Warning: Use Cipheriv for counter mode of aes-256-ctr
2019-03-11T18:40:49.1615707Z (node:12116) Warning: Use Cipheriv for counter mode of aes-256-ctr
2019-03-11T18:40:49.1615953Z (node:12116) Warning: Use Cipheriv for counter mode of aes-256-ctr
2019-03-11T18:40:49.1616370Z (node:12116) Warning: Use Cipheriv for counter mode of aes-256-ctr
2019-03-11T18:40:49.1616604Z (node:12116) Warning: Use Cipheriv for counter mode of aes-256-ctr
2019-03-11T18:40:49.1616978Z (node:12116) Warning: Use Cipheriv for counter mode of aes-256-ctr
2019-03-11T18:40:49.1617228Z (node:12116) Warning: Use Cipheriv for counter mode of aes-256-ctr
2019-03-11T18:40:49.1617448Z (node:12116) Warning: Use Cipheriv for counter mode of aes-256-ctr
2019-03-11T18:40:49.1632032Z (node:12116) Warning: Use Cipheriv for counter mode of aes-256-ctr
2019-03-11T18:40:49.1632319Z (node:12116) Warning: Use Cipheriv for counter mode of aes-256-ctr
2019-03-11T18:40:49.1632727Z (node:12116) Warning: Use Cipheriv for counter mode of aes-256-ctr
2019-03-11T18:40:49.1632962Z (node:12116) Warning: Use Cipheriv for counter mode of aes-256-ctr
2019-03-11T18:40:49.6390188Z ##[debug]task result: Failed
2019-03-11T18:40:49.6391169Z ##[error]response.body.forEach is not a function
2019-03-11T18:40:49.6391683Z ##[debug]Processed: ##vso[task.issue type=error;]response.body.forEach is not a function
2019-03-11T18:40:49.6392353Z ##[debug]Processed: ##vso[task.complete result=Failed;]response.body.forEach is not a function
2019-03-11T18:40:49.6392763Z ##[debug]connectionType=Azure Resource Manager
2019-03-11T18:40:49.6481395Z ##[section]Finishing: Log out of AKS

Error logs

##[error]response.body.forEach is not a function

[damienwebdev]

Can confirm that this has nothing to do with Github, this is 100% a task problem.

2019-03-12T17:17:14.1401448Z ##[section]Starting: Delete web deployment
2019-03-12T17:17:14.1407804Z ==============================================================================
2019-03-12T17:17:14.1408126Z Task         : Deploy to Kubernetes
2019-03-12T17:17:14.1408188Z Description  : Deploy, configure, update your Kubernetes cluster in Azure Container Service by running kubectl commands.
2019-03-12T17:17:14.1408288Z Version      : 1.1.24
2019-03-12T17:17:14.1408333Z Author       : Microsoft Corporation
2019-03-12T17:17:14.1408400Z Help         : [More Information](https://go.microsoft.com/fwlink/?linkid=851275)
2019-03-12T17:17:14.1408502Z ==============================================================================
2019-03-12T17:17:15.2507651Z ##[error]response.body.forEach is not a function
2019-03-12T17:17:15.2517436Z ##[section]Finishing: Delete web deployment

The command being run is:

kubectl delete -f /path/to/some/file.yaml

Here's the same issue in a different thread:

https://developercommunity.visualstudio.com/content/problem/479271/kubectl-apply-failing-with-responsebodyforeach-is.html

[oskarm93]

@damienwebdev If you do not have system.debug set to true, the task will not show you that it's querying github API. Every time we get this failure and have debug logs enabled, this github call appears before it.

[damienwebdev]

@xenalite, ah I see. Really good catch there. I'm assuming that this must be Azure Pipelines reaching the Github API rate limit for that particular query... which I now realise you already discovered.

[damienwebdev]

@xenalite I think I have a short-fix while MSFT attempts to solve this long term... there's two issues at hand:

Issues

1. The offending code doesn't handle rate-limiting
2. The offending code isn't actually backwards compatible as intended

In my case, my settings look like:

Notice that in the getKubectlVersion function, the version is 1.7 and additionally that mine (as set originally by MSFT when I created the task) is "1.7.0", not "1.7". While I didn't run the code to see the issue, I'd imagine this is part of the problem as MSFT is using === to do a typecheck as string.

Solutions

Solution for problem 1

Rework the underlying functionality to handle the rate-limiting. An internal MSFT dev should most likely do this, or a willing contributor familiar with the project.

Solution for problem 2

Remove the Check for latest version flag and additionally ensure that your Version Spec field is empty. This will prevent the API calls, and resolve the blocking rate-limiting issue.

[oskarm93]

@damienwebdev My task doesn't even have a version drop down or the advanced section, because I only every do login and logout commands.

[madhuv-msft]

we have fixed this issue and deployed. this should no more be problamatic.
#9817 PR has fixed this issue