JenkinsQueueJob: Job progress tracking failed to read job progress

[Nekto89]

Required Information

Entering this information will route you directly to the right team and expedite traction.

Question, Bug, or Feature?
Type: Bug

Enter Task Name: JenkinsQueueJob

Environment

• Server - Azure Pipelines or TFS on-premises?

  • TFS on-premise Version Dev17.M153.5

• Agent - Hosted or Private:

  • If using private agent, provide the OS of the machine running the agent and the agent version:
    Windows 10.0.18363, Agent.Version 2.153.1

Issue Description

Sometimes task loses connection to Jenkins server temporarily. But it continues to wait until job finishes on Jenkins side.

Task fails, build fails. At the same time info is available that Jenkins job has finished successfully on Summary tab.

Task logs

In the log there may be errors like this in the middle:
"
##[error]Job progress tracking failed to read job progress
Job progress tracking failed to read job progress
HttpResponse.statusCode=500
HttpResponse.statusMessage=Server Error
##[error]Job progress tracking failed to read job progress
"

[ericstrongDevOps]

any update on this issue? TIA

[walliski]

I am also still seeing this issue now and then, very inconvenient.

[anatolybolshakov]

Hi @Nekto89 @ericstrongDevOps @walliski could you please share the logs with this error - for us to understand better how we could fix this? We could add retry logic - but we need to determine where exactly this issue happens.

[walliski]

@anatolybolshakov

I managed to reproduce the problem with debug on, by queuing a job in Jenkins that simply had a sleep step, and following the (non-existent) console output.

This is what I got in the logs in AzDo:

...

##[debug](2:Test Stuff:167) debug: Tracking progress of job URL: https://jenkins.int/job/Test%2520Stuff/167/logText/progressiveText/?start=3827
##[debug]streamConsole().requestCallback()

##[debug](2:Test Stuff:167) debug: Tracking progress of job URL: https://jenkins.int/job/Test%2520Stuff/167/logText/progressiveText/?start=3827
Job progress tracking failed to read job progress

##[debug]streamConsole().requestCallback()
HttpResponse.statusCode=500

##[debug]task result: Failed
HttpResponse.statusMessage=Server Error

##[error]Job progress tracking failed to read job progress
##[debug]Processed: ##vso[task.issue type=error;]Job progress tracking failed to read job progress
##[debug]Processed: ##vso[task.complete result=Failed;]Job progress tracking failed to read job progress

##[debug](2:Test Stuff:167) debug: Tracking progress of job URL: https://jenkins.int/job/Test%2520Stuff/167/logText/progressiveText/?start=3827
##[debug]streamConsole().requestCallback()

...

Before and after this was a long list of debug messages for the tracking. So it seems like even if it sets the build as failed, it still continues to poll and run the task in Jenkins (which finished successfully). Build ran for 60 minutes, and there was only one failed poll attempt during that time.

The reason why Jenkins threw a random 500 was a "Socket timeout" for the authentication plugin that we are using.

[anatolybolshakov]

@Nekto89 @walliski thanks! Let us take a look - probably it make sense to add some retry logic for these requests - to avoid intermittent issues.

[cmwiebe24]

Any update on this issue? Or is there a work around?

[walliski]

@anatolybolshakov Sorry to be annoying, but any progress on this? This is an issue that we keep seeing almost daily.

[kuleshovilya]

Hello @walliski We've already implemented this feature, and awaiting for deployment now, it will take few weeks, we will notify you when it's done

[kuleshovilya]

Feature has been merged, you can check this out now, closing the issue

[walliski]

@kuleshovilya

I just got this again, same error message:

Job progress tracking failed to read job progress
HttpResponse.statusCode=500
HttpResponse.statusMessage=Server Error
##[error]Job progress tracking failed to read job progress

But this message in the end:

Finished: SUCCESS

On this version:

==============================================================================
Task         : Jenkins queue job
Description  : Queue a job on a Jenkins server
Version      : 2.192.0
Author       : Microsoft
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/build/jenkins-queue-job
==============================================================================

[kuleshovilya]

@walliski Could you please send a full logs for this job for the analysis? You can send it in private to v-ikuleshov@microsoft.com

[walliski]

The run did not run in debug mode, so what I posted above is the only log output I can see that would be outside of our own scripts output, unless there is something that you are clearly missing? I could try to reproduce the issue again in debug mode though and see if that would bring some more clarity. IIRC I saw some 500 error before in Jenkins logs, but I cannot seem to find that now, so not really sure either what's going on...

[kuleshovilya]

@walliski Yes, can you please try to reproduce and send logs?

[kuleshovilya]

@walliski Any update?

[walliski]

I sent an email to the email you mentioned two weeks ago. There was however no interesting things in the logs, the place where it fails looks the same as before. Or what did you mean?

[anatolybolshakov]

Hi @walliski sorry for the delay - we've added new inputs for JenkinsQueueJob task v2.192.1 - 'retryCount' and 'delayBetweenRetries' (you can find more info in docs) in PR - could you please check it now? As I see there is a 2.192.0 task version in the last logs - which does not have a fix.

[walliski]

Will take a look. I saw that the task was updated compared to previous jobs, so I assumed it would have had the changes in it, but apparently not. Now I see that we are currently on 2.193.0, so I guess that should be good to go. I'll get back to you if I see more issues, thanks!