Test result attached to a different workflow that is being run concurrently

[lgolez]

Describe the bug

• I have two separate workflows with different goals. One to test Android, the other IOS
• Both of these workflows are being run together when a PR is created.
• Both of these workflows utilise a composite step where it prints out the results e.g.

name: "Print test results"
description: "Prints test results into github actions"

runs:
  using: "composite"
  steps:
      - name: Test Report
        uses: github-actions/dorny-test-reporter@v1.0.0
        if: success() || failure()   
        with:
          name: Test results            
          path: ./results/*.xml    # Path to test results
          reporter: jest-junit

To Reproduce

Steps to reproduce the behavior:

1. Create two separate workflows that run tests and produce an XML test result
2. Both workflows should be triggered at the same time e.g. both should have a similar trigger like on push
3. At the end of the two workflows, add a step that reads those result
4. Check result

Expected behavior

Job should be attached to their respective workflows

Screenshots

Two different workflows

Last one does not contain the test result

First workflow to be triggered contained both the results

Additional context

I attempted to separate test results without using the custom action and explicitly added the test report script into separate workflows but still got the same issue.

[i2van]

• No tests under Jobs: https://github.com/i2van/hourglass/actions/runs/10468117626

• Two tests under Jobs: https://github.com/i2van/hourglass/actions/runs/10468062689

[peterbrendel]

Edit

Actually the workflow that it got attached to wasn't even running, I noticed this now because it is skipped for that PR, so it attached to a finished (skipped) workflow.

---

I experienced the same issue with different steps to reproduce, the other workflow doesn't use this action:

To Reproduce

Steps to reproduce the behavior:

1. Have one or more workflows running
2. Generate test report

[alevincenzi]

Hello
I saw the same behavior too. We fire two workflows at the PR and sometimes it's appended to the other one.
But isn't this problem that we all see related to the github limitation mentioned in the readme?

https://github.com/dorny/test-reporter?tab=readme-ov-file#github-limitations

I found other github actions for TRX reports and they all mention the same. Only GitHub today "is allowed to create a check suite for each workwflow run and there is no public Check Suite API" [cit].

Take it or leave it ...

[alevincenzi]

Check this line https://github.com/dorny/test-reporter/blob/main/src/main.ts#L178

where the action uses the octokit client to create the check using the sha of the commit.

Here the official API called to create the run: https://docs.github.com/en/rest/checks/runs?apiVersion=2022-11-28#create-a-check-run

In the body there is no mention "of the workflow" and only name and head_sha are required.
If it goes to another workflow, it's probably GitHub's fault.

[alevincenzi]

Also, this has never been solved on GitHub side: https://github.com/orgs/community/discussions/24616

[peterbrendel]

I can't say for sure, but I might have a workaround, give it a try and see if it works:

I added another step after the report step and now GHA is attaching to the correct workflow, it worked 3 times in a row, could be coincidence tho.

Example

- name: Test Report
  uses: dorny/test-reporter@v1
  with:
    name: Unit Tests
    path: 'test-results/*.xml'
    reporter: java-junit
    list-tests: none
    max-annotations: '50'
- name: Random step
  run: echo "Test"

[rlordcardano]

@peterbrendel - that didn't work for me. In our setup the master build runs on a cron job, and the test reports get appended to the original build that was merged back into master.

[ccpp]

[EDIT] Sorry, I just found tat you already have several duplicate issues to this one, like #67 and #224, so you know this problem well...
Anyways, here's my observation which might still be an issue...

---

I have the same issue and I suspect that the problem is somewhere here. The github API seems not used correctly as far as I can judge:

export function getCheckRunContext(): {sha: string; runId: number} {
  if (github.context.eventName === 'workflow_run') {
    // should be "event_name" ?
    core.info('Action was triggered by workflow_run: using SHA and RUN_ID from triggering workflow')
    const event = github.context.payload
    if (!event.workflow_run) {
      throw new Error("Event of type 'workflow_run' is missing 'workflow_run' field")
    }
    return {
      sha: event.workflow_run.head_commit.id,
      runId: event.workflow_run.id
    }
  }

  const runId = github.context.runId
  // should be "run_id" ?
  if (github.context.payload.pull_request) {
    core.info(`Action was triggered by ${github.context.eventName}: using SHA from head of source branch`)
    const pr = github.context.payload.pull_request as PullRequest
    return {sha: pr.head.sha, runId}
  }

  return {sha: github.context.sha, runId}
}

[lneuhaus]

@ccpp are you able to confirm that this the workflow is the correct one with your patch?

[ccpp]

No, sorry, this is untested! @lneuhaus

I even cannot reproduce the problem at the moment, probably since I switched to the artifact configuration. now the report is shown in the summary of the correct workflow

[bt-abarber]

No, sorry, this is untested! @lneuhaus

I even cannot reproduce the problem at the moment, probably since I switched to the artifact configuration. now the report is shown in the summary of the correct workflow

What exactly do you mean by switched to the artifact configuration ? I've been fighting this issue on and off for a few days thinking it was something I had done wrong only to get looking for solutions and opening the can of worms that is multiples of bugs opened for this shortcoming of GitHub's.

[ccpp]

Hi @bt-abarber ,

What exactly do you mean by switched to the artifact configuration ?

I don't have the codebast at hand, but what I did was to save my .xml reports in artifacts, and to create a fresh job which picks up those reports using the artifact: statement.
Would be interesting if it solves the issue alsoo on your side!

[liyaka]

Any update on a fix or workaround?

[ccpp]

Any update on a fix or workaround?

@liyaka can you confirm that my approach works for you too ( #512 (comment) )? Then that would be the workaround you're looking for

[bt-abarber]

@ccpp I'm not quite sure what kind of job you have set up but I tired the following workflow (ok not literally this, but essentially this workflow) without any luck:

name: fake_job_as_example
  workflow_call:  
    inputs:
      target:
        required: false
        default: ''
        type: string


jobs:
  checkout:  
    runs-on: [self-hosted, Windows]
    continue-on-error: false
    name: Checkout
    timeout-minutes: 0
    environment: my_env
    steps:
    - name: checkout
      uses: actions/checkout@v4.1.0

  
  build_project:  
    runs-on: [self-hosted, Windows]
    continue-on-error: false
    environment: my_env
    name: Build Project
    needs: [checkout]
    steps:
    - shell: powershell
      run: get-childItem ./ -recurse | Where {$_.name -like "*.csproj"} | dotnet build $_

    
  job_1:  
    runs-on: [self-hosted, Windows]
    environment: my_env
    name: Job 1
    needs: [build_project]
    if: ${{ !cancelled() }} 
    steps:
    - name: Run Tests
      shell: powershell
      run: dotnet test ./job_1.csproj --logger "trx;logfilename=job_1.trx" -v n --no-build --no-restore MSTest.Parallelize.Scope="ClassLevel"
    - name: Archive Job 1 results
      if: success() || failure()
      uses: actions/upload-artifact@v4
      with:
        name: job_1_results
        path: ./TestResults/job_1.trx

  job_1_results:  
    runs-on: [self-hosted, Windows]
    name: Job 1 Results
    needs: [job_1]
    if: ${{ !cancelled() }} 
    steps:
    - name: Get the artifacts from previous step
      if: success() || failure()
      uses: actions/download-artifact@v4
      with:
        path: "TestResults"
        name: job_1_results
    - uses: dorny/test-reporter@v1.9.1      
      if: success() || failure()
      with:
        name: Job 1 Test Summary
        path: "./TestResults/job_1.trx"                           
        reporter: dotnet-trx
        fail-on-error: false
        list-tests: 'failed'   
        list-suites: 'all'


 job_2:  
    runs-on: [self-hosted, Windows]
    environment: my_env
    name: Job 2
    needs: [build_project]
    if: ${{ !cancelled() }} 
    steps:
    - name: Run Tests
      shell: powershell
      run: dotnet test ./job_2.csproj --logger "trx;logfilename=job_2.trx" -v n --no-build --no-restore MSTest.Parallelize.Scope="ClassLevel"
    - name: Archive Job 2 results
      if: success() || failure()
      uses: actions/upload-artifact@v4
      with:
        name: job_2_results
        path: ./TestResults/job_2.trx

  job_2_results:  
    runs-on: [self-hosted, Windows]
    name: Job 2 Results
    needs: [job_2]
    if: ${{ !cancelled() }} 
    steps:
    - name: Get the artifacts from previous step
      if: success() || failure()
      uses: actions/download-artifact@v4
      with:
        path: "TestResults"
        name: job_2_results
    - uses: dorny/test-reporter@v1.9.1      
      if: success() || failure()
      with:
        name: Job 2 Test Summary
        path: "./TestResults/job_2.trx"                           
        reporter: dotnet-trx
        fail-on-error: false
        list-tests: 'failed'   
        list-suites: 'all'



test_summary:
    needs: [job_6]
    runs-on: [self-hosted, Windows]
    name: Test Summary
    environment: my_env
    if: ${{ !cancelled() }}
    steps:
        - name: Generate test summary
        uses: dorny/test-reporter@v1.9.1
        with:
            name: Test Summary
            path: "TestResults/*.trx"    
            only-summary: 'true'                        
            reporter: dotnet-trx
            fail-on-error: true
            list-tests: 'failed'   
            list-suites: 'all'

I'm sure I've missed something scrubbing the above example but essentially my workflow works, it generates the test summary reports as expected, but they're not visible within the workflow, they all get attached to the first instance of the workflow being run after I commit changes....

[ccpp]

@bt-abarber almost, but don't use the "download artifact" action. Instead, use the "artifact:" statement of dorny/test-reporter, as documented https://github.com/dorny/test-reporter?tab=readme-ov-file#usage

[bt-abarber]

Unfortunately I'm still not having any luck:

name: fake_job_as_example
  workflow_call:  
    inputs:
      target:
        required: false
        default: ''
        type: string


jobs:
  checkout:  
    runs-on: [self-hosted, Windows]
    continue-on-error: false
    name: Checkout
    timeout-minutes: 0
    environment: my_env
    steps:
    - name: checkout
      uses: actions/checkout@v4.1.0

  
  build_project:  
    runs-on: [self-hosted, Windows]
    continue-on-error: false
    environment: my_env
    name: Build Project
    needs: [checkout]
    steps:
    - shell: powershell
      run: get-childItem ./ -recurse | Where {$_.name -like "*.csproj"} | dotnet build $_

    
  job_name:  
    runs-on: [self-hosted, Windows]
    environment: my_env
    name: Job 1
    needs: [build_project]
    if: ${{ !cancelled() }} 
    steps:
    - name: Run Tests
      shell: powershell
      run: dotnet test ./job_1.csproj --logger "trx;logfilename=job_1.trx" -v n --no-build --no-restore MSTest.Parallelize.Scope="ClassLevel"
     - name: Run Tests 2
      shell: powershell
      run: dotnet test ./job_2.csproj --logger "trx;logfilename=job_2.trx" -v n --no-build --no-restore MSTest.Parallelize.Scope="ClassLevel"
    - name: Archive results
      if: success() || failure()
      uses: actions/upload-artifact@v4
      with:
        name: test_results
        path: ./TestResults/*.trx

 

test_summary:
    needs: [job_name]
    runs-on: [self-hosted, Windows]
    name: Test Summary
    environment: my_env
    if: ${{ !cancelled() }}
    steps:
        - name: Generate test summary
        uses: dorny/test-reporter@v1.9.1
        with:
            name: Test Summary
            artifact: test_results
            path: "TestResults/*.trx"    
            only-summary: 'true'                        
            reporter: dotnet-trx
            fail-on-error: true
            list-tests: 'failed'   
            list-suites: 'all'

Files are made in to an artifact as expected, the test_summary job opens them, sees them and generates its reports but the Summary results are still attached to the first instance of the commit for the workflow.

[ccpp]

@bt-abarber OK I guess I cannot help then. Last time I used this in a project it seemed to resolve the issue for me.
Sharing shortened for readability, essentially looks just like your setup. You could try changing to the specific version 1a288b6 that was needed for our team to get NUnit working:

name: Build
on: [push, workflow_dispatch]

jobs:
  merge_conflict_check:
    (...)

  build_uwp_pass1:
    (...)
    steps:
      (...)
      - name: Test Project
        run: .github/workflows/run-tests.ps1

      - name: Upload Test Results
        uses: actions/upload-artifact@v4
        with:
          name: Test-Results
          path: TestResults
        if: always()

    (...)

  check_test_result:
    name: Test Report
    needs: build_uwp_pass1
    runs-on: ubuntu-latest
    steps:
      - name: Test Report
        uses: dorny/test-reporter@1a288b62f8b75c0f433cbfdbc2e4800fbae50bd7
        with:
          name: Unity Test Results
          artifact: Test-Results
          path: ./*.xml
          reporter: dotnet-nunit
          fail-on-error: true
(...)