Improve flaky integration tests

[brendankowitz]

Description

This pull request primarily focuses on improving the test suite for integration tests. The changes include modifications to the Azure DevOps pipeline YAML file to separate integration tests into different jobs, changes to the test suite to improve database setup and teardown, and enhancements to the test suite to better handle cancellation and disposal of resources.

Changes to the CI/CD pipeline:

• build/jobs/run-tests.yml: The integration test job was split into two separate jobs, CosmosIntegrationTests and SqlIntegrationTests, to allow for more granular control over the execution of different types of integration tests. This change also involves the addition of new tasks to each job to execute the respective tests. [1] [2]

Improvements to test suite:

• test/Microsoft.Health.Fhir.Shared.Tests.Integration/Persistence/SqlServerFhirStorageTestHelper.cs: The database setup and teardown process was improved by adding a semaphore to limit the number of concurrent operations and increasing the number of retries for certain operations. This should make the test suite more robust in the face of intermittent database connectivity issues. [1] [2] [3] [4] [5] [6]
• Various test files: Added the FhirStorageTestsFixtureArgumentSets attribute to several test classes to specify that they should be run with the SqlServer data store. This change makes the test suite more flexible and easier to configure. [1] [2] [3] [4] [5] [6] [7]

Enhancements to cancellation and disposal:

• src/Microsoft.Health.Fhir.SqlServer/Features/Watchdogs/FhirTimer.cs: Improved the handling of cancellation in the RunInternalAsync method to immediately return if cancellation is requested.
• test/Microsoft.Health.Fhir.Shared.Tests.Integration/Features/ChangeFeed/SqlServerFhirResourceChangeCaptureDisabledTests.cs and test/Microsoft.Health.Fhir.Shared.Tests.Integration/Features/ChangeFeed/SqlServerFhirResourceChangeCaptureFixture.cs: Modified the disposal of resources in test classes to check if the resource is null before attempting to dispose of it. This change makes the test suite more robust in the face of unexpected errors during setup. [1] [2]

Related issues

Addresses AB#120046

Testing

Describe how this change was tested.

FHIR Team Checklist

• Update the title of the PR to be succinct and less than 65 characters
• Add a milestone to the PR for the sprint that it is merged (i.e. add S47)
• Tag the PR with the type of update: Bug, Build, Dependencies, Enhancement, New-Feature or Documentation
• Tag the PR with Open source, Azure API for FHIR (CosmosDB or common code) or Azure Healthcare APIs (SQL or common code) to specify where this change is intended to be released.
• Tag the PR with Schema Version backward compatible or Schema Version backward incompatible or Schema Version unchanged if this adds or updates Sql script which is/is not backward compatible with the code.
• CI is green before merge
• Review squash-merge requirements

Semver Change (docs)

Patch|Skip|Feature|Breaking (reason)

[LTA-Thinking]

Why do we not want to retry the cleanup?

[brendankowitz]

I was mainly trying to find where all the time is spent for integration tests. I think we wait the max amount of time on retries to delete the database at the end of every fixture.
Strangely enough, when watching the resource group the databases all seemed to eventually drop, perhaps there's something async going on under the covers on the sql side.

On top of this, we delete the resource group at the end of the test which should cover anything that failed, and CI has a task to remove orphaned test databases.