Fix sparse checkout and set list of service directories based upon the project list #29151
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,116 @@ | ||
| # Copyright (c) Microsoft Corporation. All rights reserved. | ||
| # Licensed under the MIT License. | ||
|
|
||
| # Use case: | ||
| # Given a project list, set following environment variables in JSON format: | ||
| # 1. SparseCheckoutDirectories - This the list of sparse checkout paths that will be used by sparse-checkout.yml | ||
| # 2. ServiceDirectories - A list of ServiceDirectories. | ||
| # | ||
| # The project list should be generated by a call to generate-project-list-and-cache-maven-repository.yml | ||
| # prior to calling this script. | ||
| param( | ||
| # $(Build.SourcesDirectory) - root of the repository | ||
| [Parameter(Mandatory=$true)][string]$SourcesDirectory, | ||
| # ArtifactsList will be using ('${{ convertToJson(parameters.Artifacts) }}' | ConvertFrom-Json | Select-Object name, groupId) | ||
| [Parameter(Mandatory=$true)][array] $ProjectList | ||
| ) | ||
|
|
||
| $StartTime = $(get-date) | ||
| . "${PSScriptRoot}/../common/scripts/common.ps1" | ||
| $Path = Resolve-Path ($PSScriptRoot + "/../../") | ||
| $script:FoundError = $false | ||
|
|
||
| function Build-Unreleased-List-From-File { | ||
| param([string]$versionFile) | ||
| $unreleasedList = @() | ||
| foreach($line in Get-Content $versionFile) { | ||
| if (!$line -or $line.Trim() -eq '' -or $line.StartsWith("#") -or !$line.StartsWith("unreleased_")) { | ||
| continue | ||
| } else { | ||
| $split = $line.Split(";") | ||
| if ($split.Length -ne 2) { | ||
| LogError("Malformed unreleased_ dependency line in $versionFile. line=$line") | ||
| # If there's a malformed unreleased_ dependency line in version_client.txt then | ||
| # immediately exit. This is the only place we need to immediately exit. | ||
| exit(1) | ||
JimSuplizio marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| } | ||
| $temp = $split[0].Replace("unreleased_","") | ||
| $unreleasedList += $temp | ||
| } | ||
| } | ||
| return ,$unreleasedList | ||
| } | ||
|
|
||
| $unreleasedList = @() | ||
| # version_client is the only version file that will have unreleased_ dependencies as these | ||
| # are disallowed for data track libraries. | ||
| $unreleasedList = Build-Unreleased-List-From-File $Path\eng\versioning\version_client.txt | ||
|
|
||
| $sparseCheckoutDirHash = @{} | ||
| $serviceDirHash = @{} | ||
| foreach($file in Get-ChildItem -Path $SourcesDirectory -Filter pom*.xml -Recurse -Depth 3 -File) { | ||
|
There was a problem hiding this comment. You should consider using Get-java-PackageInfoFromRepo from language-settings.ps1 to try and start using a common function for gathering properties for the projects. There was a problem hiding this comment. @weshaggard, we'd have a chicken and egg there. That function requires the service directory. The common functions, for Java anyways, are far less common than one would think. Things under common aren't really common outside of the fact that they're specifically being used for a targeted piece of processing (like docs updating or .MD file verifications) and are mostly not useful outside of the context in which they were initially written. There was a problem hiding this comment. I believe that function should work without a servicedirectory as well. Mostly I was wanting to try and have a common function we can use to parse and produce the properties. |
||
| $xmlPomFile = $null | ||
| $xmlPomFile = New-Object xml | ||
| $xmlPomFile.Load($file.FullName) | ||
| $library = $xmlPomFile.project.groupId + ":" + $xmlPomFile.project.artifactId | ||
| $serviceDirectory = (Get-Item $file).Directory.Parent | ||
| # This if check is only necessary because resourcemanager and resourcemanagerhybrid contain the | ||
| # exact same group/artifact ids | ||
| if ($file.FullName.Split([IO.Path]::DirectorySeparatorChar) -notcontains "resourcemanagerhybrid") { | ||
benbp marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| # The directories for sparse checkout and ServiceDirectories need the $SourcesDirectory | ||
| # stripped off | ||
| $tempDir = $serviceDirectory.FullName.Replace("$SourcesDirectory", "") | ||
| $tempDir = $tempDir.Replace([IO.Path]::DirectorySeparatorChar, '/') | ||
| $sparseCheckoutDirHash.Add($library, $tempDir) | ||
| if ($tempDir.StartsWith("/sdk/")) { | ||
| # Strip off the "/sdk/" to get the service directory. A ServiceDirectory will always have | ||
| # the format of /SDK/<ServiceDirectory>. Whereas sparse checkout can have other directories | ||
| # outside of /SDK, eg. /common | ||
| $tempDir = $tempDir.Replace("/sdk/", "") | ||
JimSuplizio marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| $serviceDirHash.Add($library, $tempDir) | ||
| } | ||
| } | ||
| } | ||
|
|
||
| $sparseCheckoutDirectories = @() | ||
| $serviceDirectories = @() | ||
| $sparseCheckoutDirectories += "/sdk/parents" | ||
| foreach ($project in $ProjectList) { | ||
| if ($sparseCheckoutDirHash.ContainsKey($project)) { | ||
| $sparseCheckoutDirectories += $sparseCheckoutDirHash[$project] | ||
| $serviceDirectories += $serviceDirHash[$project] | ||
| } else { | ||
| LogError("Did not find $project in any pom files") | ||
| $script:FoundError = $true | ||
| } | ||
| } | ||
| # Unreleased_ libraries are special. They're the only case, outside of FromSource runs where | ||
| # libraries from other service directories, outside of the one we're building, need to get built. | ||
| # It's easier to just add any unreleased_ dependency service directories to the sparse checkout | ||
| # and service directory variables. The alternative requires a lot of specific pom file and | ||
| # dependent pom file processing which is messy. The result here is that we may have an extra | ||
| # service directory being sunk up when it wasn't necessary. The overall time of that sync | ||
| # is a measured in seconds which isn't expensive and far outweighs the alternative mentioned | ||
| # above. | ||
| foreach ($project in $unreleasedList) { | ||
| if ($sparseCheckoutDirHash.ContainsKey($project)) { | ||
| $sparseCheckoutDirectories += $sparseCheckoutDirHash[$project] | ||
| } else { | ||
| LogError("Did not find unreleased $project in any pom files") | ||
| $script:FoundError = $true | ||
| } | ||
| } | ||
|
|
||
| if ($script:FoundError) { | ||
| exit 1 | ||
| } | ||
|
|
||
| $temp = ConvertTo-Json @($sparseCheckoutDirectories | Sort-Object | Get-Unique) -Compress | ||
| Write-Host "setting env variable SparseCheckoutDirectories = $temp" | ||
| Write-Host "##vso[task.setvariable variable=SparseCheckoutDirectories;]$temp" | ||
|
There was a problem hiding this comment. I'm not sure what the limit is but I suspect there is a limit on the size of a DevOps variable or a Env variable. So might be something we need to think about. There was a problem hiding this comment. @weshaggard - Previously the sparse checkout was granular down to the artifact's directory in some cases there were 120+ items added to the sparse checkout (like core's FromSource run for example). I changed things to just simply do the service directory which turned 120+ into like 30. If there's a limit it would have been hit long before I'd made my changes which dramatically trimmed things down. There was a problem hiding this comment. I can't find the docs for it right now, but I remember the limit being large enough that it hasn't been a concern thus far. |
||
| $temp = ConvertTo-Json @($serviceDirectories | Sort-Object | Get-Unique) -Compress | ||
| Write-Host "setting env variable ServiceDirectories = $temp" | ||
| Write-Host "##vso[task.setvariable variable=ServiceDirectories;]$temp" | ||
| $ElapsedTime = $(get-date) - $StartTime | ||
| $TotalRunTime = "{0:HH:mm:ss}" -f ([datetime]$ElapsedTime.Ticks) | ||
| Write-Host "Total run time=$($TotalRunTime)" | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did you intent to move the PreBuildSteps after the sparse checkout?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was intentional, otherwise there's a chicken or egg problem with the pre build scripts not being checked out with the initial sparse checkout pass.