Merge master into feature/iot_pnp (#1520)

* Make sure we always log in as app or provisioner (#1480) This also adds previously-required parameters back into @PSBoundParameters to pass down to pre- and post-scripts. Co-authored-by: Heath Stewart <heaths@microsoft.com> * update changelog for beta 2 (#1484) * Sync eng/common directory with azure-sdk-tools for PR 1163 (#1483) * Update subscription configuration schema to include new parameters * Support platform specific arm template parameters and legacy hashtable format * Update arm template parameter comment to include top level key * Restore AdditionalParameters. Merge ArmTemplateParameters from stringified hash literal * Handle duplicate keys more explicitly for arm and env vars * Regenerate New-TestResources.ps1 markdown * revert variable name to environmentVariables to fix post-scripts * Handle empty arm template parameters better * Remove arm template parameter merge logic from deploy template * Add merge hashes function to New-TestResources.ps1 * Add merge hashes function to New-TestResources.ps1 * Add env variable overwrite warning. Use ContainsKey checks * Temporarily manually fix invalid generated markdown links Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com> * Sync eng/common directory with azure-sdk-tools for PR 1178 (#1487) * Refactor eng/common/README.md * Add doc directory Co-authored-by: Chidozie Ononiwu <chononiw@microsoft.com> * Fix broken eng/common doc link (#1489) Co-authored-by: Chidozie Ononiwu <chononiw@microsoft.com> * Explicitly ignore return of _az_span_token on c2d parsing (#1482) (#1490) * add latest static analyzer fix to changelog (#1491) * Sync eng/common directory with azure-sdk-tools for PR 1188 (#1502) * Add debug flag to arm deployment command * Only set debug preference when $CI is true Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com> * Remove invalid characters in basename sourced from username (#1503) Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com> * Sync eng/common directory with azure-sdk-tools for PR 1170 (#1481) * Added the preprocess scripts. * string array to string Co-authored-by: Sima Zhu <sizhu@microsoft.com> * Reorganization of samples readme (#1505) * Completion of Sample Readme update (#1508) * Add Invoke-DevOpsAPI.ps1, Add functions for Canceling and Listing Builds (#1474) Co-authored-by: Chidozie Ononiwu <chononiw@microsoft.com> * Change live test resource DeleteAfterHours tag to 8 hours (#1509) Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com> * adding notes about security on readme file (#1498) * adding notes about security on readme file * Update sdk/samples/iot/docs/how_to_iot_hub_esp8266_nodemcu.md Co-authored-by: Mollie Munoz <mollie.munoz@microsoft.com> * Update sdk/samples/iot/docs/how_to_iot_hub_esp8266_nodemcu.md Co-authored-by: Ahson Khan <ahson_ahmedk@yahoo.com> * Update sdk/samples/iot/docs/how_to_iot_hub_esp8266_nodemcu.md Co-authored-by: Ahson Khan <ahson_ahmedk@yahoo.com> * Update sdk/samples/iot/docs/how_to_iot_hub_esp8266_nodemcu.md Co-authored-by: Mollie Munoz <mollie.munoz@microsoft.com> * Update sdk/samples/iot/docs/how_to_iot_hub_esp8266_nodemcu.md Co-authored-by: Ahson Khan <ahson_ahmedk@yahoo.com> * Update sdk/samples/iot/docs/how_to_iot_hub_esp8266_nodemcu.md Co-authored-by: Mollie Munoz <mollie.munoz@microsoft.com> * Update how_to_iot_hub_esp8266_nodemcu.md * Update sdk/samples/iot/docs/how_to_iot_hub_esp8266_nodemcu.md Co-authored-by: Eric Wolz <ericwol@microsoft.com> * Update sdk/samples/iot/docs/how_to_iot_hub_esp8266_nodemcu.md Co-authored-by: Eric Wolz <ericwol@microsoft.com> * Update sdk/samples/iot/docs/how_to_iot_hub_esp8266_nodemcu.md Co-authored-by: Eric Wolz <ericwol@microsoft.com> Co-authored-by: Mollie Munoz <mollie.munoz@microsoft.com> Co-authored-by: Ahson Khan <ahson_ahmedk@yahoo.com> Co-authored-by: Eric Wolz <ericwol@microsoft.com> * Sync eng/common directory with azure-sdk-tools for PR 1202 (#1510) * Add debugging link on resource deployment failures to log output * Update aka link for live test help docs. Use here string and empty throw. Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com> * Sync eng/common directory with azure-sdk-tools for PR 1153 (#1513) * Improve Update-ChangeLog Logic * Updates to ChangeLog-Operations.ps1, copy-docs-to-blobstorage.ps1, Invoke-GitHubAPI.ps1 and Package-Properties.ps1 * More changeLog Logic Improvements * Update date parsing Co-authored-by: Chidozie Ononiwu <chononiw@microsoft.com> * Revert "Places where we have const* arguments in source, optionally mark the value as const as well. (#1261)" (#1517) This reverts commit 5723678. * Update AddAzureSDKforC.cmake to reference the latest tagged release of the SDK (#1495) * Update AddAzureSDKforC.cmake to reference the latest release of the SDK * Remove pnp from the tag being referenced. * Incrementing master CL and version to be above the next release from the feature branch (#1506) * Update LanguageSetting.ps1 (#1518) Co-authored-by: Azure SDK Bot <53356347+azure-sdk@users.noreply.github.com> Co-authored-by: Heath Stewart <heaths@microsoft.com> Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com> Co-authored-by: Chidozie Ononiwu <chononiw@microsoft.com> Co-authored-by: ewertons <ewertons@microsoft.com> Co-authored-by: Sima Zhu <sizhu@microsoft.com> Co-authored-by: Mollie Munoz <mollie.munoz@microsoft.com> Co-authored-by: Wellington Duraes <wellington.duraes@gmail.com> Co-authored-by: Ahson Khan <ahson_ahmedk@yahoo.com> Co-authored-by: Eric Wolz <ericwol@microsoft.com> Co-authored-by: Chidozie Ononiwu <31145988+chidozieononiwu@users.noreply.github.com>
Azure · Nov 16, 2020 · 2fb0d51 · 2fb0d51
1 parent 5c04ff6
commit 2fb0d51
Show file tree

Hide file tree

Showing 36 changed files with 863 additions and 272 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,7 +1,19 @@
 # Release History
 
-## 1.1.0-beta.3 (Unreleased)
+## 1.1.0-beta.4 (Unreleased)
 
+### New Features
+
+
+### Bug Fixes
+
+- [[#1472]](https://github.com/Azure/azure-sdk-for-c/pull/1472) Fix `az_iot_message_properties_next()` when the buffer in which the properties were kept was bigger than the length of characters in the buffer.
+
+### Other Changes and Improvements
+
+- [[#1473]](https://github.com/Azure/azure-sdk-for-c/pull/1473) Add remove server certificate validation on paho and ESP8266 samples.
+- [[#1449]](https://github.com/Azure/azure-sdk-for-c/pull/1449) Add basic reconnection capability for the ESP8266 sample.
+- [[#1490]](https://github.com/Azure/azure-sdk-for-c/pull/1490) Fix static analyzer flagging of non-checked return value in `az_iot_hub_client_c2d_parse_received_topic()`.
 
 ## 1.1.0-beta.2 (2020-11-11)
 

diff --git a/cmake-modules/AddAzureSDKforC.cmake b/cmake-modules/AddAzureSDKforC.cmake
@@ -17,7 +17,7 @@
 include(FetchContent)
 FetchContent_Declare(azuresdkforc
     GIT_REPOSITORY      https://github.com/Azure/azure-sdk-for-c.git
-    GIT_TAG             1.1.0-beta.1)
+    GIT_TAG             1.1.0-beta.2)
 FetchContent_GetProperties(azuresdkforc)
 if(NOT azuresdkforc_POPULATED)
     FetchContent_Populate(azuresdkforc)

diff --git a/eng/common/TestResources/New-TestResources.ps1 b/eng/common/TestResources/New-TestResources.ps1
@@ -152,6 +152,8 @@ if (!$templateFiles) {
 }
 
 $UserName =  if ($env:USER) { $env:USER } else { "${env:USERNAME}" }
+# Remove spaces, etc. that may be in $UserName
+$UserName = $UserName -replace '\W'
 
 # If no base name is specified use current user name
 if (!$BaseName) {
@@ -308,7 +310,7 @@ if ($CI) {
     if ($EnvironmentVariables.ContainsKey('AZURE_RESOURCEGROUP_NAME') -and `
         $EnvironmentVariables['AZURE_RESOURCEGROUP_NAME'] -ne $ResourceGroupName)
     {
-        Write-Warning ("Overwriting 'EnvironmentVariables.AZURE_RESOURCEGROUP_NAME' with value " + 
+        Write-Warning ("Overwriting 'EnvironmentVariables.AZURE_RESOURCEGROUP_NAME' with value " +
             "'$($EnvironmentVariables['AZURE_RESOURCEGROUP_NAME'])' " + "to new value '$($ResourceGroupName)'")
     }
     $EnvironmentVariables['AZURE_RESOURCEGROUP_NAME'] = $ResourceGroupName
@@ -385,7 +387,23 @@ foreach ($templateFile in $templateFiles) {
 
     Log "Deploying template '$templateFile' to resource group '$($resourceGroup.ResourceGroupName)'"
     $deployment = Retry {
-        New-AzResourceGroupDeployment -Name $BaseName -ResourceGroupName $resourceGroup.ResourceGroupName -TemplateFile $templateFile -TemplateParameterObject $templateFileParameters
+        $lastDebugPreference = $DebugPreference
+        try {
+            if ($CI) {
+                $DebugPreference = "Continue"
+            }
+            New-AzResourceGroupDeployment -Name $BaseName -ResourceGroupName $resourceGroup.ResourceGroupName -TemplateFile $templateFile -TemplateParameterObject $templateFileParameters
+        } catch {
+            Write-Output @"
+==================================================
+For help debugging live test provisioning issues,
+see http://aka.ms/azsdk/engsys/live-test-help,
+==================================================
+"@
+            throw
+        } finally {
+            $DebugPreference = $lastDebugPreference
+        }
     }
 
     if ($deployment.ProvisioningState -eq 'Succeeded') {

diff --git a/eng/common/TestResources/deploy-test-resources.yml b/eng/common/TestResources/deploy-test-resources.yml
@@ -1,7 +1,7 @@
 parameters:
   ServiceDirectory: not-set
   ArmTemplateParameters: '@{}'
-  DeleteAfterHours: 24
+  DeleteAfterHours: 8
   Location: ''
   SubscriptionConfiguration: $(sub-config-azure-cloud-test-resources)
 

diff --git a/eng/common/scripts/ChangeLog-Operations.ps1 b/eng/common/scripts/ChangeLog-Operations.ps1
@@ -1,6 +1,10 @@
 # Common Changelog Operations
+. "${PSScriptRoot}\logging.ps1"
+. "${PSScriptRoot}\SemVer.ps1"
 
-$RELEASE_TITLE_REGEX = "(?<releaseNoteTitle>^\#+.*(?<version>\b\d+\.\d+\.\d+([^0-9\s][^\s:]+)?)(\s(?<releaseStatus>\(Unreleased\)|\(\d{4}-\d{2}-\d{2}\)))?)"
+$RELEASE_TITLE_REGEX = "(?<releaseNoteTitle>^\#+.*(?<version>\b\d+\.\d+\.\d+([^0-9\s][^\s:]+)?)(\s+(?<releaseStatus>\(Unreleased\)|\(\d{4}-\d{2}-\d{2}\)))?)"
+$CHANGELOG_UNRELEASED_STATUS = "(Unreleased)"
+$CHANGELOG_DATE_FORMAT = "yyyy-MM-dd"
 
 # Returns a Collection of changeLogEntry object containing changelog info for all version present in the gived CHANGELOG
 function Get-ChangeLogEntries {
@@ -11,7 +15,7 @@ function Get-ChangeLogEntries {
 
   $changeLogEntries = @{}
   if (!(Test-Path $ChangeLogLocation)) {
-    Write-Error "ChangeLog[${ChangeLogLocation}] does not exist"
+    LogError "ChangeLog[${ChangeLogLocation}] does not exist"
     return $null
   }
 
@@ -23,9 +27,9 @@ function Get-ChangeLogEntries {
       if ($line -match $RELEASE_TITLE_REGEX) {
         $changeLogEntry = [pscustomobject]@{ 
           ReleaseVersion = $matches["version"]
-          ReleaseStatus  = $matches["releaseStatus"]
-          ReleaseTitle   = $line
-          ReleaseContent = @() # Release content without the version title
+          ReleaseStatus  =  $matches["releaseStatus"]
+          ReleaseTitle   = "## {0} {1}" -f $matches["version"], $matches["releaseStatus"]
+          ReleaseContent = @()
         }
         $changeLogEntries[$changeLogEntry.ReleaseVersion] = $changeLogEntry
       }
@@ -72,6 +76,7 @@ function Get-ChangeLogEntryAsString {
   return ChangeLogEntryAsString $changeLogEntry
 }
 
+
 function ChangeLogEntryAsString($changeLogEntry) {
   if (!$changeLogEntry) {
     return "[Missing change log entry]"
@@ -93,7 +98,7 @@ function Confirm-ChangeLogEntry {
   $changeLogEntry = Get-ChangeLogEntry -ChangeLogLocation $ChangeLogLocation -VersionString $VersionString
 
   if (!$changeLogEntry) {
-    Write-Error "ChangeLog[${ChangeLogLocation}] does not have an entry for version ${VersionString}."
+    LogError "ChangeLog[${ChangeLogLocation}] does not have an entry for version ${VersionString}."
     return $false
   }
 
@@ -103,28 +108,108 @@ function Confirm-ChangeLogEntry {
   Write-Host "-----"
 
   if ([System.String]::IsNullOrEmpty($changeLogEntry.ReleaseStatus)) {
-    Write-Error "Entry does not have a correct release status. Please ensure the status is set to a date '(yyyy-MM-dd)' or '(Unreleased)' if not yet released."
+    LogError "Entry does not have a correct release status. Please ensure the status is set to a date '($CHANGELOG_DATE_FORMAT)' or '$CHANGELOG_UNRELEASED_STATUS' if not yet released."
     return $false
   }
 
   if ($ForRelease -eq $True) {
-    if ($changeLogEntry.ReleaseStatus -eq "(Unreleased)") {
-      Write-Error "Entry has no release date set. Please ensure to set a release date with format 'yyyy-MM-dd'."
+    if ($changeLogEntry.ReleaseStatus -eq $CHANGELOG_UNRELEASED_STATUS) {
+      LogError "Entry has no release date set. Please ensure to set a release date with format '$CHANGELOG_DATE_FORMAT'."
       return $false
     }
+    else {
+      $status = $changeLogEntry.ReleaseStatus.Trim().Trim("()")
+      try {
+        [DateTime]$status
+      }
+      catch {
+          LogError "Invalid date [ $status ] passed as status for Version [$($changeLogEntry.ReleaseVersion)]."
+          return $false
+      }
+    }
 
     if ([System.String]::IsNullOrWhiteSpace($changeLogEntry.ReleaseContent)) {
-      Write-Error "Entry has no content. Please ensure to provide some content of what changed in this version."
+      LogError "Entry has no content. Please ensure to provide some content of what changed in this version."
       return $false
     }
   }
   return $true
 }
 
-function Set-TestChangeLog($TestVersion, $changeLogFile, $ReleaseEntry) {
-  Set-Content -Path $changeLogFile -Value @"
-# Release History
-## $TestVersion ($(Get-Date -f "yyyy-MM-dd"))
-- $ReleaseEntry
-"@
+function New-ChangeLogEntry {
+  param (
+    [Parameter(Mandatory = $true)]
+    [ValidateNotNullOrEmpty()]
+    [String]$Version,
+    [String]$Status=$CHANGELOG_UNRELEASED_STATUS,
+    [String[]]$Content
+  )
+
+  # Validate RelaseStatus
+  $Status = $Status.Trim().Trim("()")
+  if ($Status -ne "Unreleased") {
+    try {
+      $Status = ([DateTime]$Status).ToString($CHANGELOG_DATE_FORMAT)
+    }
+    catch {
+        LogWarning "Invalid date [ $Status ] passed as status for Version [$Version]. Please use a valid date in the format '$CHANGELOG_DATE_FORMAT' or use '$CHANGELOG_UNRELEASED_STATUS'"
+        return $null
+    }
+  }
+  $Status = "($Status)"
+
+  # Validate Version
+  try {
+    $Version = ([AzureEngSemanticVersion]::ParseVersionString($Version)).ToString()
+  }
+  catch {
+    LogWarning "Invalid version [ $Version ]."
+    return $null
+  }
+
+  if (!$Content) { $Content = @() }
+
+  $newChangeLogEntry = [pscustomobject]@{ 
+    ReleaseVersion = $Version
+    ReleaseStatus  = $Status
+    ReleaseTitle   = "## $Version $Status"
+    ReleaseContent = $Content
+  }
+
+  return $newChangeLogEntry
+}
+
+function Set-ChangeLogContent {
+  param (
+    [Parameter(Mandatory = $true)]
+    [String]$ChangeLogLocation,
+    [Parameter(Mandatory = $true)]
+    $ChangeLogEntries
+  )
+
+  $changeLogContent = @()
+  $changeLogContent += "# Release History"
+  $changeLogContent += ""
+
+  try
+  {
+    $VersionsSorted = [AzureEngSemanticVersion]::SortVersionStrings($ChangeLogEntries.Keys)
+  }
+  catch {
+    LogError "Problem sorting version in ChangeLogEntries"
+    return
+  }
+
+  foreach ($version in $VersionsSorted) {
+    $changeLogEntry = $ChangeLogEntries[$version]
+    $changeLogContent += $changeLogEntry.ReleaseTitle
+    if ($changeLogEntry.ReleaseContent.Count -eq 0) {
+      $changeLogContent += @("","")
+    }
+    else {
+      $changeLogContent += $changeLogEntry.ReleaseContent
+    }
+  }
+
+  Set-Content -Path $ChangeLogLocation -Value $changeLogContent
 }
diff --git a/eng/common/scripts/Get-PullRequestCreator.ps1 b/eng/common/scripts/Get-PullRequestCreator.ps1
@@ -6,14 +6,18 @@ param (
     [string]$RepoName,
 
     [Parameter(Mandatory = $true)]
-    $PullRequestNumber
+    $PullRequestNumber,
+
+    [Parameter(Mandatory = $true)]
+    [string]$AuthToken
 )
 
 . "${PSScriptRoot}\common.ps1"
 
 try
 {
-    $pullRequest = Get-GithubPullRequest -RepoOwner $RepoOwner -RepoName $RepoName -PullRequestNumber $PullRequestNumber
+    $pullRequest = Get-GithubPullRequest -RepoOwner $RepoOwner -RepoName $RepoName `
+    -PullRequestNumber $PullRequestNumber -AuthToken $AuthToken
     Write-Host "##vso[task.setvariable variable=System.PullRequest.Creator;]$($pullRequest.user.login)"
 }
 catch

diff --git a/eng/common/scripts/Invoke-DevOpsAPI.ps1 b/eng/common/scripts/Invoke-DevOpsAPI.ps1
@@ -0,0 +1,92 @@
+. "${PSScriptRoot}\logging.ps1"
+
+$DevOpsAPIBaseURI = "https://dev.azure.com/{0}/{1}/_apis/{2}/{3}?{4}api-version=6.0"
+
+function Get-DevOpsApiHeaders ($Base64EncodedToken) {
+  $headers = @{
+    Authorization = "Basic $Base64EncodedToken"
+  }
+  return $headers
+}
+
+function Start-DevOpsBuild {
+  param (
+    $Organization="azure-sdk",
+    $Project="internal",
+    [Parameter(Mandatory = $true)]
+    $SourceBranch,
+    [Parameter(Mandatory = $true)]
+    $DefinitionId,
+    [ValidateNotNullOrEmpty()]
+    [Parameter(Mandatory = $true)]
+    $Base64EncodedAuthToken
+  )
+
+  $uri = "$DevOpsAPIBaseURI" -F $Organization, $Project , "build" , "builds", ""
+
+  $parameters = @{
+    sourceBranch = $SourceBranch
+    definition = @{ id = $DefinitionId }
+  }
+
+  return Invoke-RestMethod `
+          -Method POST `
+          -Body ($parameters | ConvertTo-Json) `
+          -Uri $uri `
+          -Headers (Get-DevOpsApiHeaders -Base64EncodedToken $Base64EncodedAuthToken) `
+          -MaximumRetryCount 3 `
+          -ContentType "application/json"
+}
+
+function Update-DevOpsBuild {
+  param (
+    $Organization="azure-sdk",
+    $Project="internal",
+    [ValidateNotNullOrEmpty()]
+    [Parameter(Mandatory = $true)]
+    $BuildId,
+    $Status, # pass canceling to cancel build
+    [ValidateNotNullOrEmpty()]
+    [Parameter(Mandatory = $true)]
+    $Base64EncodedAuthToken
+  )
+
+  $uri = "$DevOpsAPIBaseURI" -F $Organization, $Project, "build", "builds/$BuildId", ""
+  $parameters = @{}
+
+  if ($Status) { $parameters["status"] = $Status}
+
+  return Invoke-RestMethod `
+          -Method PATCH `
+          -Body ($parameters | ConvertTo-Json) `
+          -Uri $uri `
+          -Headers (Get-DevOpsApiHeaders -Base64EncodedToken $Base64EncodedAuthToken) `
+          -MaximumRetryCount 3 `
+          -ContentType "application/json"
+}
+
+function Get-DevOpsBuilds {
+  param (
+    $Organization="azure-sdk",
+    $Project="internal",
+    $BranchName, #Should start with 'refs/heads/'
+    $Definitions, # Comma seperated string of definition IDs
+    $StatusFilter, # Comma seperated string 'cancelling, completed, inProgress, notStarted'
+    [ValidateNotNullOrEmpty()]
+    [Parameter(Mandatory = $true)]
+    $Base64EncodedAuthToken
+  )
+
+  $query = ""
+
+  if ($BranchName) { $query += "branchName=$BranchName&" }
+  if ($Definitions) { $query += "definitions=$Definitions&" }
+  if ($StatusFilter) { $query += "statusFilter=$StatusFilter&" }
+  $uri = "$DevOpsAPIBaseURI" -F $Organization, $Project , "build" , "builds", $query
+
+  return Invoke-RestMethod `
+          -Method GET `
+          -Uri $uri `
+          -Headers (Get-DevOpsApiHeaders -Base64EncodedToken $Base64EncodedAuthToken) `
+          -MaximumRetryCount 3
+}