A series of Public swagger update (modification) #6064

Yiliu-microsoft · 2019-05-22T07:58:19Z

Latest improvements:

MSFT employees can try out our new experience at OpenAPI Hub - one location for using our validation tools and finding your workflow.

Contribution checklist:

I have reviewed the documentation for the workflow.
Validation tools were run on swagger spec(s) and have all been fixed in this PR.
The OpenAPI Hub was used for checking validation status and next steps.

ARM API Review Checklist

Service team MUST add the "WaitForARMFeedback" label if the management plane API changes fall into one of the below categories.
adding/removing APIs.
adding/removing properties.
adding/removing API-version.
adding a new service in Azure.

Failure to comply may result in delays for manifest application. Note this does not apply to data plane APIs.

If you are blocked on ARM review and want to get the PR merged urgently, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.
Please follow the link to find more details on API review process.

update version

update

AutorestCI · 2019-05-22T07:59:05Z

Automation for azure-sdk-for-js

A PR has been created for you based on this PR content.

Once this PR will be merged, content will be added to your service PR:
Azure/azure-sdk-for-js#3778

AutorestCI · 2019-05-22T07:59:08Z

Automation for azure-sdk-for-ruby

Nothing to generate for azure-sdk-for-ruby

AutorestCI · 2019-05-22T07:59:10Z

Automation for azure-sdk-for-python

A PR has been created for you based on this PR content.

Once this PR will be merged, content will be added to your service PR:
Azure/azure-sdk-for-python#5747

AutorestCI · 2019-05-22T08:00:17Z

Automation for azure-sdk-for-go

A PR has been created for you based on this PR content.

Once this PR will be merged, content will be added to your service PR:
Azure/azure-sdk-for-go#4990

AutorestCI · 2019-05-22T08:19:26Z

Automation for azure-sdk-for-java

Encountered a Subprocess error: (azure-sdk-for-java)

Command: ['/usr/local/bin/autorest', '/tmp/tmpxkumramh/rest/specification/datafactory/resource-manager/readme.md', '--perform-load=false', '--swagger-to-sdk', '--output-artifact=configuration.json', '--input-file=foo', '--output-folder=/tmp/tmp5mro5nd8']
Finished with return code 7
and output:

AutoRest code generation utility [version: 2.0.4283; node: v8.12.0]
(C) 2018 Microsoft Corporation.
https://aka.ms/autorest
Failure:
Error: Unable to start AutoRest Core from /root/.autorest/@microsoft.azure_autorest-core@2.0.4373/node_modules/@microsoft.azure/autorest-core
Error: Unable to start AutoRest Core from /root/.autorest/@microsoft.azure_autorest-core@2.0.4373/node_modules/@microsoft.azure/autorest-core
    at main (/opt/node_modules/autorest/dist/app.js:232:19)
    at <anonymous>

/root/.autorest/@microsoft.azure_autorest-core@2.0.4373/node_modules/@microsoft.azure/autorest-core/dist/app.js:33
    autorest_core_1.Shutdown();
    ^
ReferenceError: autorest_core_1 is not defined
    at process.on (/root/.autorest/@microsoft.azure_autorest-core@2.0.4373/node_modules/@microsoft.azure/autorest-core/dist/app.js:33:5)
    at emitOne (events.js:121:20)
    at process.emit (events.js:211:7)
    at process.emit (/node_modules/source-map-support/source-map-support.js:439:21)
fs.js:612
  return binding.close(fd);
                 ^

Error: EBADF: bad file descriptor, close
    at Object.fs.closeSync (fs.js:612:18)
    at StaticVolumeFile.shutdown (/opt/node_modules/autorest/dist/static-loader.js:352:10)
    at StaticFilesystem.shutdown (/opt/node_modules/autorest/dist/static-loader.js:406:17)
    at process.exit.n [as exit] (/opt/node_modules/autorest/dist/static-loader.js:169:11)
    at printErrorAndExit (/node_modules/source-map-support/source-map-support.js:423:11)
    at process.emit (/node_modules/source-map-support/source-map-support.js:435:16)
    at process._fatalException (bootstrap_node.js:391:26)

azuresdkci · 2019-05-22T08:34:23Z

Can one of the admins verify this patch?

...y/resource-manager/Microsoft.DataFactory/preview/2017-09-01-preview/entityTypes/Dataset.json

...tafactory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/entityTypes/Pipeline.json

hvermis · 2019-05-23T18:21:10Z

@Yiliu-microsoft Please make sure to fix all checks as well

hvermis

Please address comments

...atafactory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/entityTypes/Dataset.json

...tafactory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/entityTypes/Pipeline.json

REST Spec PR 'Azure/azure-rest-api-specs#6064' REST Spec PR Author 'Yiliu-microsoft' REST Spec PR Last commit

adxsdknet · 2019-05-29T07:49:13Z

Automation for azure-sdk-for-net

A PR has been created for you:
Azure/azure-sdk-for-net#6429
.NET SDK Commits:
adxsdknet/azure-sdk-for-net@0d8ebe7
adxsdknet/azure-sdk-for-net@c85b5c9
adxsdknet/azure-sdk-for-net@cde3db1

shahabhijeet

I will need an approval from someone from your team in addition to addressing to couple of breaking changes you have introduced in this PR

shahabhijeet · 2019-05-31T17:32:29Z

...atafactory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/entityTypes/Dataset.json

@@ -197,7 +330,7 @@
    },
    "JsonFormatFilePattern": {
      "description": "JSON format file pattern. A property of JsonFormat.",
-      "type": "string",
+      "type": "object",


breaking change in a stable version?
Any reason this cannot be done in the next API version?

Ok, I invited Hermine to explain this change, please wait a moment.

@shahabhijeet the type is changing from string to object, so it's not API breaking change as our service will accept both.
We could mark the SDK-s with breaking tag if you think it's appropriate, however we had another PR like this that wasn't marked.

@hvermis it is a binary breaking change.
I was concerned about a scenario where your customer has written code against your model as

JsonFormat.FilePattern.ToLower().StartWith("array")

Also what I noticed that you have modeled your enum as "modelAsString", but you would like to make a change from string to object?

@shahabhijeet We'll mark the SDK release as breaking.

shahabhijeet · 2019-05-31T17:32:38Z

...atafactory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/entityTypes/Dataset.json

@@ -313,7 +446,7 @@
    "CompressionLevel": {
      "title": "Microsoft.Azure.Management.DataFactories.Models.CompressionLevel",
      "description": "All available compression levels.",
-      "type": "string",
+      "type": "object",


breaking change

...tafactory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/entityTypes/Pipeline.json

openapi-sdkautomation · 2019-06-04T02:55:06Z

SDK Automation [Logs] (Generated from 41ae155)

shahabhijeet

I have asked ARM review for this.
What I would like to know is, can you make such changes in stable APIs?
I would like to understand such process, which will help me in future reviews.

shahabhijeet · 2019-06-04T07:30:30Z

...atafactory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/entityTypes/Dataset.json

@@ -197,7 +330,7 @@
    },
    "JsonFormatFilePattern": {
      "description": "JSON format file pattern. A property of JsonFormat.",
-      "type": "string",
+      "type": "object",


@hvermis it is a binary breaking change.
I was concerned about a scenario where your customer has written code against your model as

JsonFormat.FilePattern.ToLower().StartWith("array")

Also what I noticed that you have modeled your enum as "modelAsString", but you would like to make a change from string to object?

hvermis · 2019-06-04T17:38:16Z

@shahabhijeet All these changes are again to support oneOf scenario. The reason we are changing types to object is so our customers can parametrize those properties. Our UI and backend already allow this, only SDK-s are pending.

hvermis · 2019-06-06T03:11:53Z

@shahabhijeet and @KrisBash Could you please help get this PR in? It contains a lot of features that our customers are waiting for and we need to release our SDKs.

KrisBash

Discussion about SDK breaking changes aside, there were quite a few properties changed to type: object, but with string enums and modelAsString=true. That seems spurious?

KrisBash · 2019-06-07T04:23:59Z

.../resource-manager/Microsoft.DataFactory/preview/2017-09-01-preview/entityTypes/Pipeline.json

@@ -1148,10 +1148,11 @@
    },
    "StoredProcedureParameterType": {
      "description": "Stored procedure parameter type.",
-      "type": "string",
+      "type": "object",


If all allowed enum values are a string, why change to object?

This is to support Expressions on these properties. OpenApi 2.0 doesn't support oneOf, and we are forced to use object type in order to support parameterization, which is an important feature of ADF for our customers. It allows them to treat their entities as templates and pass different values for properties.

KrisBash · 2019-06-07T04:24:24Z

.../resource-manager/Microsoft.DataFactory/preview/2017-09-01-preview/entityTypes/Pipeline.json

@@ -1206,7 +1207,7 @@
      "properties": {
        "writeBehavior": {
          "description": "The write behavior for the operation. Default is 'Insert'.",
-          "type": "string",
+          "type": "object",


Like the above, does this need to be an object if allowed values are all strings?

KrisBash · 2019-06-07T04:27:58Z

...tory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/entityTypes/LinkedService.json

+          "type": "object",
+          "description": "Username to access the SAP server where the table is located. Type: string (or Expression with resultType string)."
+        },
+        "password": {


Friendly reminder: please ensure that the server never returns secrets on get nor logs requests.

This is already taken care of - we encrypt all secrets, and our service returns only encrypted values.

hvermis · 2019-06-07T18:36:38Z

@KrisBash This is the PR that we need merged. Can you please finish reviewing this one? I will follow up with Yi to close any others, if they are irrelevant.

nschonni · 2019-06-07T22:01:08Z

FYI, you can add bzip to https://github.com/Azure/azure-rest-api-specs/blob/master/custom-words.txt if you want the spellcheck to pass. The other issue it was flagging in a different PR was fixed in master

hvermis · 2019-06-10T18:29:43Z

@KrisBash @shahabhijeet We need to release this new SDK with Powershell release that is going out on 6/18.
Can you please help as we have a lot of customers waiting for the features in this PR?

hvermis · 2019-06-11T19:03:56Z

...tafactory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/entityTypes/Pipeline.json

@@ -205,6 +205,511 @@
        "type": "object"
      }
    },
+    "ConnectorReadSetting": {


Why are the type names here Setting vs Settings?

Applies to all names

I change "description": "Connector read settings." to "description": "Connector read setting."

KrisBash

No objection from ARM, assuming changes are legitimate in SDK

Yiliu added 6 commits March 14, 2019 15:56

Modify the public swagger by reference to private swagger

f756788

Merge remote-tracking branch 'upstream/master'

3015288

update version

Merge remote-tracking branch 'upstream/master'

2727b7c

update

Merge remote-tracking branch 'upstream/master'

047db66

Merge remote-tracking branch 'upstream/master'

1a04dc6

update

47c70ac

azuresdkci assigned shahabhijeet May 22, 2019

azuresdkci requested a review from shahabhijeet May 22, 2019 08:20

hvermis reviewed May 22, 2019

View reviewed changes

...y/resource-manager/Microsoft.DataFactory/preview/2017-09-01-preview/entityTypes/Dataset.json Outdated Show resolved Hide resolved

hvermis reviewed May 22, 2019

View reviewed changes

...tafactory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/entityTypes/Pipeline.json Outdated Show resolved Hide resolved

hvermis reviewed May 22, 2019

View reviewed changes

...tafactory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/entityTypes/Pipeline.json Outdated Show resolved Hide resolved

nschonni reviewed May 22, 2019

View reviewed changes

...tafactory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/entityTypes/Pipeline.json Outdated Show resolved Hide resolved

hvermis self-requested a review May 23, 2019 18:21

hvermis requested changes May 23, 2019

View reviewed changes

davidzhaoyue requested changes May 29, 2019

View reviewed changes

...atafactory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/entityTypes/Dataset.json Show resolved Hide resolved

davidzhaoyue requested changes May 29, 2019

View reviewed changes

...tafactory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/entityTypes/Pipeline.json Show resolved Hide resolved

finally modify

4e1af64

This was referenced May 29, 2019

[AutoPR datafactory/resource-manager] A series of Public swagger update (modification) Azure/azure-sdk-for-python#5510

Closed

[AutoPR datafactory/resource-manager] A series of Public swagger update (modification) Azure/azure-sdk-for-go#4888

Closed

davidzhaoyue requested changes May 29, 2019

View reviewed changes

...tafactory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/entityTypes/Pipeline.json Outdated Show resolved Hide resolved

adxsdknet added a commit to adxsdknet/azure-sdk-for-net that referenced this pull request May 29, 2019

.NET SDK Resource Provider:'DataFactory'

0d8ebe7

REST Spec PR 'Azure/azure-rest-api-specs#6064' REST Spec PR Author 'Yiliu-microsoft' REST Spec PR Last commit

AutorestCI mentioned this pull request May 29, 2019

[AutoPR datafactory/resource-manager] A series of Public swagger update (modification) Azure/azure-sdk-for-js#3245

Closed

modify

e64566f

shahabhijeet requested changes May 31, 2019

View reviewed changes

hvermis reviewed Jun 4, 2019

View reviewed changes

...tafactory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/entityTypes/Pipeline.json Outdated Show resolved Hide resolved

hvermis approved these changes Jun 4, 2019

View reviewed changes

modify mappings schemaMappihng columnMapping description

e310206

AutorestCI mentioned this pull request Jun 4, 2019

[AutoPR datafactory/resource-manager] A series of Public swagger update (modification) Azure/azure-sdk-for-js#3425

Closed

shahabhijeet added WaitForARMFeedback <valid label in PR review process> add this label when ARM review is required Breaking Change Review labels Jun 4, 2019

shahabhijeet requested changes Jun 4, 2019

View reviewed changes

KrisBash added the ARMReviewInProgress label Jun 5, 2019

KrisBash suggested changes Jun 7, 2019

View reviewed changes

KrisBash added ARMChangesRequested <valid label in PR review process>add this label when require changes after ARM review and removed ARMReviewInProgress labels Jun 7, 2019

modify SapTable

922aef8

AutorestCI mentioned this pull request Jun 11, 2019

[AutoPR datafactory/resource-manager] A series of Public swagger update (modification) Azure/azure-sdk-for-js#3657

Closed

hvermis reviewed Jun 11, 2019

View reviewed changes

modify ConnectorReadSetting

41ae155

AutorestCI mentioned this pull request Jun 12, 2019

[AutoPR datafactory/resource-manager] A series of Public swagger update (modification) Azure/azure-sdk-for-js#3689

Closed

KrisBash approved these changes Jun 13, 2019

View reviewed changes

shahabhijeet merged commit 9060257 into Azure:master Jun 13, 2019

AutorestCI mentioned this pull request Jun 13, 2019

[AutoPR datafactory/resource-manager] A series of Public swagger update (modification) Azure/azure-sdk-for-js#3777

Merged

A series of Public swagger update (modification) #6064

A series of Public swagger update (modification) #6064

Conversation

Yiliu-microsoft commented May 22, 2019

Latest improvements:

Contribution checklist:

ARM API Review Checklist

AutorestCI commented May 22, 2019 • edited Loading

Automation for azure-sdk-for-js

AutorestCI commented May 22, 2019 • edited Loading

Automation for azure-sdk-for-ruby

AutorestCI commented May 22, 2019 • edited Loading

Automation for azure-sdk-for-python

AutorestCI commented May 22, 2019 • edited Loading

Automation for azure-sdk-for-go

AutorestCI commented May 22, 2019 • edited Loading

Automation for azure-sdk-for-java

azuresdkci commented May 22, 2019

hvermis commented May 23, 2019

hvermis left a comment

Choose a reason for hiding this comment

adxsdknet commented May 29, 2019 • edited Loading

Automation for azure-sdk-for-net

shahabhijeet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openapi-sdkautomation bot commented Jun 4, 2019 • edited Loading

SDK Automation [Logs] (Generated from 41ae155)

shahabhijeet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hvermis commented Jun 4, 2019

hvermis commented Jun 6, 2019

KrisBash left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hvermis commented Jun 7, 2019

nschonni commented Jun 7, 2019

hvermis commented Jun 10, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KrisBash left a comment

Choose a reason for hiding this comment

AutorestCI commented May 22, 2019 •

edited

Loading

AutorestCI commented May 22, 2019 •

edited

Loading

AutorestCI commented May 22, 2019 •

edited

Loading

AutorestCI commented May 22, 2019 •

edited

Loading

AutorestCI commented May 22, 2019 •

edited

Loading

adxsdknet commented May 29, 2019 •

edited

Loading

openapi-sdkautomation bot commented Jun 4, 2019 •

edited

Loading