Azure OpenAI: 2023-12-01-preview TypeSpec fix for ChatMessageImageContentItem #26967

trrwilson · 2023-12-06T18:09:26Z

The gpt-4-vision-preview variant of the /chat/completions API accepts a broader structure for the content property that includes a new capability to include image_url items. OpenAI does not yet document everything thoroughly but their spec as well as ours for Azure OpenAI make it clear that we have a subtle but important bug in what we just merged for the .tsp representation: detail is one level higher than it should be.

This very small patch just adjusts the level to match the spec.

Credit and sincere thanks to @extraes for catching this proactively, per this .NET repo comment!
Azure/azure-sdk-for-net#40078 (comment)

openapi-pipeline-app · 2023-12-06T18:09:31Z

Next Steps to Merge

✔️ All automated merging requirements have been met! Refer to step 4 in the PR workflow diagram (even if your PR is for data plane, not ARM).

openapi-pipeline-app · 2023-12-06T18:09:31Z

Swagger Validation Report

️️✔️BreakingChange succeeded [Detail] [Expand]

There are no breaking changes.

️️✔️Breaking Change(Cross-Version) succeeded [Detail] [Expand]

There are no breaking changes.

️️✔️CredScan succeeded [Detail] [Expand]

There is no credential detected.

️️✔️LintDiff succeeded [Detail] [Expand]

Validation passes for LintDiff.

️️✔️Avocado succeeded [Detail] [Expand]

Validation passes for Avocado.

️️✔️SwaggerAPIView succeeded [Detail] [Expand]

️️✔️TypeSpecAPIView succeeded [Detail] [Expand]

️️✔️ModelValidation succeeded [Detail] [Expand]

Validation passes for ModelValidation.

️️✔️SemanticValidation succeeded [Detail] [Expand]

Validation passes for SemanticValidation.

️️✔️PoliCheck succeeded [Detail] [Expand]

Validation passed for PoliCheck.

️️✔️SpellCheck succeeded [Detail] [Expand]

Validation passes for SpellCheck.

️️✔️Lint(RPaaS) succeeded [Detail] [Expand]

Validation passes for Lint(RPaaS).

️️✔️PR Summary succeeded [Detail] [Expand]

Validation passes for Summary.

️️✔️Automated merging requirements met succeeded [Detail] [Expand]

_{Posted by Swagger Pipeline | How to fix these errors?}

openapi-pipeline-app · 2023-12-06T18:09:32Z

Swagger Generation Artifacts

️️✔️ApiDocPreview succeeded [Detail] [Expand]

️️✔️SDK Breaking Change Tracking succeeded [Detail] [Expand]

Breaking Changes Tracking

️❌ azure-sdk-for-net-track2 failed [Detail]

❌Failed [Logs] Generate from 8c255b352069c13e89c21e10a6cd3e44ff75ab14. SDK Automation 14.0.0

command	pwsh ./eng/scripts/Automation-Sdk-Init.ps1 ../azure-sdk-for-net_tmp/initInput.json ../azure-sdk-for-net_tmp/initOutput.json
command	pwsh ./eng/scripts/Invoke-GenerateAndBuildV2.ps1 ../azure-sdk-for-net_tmp/generateInput.json ../azure-sdk-for-net_tmp/generateOutput.json
cmderr	[Invoke-GenerateAndBuildV2.ps1] �[31;1mGeneratePackage: �[0m/mnt/vss/_work/1/s/azure-sdk-for-net/eng/scripts/Invoke-GenerateAndBuildV2.ps1:131
cmderr	[Invoke-GenerateAndBuildV2.ps1] �[36;1mLine |
cmderr	[Invoke-GenerateAndBuildV2.ps1] �[36;1m 131 | �[0m               �[36;1mGeneratePackage `�[0m
cmderr	[Invoke-GenerateAndBuildV2.ps1] �[36;1m     | �[31;1m               ~~~~~~~~~~~~~~~~~
cmderr	[Invoke-GenerateAndBuildV2.ps1] �[31;1m�[36;1m     | �[31;1mFailed to generate sdk. exit code: False
cmderr	[Invoke-GenerateAndBuildV2.ps1] �[0m
cmderr	[Invoke-GenerateAndBuildV2.ps1] �[31;1mGet-ChildItem: �[0m/mnt/vss/_work/1/s/azure-sdk-for-net/eng/scripts/automation/GenerateAndBuildLib.ps1:807
cmderr	[Invoke-GenerateAndBuildV2.ps1] �[36;1mLine |
cmderr	[Invoke-GenerateAndBuildV2.ps1] �[36;1m 807 | �[0m … rtifacts += �[36;1mGet-ChildItem $artifactsPath -Filter *.nupkg -exclude *.s�[0m …
cmderr	[Invoke-GenerateAndBuildV2.ps1] �[36;1m     | �[31;1m               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cmderr	[Invoke-GenerateAndBuildV2.ps1] �[31;1m�[36;1m     | �[31;1mCannot find path
cmderr	[Invoke-GenerateAndBuildV2.ps1] �[36;1m     | �[31;1m'/mnt/vss/_work/1/s/azure-sdk-for-net/artifacts/packages/Debug/' because
cmderr	[Invoke-GenerateAndBuildV2.ps1] �[36;1m     | �[31;1mit does not exist.
cmderr	[Invoke-GenerateAndBuildV2.ps1] �[0m
cmderr	[Invoke-GenerateAndBuildV2.ps1] �[31;1mGeneratePackage: �[0m/mnt/vss/_work/1/s/azure-sdk-for-net/eng/scripts/Invoke-GenerateAndBuildV2.ps1:131
cmderr	[Invoke-GenerateAndBuildV2.ps1] �[36;1mLine |
cmderr	[Invoke-GenerateAndBuildV2.ps1] �[36;1m 131 | �[0m               �[36;1mGeneratePackage `�[0m
cmderr	[Invoke-GenerateAndBuildV2.ps1] �[36;1m     | �[31;1m               ~~~~~~~~~~~~~~~~~
cmderr	[Invoke-GenerateAndBuildV2.ps1] �[31;1m�[36;1m     | �[31;1mFailed to generate sdk artifact
cmderr	[Invoke-GenerateAndBuildV2.ps1] �[0m

❌Azure.AI.OpenAI [View full logs] [Preview SDK Changes]
```
info	[Changelog]
```

️❌ azure-sdk-for-java failed [Detail]

❌Pipeline Framework Failed [Logs] Generate from 8c255b352069c13e89c21e10a6cd3e44ff75ab14. SDK Automation 14.0.0

command	./eng/mgmt/automation/init.sh ../azure-sdk-for-java_tmp/initInput.json ../azure-sdk-for-java_tmp/initOutput.json
cmderr	[init.sh] [notice] A new release of pip is available: 23.0.1 -> 23.3.1
cmderr	[init.sh] [notice] To update, run: pip install --upgrade pip
cmderr	[init.sh] [notice] A new release of pip is available: 23.0.1 -> 23.3.1
cmderr	[init.sh] [notice] To update, run: pip install --upgrade pip
command	./eng/mgmt/automation/generate.py ../azure-sdk-for-java_tmp/generateInput.json ../azure-sdk-for-java_tmp/generateOutput.json
cmderr	[generate.py] WARN EBADENGINE Unsupported engine {
cmderr	[generate.py] npm WARN EBADENGINE   package: '@typespec/compiler@0.50.0',
cmderr	[generate.py] npm WARN EBADENGINE   required: { node: '>=18.0.0' },
cmderr	[generate.py] npm WARN EBADENGINE   current: { node: 'v16.20.2', npm: '8.19.4' }
cmderr	[generate.py] npm WARN EBADENGINE }
cmderr	[generate.py] npm WARN EBADENGINE Unsupported engine {
cmderr	[generate.py] npm WARN EBADENGINE   package: '@typespec/rest@0.50.0',
cmderr	[generate.py] npm WARN EBADENGINE   required: { node: '>=18.0.0' },
cmderr	[generate.py] npm WARN EBADENGINE   current: { node: 'v16.20.2', npm: '8.19.4' }
cmderr	[generate.py] npm WARN EBADENGINE }
cmderr	[generate.py] npm WARN EBADENGINE Unsupported engine {
cmderr	[generate.py] npm WARN EBADENGINE   package: '@typespec/http@0.50.0',
cmderr	[generate.py] npm WARN EBADENGINE   required: { node: '>=18.0.0' },
cmderr	[generate.py] npm WARN EBADENGINE   current: { node: 'v16.20.2', npm: '8.19.4' }
cmderr	[generate.py] npm WARN EBADENGINE }
cmderr	[generate.py] npm WARN EBADENGINE Unsupported engine {
cmderr	[generate.py] npm WARN EBADENGINE   package: '@typespec/openapi@0.50.0',
cmderr	[generate.py] npm WARN EBADENGINE   required: { node: '>=18.0.0' },
cmderr	[generate.py] npm WARN EBADENGINE   current: { node: 'v16.20.2', npm: '8.19.4' }
cmderr	[generate.py] npm WARN EBADENGINE }
cmderr	[generate.py] npm WARN EBADENGINE Unsupported engine {
cmderr	[generate.py] npm WARN EBADENGINE   package: '@typespec/versioning@0.50.0',
cmderr	[generate.py] npm WARN EBADENGINE   required: { node: '>=18.0.0' },
cmderr	[generate.py] npm WARN EBADENGINE   current: { node: 'v16.20.2', npm: '8.19.4' }
cmderr	[generate.py] npm WARN EBADENGINE }
cmderr	[generate.py] npm WARN EBADENGINE Unsupported engine {
cmderr	[generate.py] npm WARN EBADENGINE   package: '@azure-tools/typespec-azure-core@0.36.0',
cmderr	[generate.py] npm WARN EBADENGINE   required: { node: '>=18.0.0' },
cmderr	[generate.py] npm WARN EBADENGINE   current: { node: 'v16.20.2', npm: '8.19.4' }
cmderr	[generate.py] npm WARN EBADENGINE }
cmderr	[generate.py] npm WARN EBADENGINE Unsupported engine {
cmderr	[generate.py] npm WARN EBADENGINE   package: '@azure-tools/typespec-client-generator-core@0.36.1',
cmderr	[generate.py] npm WARN EBADENGINE   required: { node: '>=18.0.0' },
cmderr	[generate.py] npm WARN EBADENGINE   current: { node: 'v16.20.2', npm: '8.19.4' }
cmderr	[generate.py] npm WARN EBADENGINE }
cmderr	[generate.py]
cmderr	[generate.py] npm notice New major version of npm available! 8.19.4 -> 10.2.4
cmderr	[generate.py] npm notice Changelog: <https://github.com/npm/cli/releases/tag/v10.2.4>
cmderr	[generate.py] npm notice Run `npm install -g npm@10.2.4` to update!
cmderr	[generate.py] npm notice
cmderr	[generate.py] Exception in thread "main" java.lang.RuntimeException: Failed to complete postprocessing.
cmderr	[generate.py] 	at com.azure.autorest.postprocessor.Postprocessor.postProcess(Postprocessor.java:109)
cmderr	[generate.py] 	at com.azure.typespec.Main.handleDPG(Main.java:138)
cmderr	[generate.py] 	at com.azure.typespec.Main.main(Main.java:90)
cmderr	[generate.py] Caused by: java.lang.RuntimeException: Unable to complete customization
cmderr	[generate.py] 	at com.azure.autorest.postprocessor.Postprocessor.postProcess(Postprocessor.java:102)
cmderr	[generate.py] 	... 2 more
cmderr	[generate.py] Caused by: java.lang.NullPointerException
cmderr	[generate.py] 	at com.azure.autorest.postprocessor.Postprocessor.postProcess(Postprocessor.java:97)
cmderr	[generate.py] 	... 2 more
cmderr	[generate.py] Emitter "@azure-tools/typespec-java" crashed! This is a bug.
cmderr	[generate.py] Please file an issue at https://github.com/Azure/autorest.java/issues
cmderr	[generate.py] Error: JAR ended with code '1'.
cmderr	[generate.py]     at ChildProcess.<anonymous> (file:///mnt/vss/_work/1/s/azure-sdk-for-java/sdk/openai/azure-ai-openai/TempTypeSpecFiles/OpenAI.Inference/node_modules/@azure-tools/typespec-java/dist/src/emitter.js:106:37)
cmderr	[generate.py]     at ChildProcess.emit (node:events:513:28)
cmderr	[generate.py]     at Process.ChildProcess._handle.onexit (node:internal/child_process:293:12)
cmderr	[generate.py] --------------------------------------------------
cmderr	[generate.py] Library Version                0.11.3
cmderr	[generate.py] TypeSpec Compiler Version      0.50.0
cmderr	[generate.py] --------------------------------------------------
cmderr	[generate.py] �[31;1mWrite-Error: �[31;1mFailed to generate sdk project at /mnt/vss/_work/1/s/azure-sdk-for-java/sdk/openai/azure-ai-openai�[0m
error	Fatal error: The "path" argument must be of type string. Received undefined
The "path" argument must be of type string. Received undefined
TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received undefined
    at new NodeError (node:internal/errors:387:5)
    at validateString (node:internal/validators:162:11)
    at Object.basename (node:path:1309:5)
    at Object.exports.getPackageData (/mnt/vss/_work/1/a/unified-pipeline-runtime/private/openapi-sdk-automation/lib/types/PackageData.js:38:82)
    at /mnt/vss/_work/1/a/unified-pipeline-runtime/private/openapi-sdk-automation/lib/automation/workflow.js:210:194
    at Array.map (<anonymous>)
    at workflowHandleReadmeMdOrTypeSpecProject (/mnt/vss/_work/1/a/unified-pipeline-runtime/private/openapi-sdk-automation/lib/automation/workflow.js:210:164)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Object.exports.workflowMain (/mnt/vss/_work/1/a/unified-pipeline-runtime/private/openapi-sdk-automation/lib/automation/workflow.js:98:13)
    at async Object.exports.sdkAutoMain (/mnt/vss/_work/1/a/unified-pipeline-runtime/private/openapi-sdk-automation/lib/automation/entrypoint.js:211:13)
    at async /mnt/vss/_work/1/a/unified-pipeline-runtime/private/openapi-sdk-automation/lib/cli/cli.js:20:18

️🔄 azure-sdk-for-js inProgress [Detail]

_{Posted by Swagger Pipeline | How to fix these errors?}

openapi-pipeline-app · 2023-12-06T18:09:33Z

Generated ApiView

Language	Package Name	ApiView Link
TypeSpec	OpenAI.Inference	https://apiview.dev/Assemblies/Review/471ddb231c1a44059535d353fb4ca11c

fix for reported and confirmed level mismatch of detail

0b377a4

trrwilson requested a review from yangyuan as a code owner December 6, 2023 18:09

openapi-assignment-bot bot assigned lmazuel Dec 6, 2023

AzureRestAPISpecReview added the TypeSpec Authored with TypeSpec label Dec 6, 2023

lmazuel approved these changes Dec 6, 2023

View reviewed changes

lmazuel merged commit 30688b3 into main Dec 6, 2023
27 of 30 checks passed

lmazuel deleted the user/travisw/aoai-2023-12-01-preview-image-content-followup branch December 6, 2023 18:19

azure-sdk mentioned this pull request Dec 6, 2023

[AutoPR Azure.AI.OpenAI] Azure OpenAI: 2023-12-01-preview TypeSpec fix for ChatMessageImageContentItem azure-sdk/azure-sdk-for-net#6449

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Azure OpenAI: 2023-12-01-preview TypeSpec fix for ChatMessageImageContentItem #26967

Azure OpenAI: 2023-12-01-preview TypeSpec fix for ChatMessageImageContentItem #26967

trrwilson commented Dec 6, 2023

openapi-pipeline-app bot commented Dec 6, 2023 •

edited

Loading

openapi-pipeline-app bot commented Dec 6, 2023 •

edited

Loading

openapi-pipeline-app bot commented Dec 6, 2023 •

edited

Loading

Breaking Changes Tracking

openapi-pipeline-app bot commented Dec 6, 2023 •

edited

Loading

Azure OpenAI: 2023-12-01-preview TypeSpec fix for ChatMessageImageContentItem #26967

Azure OpenAI: 2023-12-01-preview TypeSpec fix for ChatMessageImageContentItem #26967

Conversation

trrwilson commented Dec 6, 2023

openapi-pipeline-app bot commented Dec 6, 2023 • edited Loading

Next Steps to Merge

openapi-pipeline-app bot commented Dec 6, 2023 • edited Loading

Swagger Validation Report

openapi-pipeline-app bot commented Dec 6, 2023 • edited Loading

Swagger Generation Artifacts

Breaking Changes Tracking

openapi-pipeline-app bot commented Dec 6, 2023 • edited Loading

Generated ApiView

openapi-pipeline-app bot commented Dec 6, 2023 •

edited

Loading

openapi-pipeline-app bot commented Dec 6, 2023 •

edited

Loading

openapi-pipeline-app bot commented Dec 6, 2023 •

edited

Loading

openapi-pipeline-app bot commented Dec 6, 2023 •

edited

Loading