Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Internal] Update Jobs GetJob API to support paginated responses #1133

Merged
merged 4 commits into from
Feb 13, 2025
Merged

[Internal] Update Jobs GetJob API to support paginated responses #1133

merged 4 commits into from
Feb 13, 2025

Conversation

gkiko10
Copy link
Contributor

@gkiko10 gkiko10 commented Jan 21, 2025

What changes are proposed in this pull request?

Introduces logic in extension for jobs GetJob call that paginates tasks and other arrays in the response. This change is necessary for SDK and API 2.2 compatibility. API 2.2 serves paginated responses as long as next_page_token field is present in the response. The pagination logic is not exposed to the customer.

How is this tested?

I enabled API 2.2 calls by modifying URL string /api/2.2/jobs/get in service/jobs/impl.go. Then I ran unit test from service/jobs/ext_api_test.go

@gkiko10 gkiko10 temporarily deployed to test-trigger-is January 21, 2025 14:14 — with GitHub Actions Inactive
@gkiko10 gkiko10 temporarily deployed to test-trigger-is January 21, 2025 14:21 — with GitHub Actions Inactive
@gkiko10 gkiko10 temporarily deployed to test-trigger-is January 24, 2025 09:45 — with GitHub Actions Inactive
renaudhartert-db
renaudhartert-db reviewed Feb 3, 2025
View reviewed changes
service/jobs/ext_api.go Outdated
Comment on lines 44 to 56
for job.NextPageToken != "" {
request.PageToken = job.NextPageToken
nextJob, err := a.jobsImpl.Get(ctx, request)
if err != nil {
return nil, err
}

job.Settings.Tasks = append(job.Settings.Tasks, nextJob.Settings.Tasks...)
job.Settings.JobClusters = append(job.Settings.JobClusters, nextJob.Settings.JobClusters...)
job.Settings.Parameters = append(job.Settings.Parameters, nextJob.Settings.Parameters...)
job.Settings.Environments = append(job.Settings.Environments, nextJob.Settings.Environments...)
job.NextPageToken = nextJob.NextPageToken
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add a comment to explain to a reader with limited context why we are doing this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 6, 2025

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/sdk-go

Inputs:

  • PR number: 1133
  • Commit SHA: 61ce55b7efa71c6386e9dad3b5e4bccb19395ba5

Checks will be approved automatically on success.

@gkiko10 gkiko10 temporarily deployed to test-trigger-is February 6, 2025 18:13 — with GitHub Actions Inactive
@hectorcast-db hectorcast-db added this pull request to the merge queue Feb 13, 2025
Merged via the queue into databricks:main with commit 5a9ddae Feb 13, 2025
15 checks passed
deco-sdk-tagging bot added a commit that referenced this pull request Mar 3, 2025
@deco-sdk-tagging
[Release] Release v0.59.0
728c6e7 
## Release v0.59.0

### Bug Fixes

* Fix unlikely issue due to conflicting error details in `APIError`.

### Internal Changes

* Update Jobs ListRuns API to support paginated responses ([#1151](#1151))
* Add `poll.SimpleError` to mock waiter objects returning errors  ([#1155](#1155))
* Refactor `APIError` to expose different types of error details ([#1153](#1153)). 
* Update Jobs ListJobs API to support paginated responses ([#1150](#1150))
* Introduce automated tagging ([#1148](#1148)).
* Update Jobs GetJob API to support paginated responses  ([#1133](#1133)).
* Update Jobs GetRun API to support paginated responses  ([#1132](#1132)).

### API Changes
* Added `GetSpace` method for [w.Genie](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/dashboards#GenieAPI) workspace-level service.
* Added `ListProviderShareAssets` method for [w.Providers](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/sharing#ProvidersAPI) workspace-level service.
* Added `BudgetPolicyId` and `EffectiveBudgetPolicyId` fields for [apps.App](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/apps#App).
* Added `Policy` field for [billing.CreateBudgetPolicyRequest](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/billing#CreateBudgetPolicyRequest).
* Added `DatabricksGcpServiceAccount` field for [catalog.ValidateCredentialRequest](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#ValidateCredentialRequest).
* Added `AttachmentId` field for [dashboards.GenieAttachment](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/dashboards#GenieAttachment).
* Added `ConversationId` field for [dashboards.GenieConversation](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/dashboards#GenieConversation).
* Added `MessageId` field for [dashboards.GenieMessage](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/dashboards#GenieMessage).
* Added `Description`, `Id`, `LastUpdatedTimestamp`, `Query`, `QueryResultMetadata` and `Title` fields for [dashboards.GenieQueryAttachment](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/dashboards#GenieQueryAttachment).
* Added `GenAiComputeTask` field for [jobs.RunTask](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#RunTask).
* Added `GenAiComputeTask` field for [jobs.SubmitTask](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#SubmitTask).
* Added `GenAiComputeTask` field for [jobs.Task](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#Task).
* Added `RunName` field for [ml.CreateRun](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/ml#CreateRun).
* Added `RunName` field for [ml.RunInfo](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/ml#RunInfo).
* Added `RunName` field for [ml.UpdateRun](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/ml#UpdateRun).
* Added `Lifetime` field for [oauth2.CreateServicePrincipalSecretRequest](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/oauth2#CreateServicePrincipalSecretRequest).
* Added `ExpireTime` field for [oauth2.CreateServicePrincipalSecretResponse](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/oauth2#CreateServicePrincipalSecretResponse).
* Added `ExpireTime` field for [oauth2.SecretInfo](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/oauth2#SecretInfo).
* Added `InstanceProfileArn` field for [serving.AmazonBedrockConfig](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/serving#AmazonBedrockConfig).
* Added `Add`, `Principal` and `Remove` fields for [sharing.PermissionsChange](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/sharing#PermissionsChange).
* Added `ColumnsToRerank` field for [vectorsearch.QueryVectorIndexRequest](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/vectorsearch#QueryVectorIndexRequest).
* Added `Oracle` and `Teradata` enum values for [catalog.ConnectionType](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#ConnectionType).
* Added `FunctionArgumentsInvalidTypeException` and `MessageCancelledWhileExecutingException` enum values for [dashboards.MessageErrorType](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/dashboards#MessageErrorType).
* Added `Waiting` enum value for [jobs.RunLifecycleStateV2State](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#RunLifecycleStateV2State).
* Added `ActiveOnly`, `All` and `DeletedOnly` enum values for [ml.ViewType](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/ml#ViewType).
* Added `OauthClientCredentials` enum value for [sharing.AuthenticationType](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/sharing#AuthenticationType).
* Added `Raw` enum value for [workspace.ExportFormat](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#ExportFormat).
* [Breaking] Changed `GetByName` method for [w.Experiments](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/ml#ExperimentsAPI) workspace-level service to return [ml.GetExperimentByNameResponse](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/ml#GetExperimentByNameResponse).
* [Breaking] Changed `LogInputs` method for [w.Experiments](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/ml#ExperimentsAPI) workspace-level service with new required argument order.
* [Breaking] Changed `SharePermissions` and `UpdatePermissions` methods for [w.Shares](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/sharing#SharesAPI) workspace-level service return type to become non-empty.
* [Breaking] Changed `SharePermissions` method for [w.Shares](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/sharing#SharesAPI) workspace-level service to return [sharing.GetSharePermissionsResponse](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/sharing#GetSharePermissionsResponse).
* [Breaking] Changed `UpdatePermissions` method for [w.Shares](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/sharing#SharesAPI) workspace-level service to return [sharing.UpdateSharePermissionsResponse](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/sharing#UpdateSharePermissionsResponse).
* Changed `PolicyId` field for [billing.BudgetPolicy](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/billing#BudgetPolicy) to no longer be required.
* [Breaking] Changed `PolicyId` field for [billing.BudgetPolicy](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/billing#BudgetPolicy) to no longer be required.
* [Breaking] Changed `Partitions` field for [cleanrooms.CleanRoomAssetTableLocalDetails](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/cleanrooms#CleanRoomAssetTableLocalDetails) to type [cleanrooms.PartitionList](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/cleanrooms#PartitionList).
* [Breaking] Changed `Query` field for [dashboards.GenieAttachment](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/dashboards#GenieAttachment) to type [dashboards.GenieQueryAttachment](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/dashboards#GenieQueryAttachment).
* Changed `Digest`, `Name`, `Source` and `SourceType` fields for [ml.Dataset](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/ml#Dataset) to be required.
* [Breaking] Changed `Digest`, `Name`, `Source` and `SourceType` fields for [ml.Dataset](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/ml#Dataset) to be required.
* [Breaking] Changed `Dataset` field for [ml.DatasetInput](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/ml#DatasetInput) to be required.
* Changed `Dataset` field for [ml.DatasetInput](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/ml#DatasetInput) to be required.
* Changed `Key` and `Value` fields for [ml.InputTag](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/ml#InputTag) to be required.
* [Breaking] Changed `Key` and `Value` fields for [ml.InputTag](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/ml#InputTag) to be required.
* [Breaking] Changed `ViewType` field for [ml.ListExperimentsRequest](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/ml#ListExperimentsRequest) to type [ml.ViewType](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/ml#ViewType).
* [Breaking] Changed `RunId` field for [ml.LogInputs](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/ml#LogInputs) to be required.
* [Breaking] Changed `ViewType` field for [ml.SearchExperiments](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/ml#SearchExperiments) to type [ml.ViewType](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/ml#ViewType).
* [Breaking] Changed `RunViewType` field for [ml.SearchRuns](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/ml#SearchRuns) to type [ml.ViewType](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/ml#ViewType).
* [Breaking] Removed `CustomTags` and `PolicyName` fields for [billing.CreateBudgetPolicyRequest](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/billing#CreateBudgetPolicyRequest).
* [Breaking] Removed `CachedQuerySchema`, `Description`, `Id`, `InstructionId`, `InstructionTitle`, `LastUpdatedTimestamp`, `Query`, `StatementId` and `Title` fields for [dashboards.QueryAttachment](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/dashboards#QueryAttachment).
* [Breaking] Removed `MaxResults` and `PageToken` fields for [sharing.UpdateSharePermissions](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/sharing#UpdateSharePermissions).
* [Breaking] Removed `ActiveOnly`, `All` and `DeletedOnly` enum values for [ml.SearchExperimentsViewType](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/ml#SearchExperimentsViewType).
* [Breaking] Removed `ActiveOnly`, `All` and `DeletedOnly` enum values for [ml.SearchRunsRunViewType](https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/ml#SearchRunsRunViewType).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hectorcast-db hectorcast-db hectorcast-db approved these changes

@renaudhartert-db renaudhartert-db Awaiting requested review from renaudhartert-db

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@gkiko10 @hectorcast-db @renaudhartert-db