Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support location in bq operator #1767

Merged
merged 6 commits into from
Oct 17, 2022
Merged

Support location in bq operator #1767

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
93f8223
Support location in bq operator
tkawachi Aug 25, 2022
a71925c
Use guava's Optional as an argument
tkawachi Oct 2, 2022
3c01bce
Update BigQueryIT to test location
tkawachi Oct 2, 2022
67cbce9
Added load and extract tests
tkawachi Oct 8, 2022
3669aa6
expand wildcard import
tkawachi Oct 15, 2022
c1d8b86
Updated docs
tkawachi Oct 15, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion digdag-standards/src/main/java/io/digdag/standards/operator/gcp/BaseBqJobOperator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@ abstract class BaseBqJobOperator
protected TaskResult run(BqClient bq, String projectId)
{
BqJobRunner jobRunner = new BqJobRunner(request, bq, projectId);
Job completed = jobRunner.runJob(jobConfiguration(projectId));
String location = params.getOptional("location", String.class).orNull();
tkawachi marked this conversation as resolved.
Show resolved Hide resolved
Job completed = jobRunner.runJob(jobConfiguration(projectId), location);
return result(completed);
}

Expand Down
4 changes: 3 additions & 1 deletion digdag-standards/src/main/java/io/digdag/standards/operator/gcp/BqClient.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -170,11 +170,13 @@ void submitJob(String projectId, Job job)
.execute();
}

Job jobStatus(String projectId, String jobId)
Job jobStatus(String projectId, String jobId, String location)
throws IOException
{
return client.jobs()
.get(projectId, jobId)
// newer version of google-api-services-bigquery-v2 has setLocation()
.set("location", location)
tkawachi marked this conversation as resolved.
Show resolved Hide resolved
.execute();
}

Expand Down
4 changes: 2 additions & 2 deletions digdag-standards/src/main/java/io/digdag/standards/operator/gcp/BqJobRunner.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ class BqJobRunner
this.projectId = Objects.requireNonNull(projectId, "projectId");
}

Job runJob(JobConfiguration config)
Job runJob(JobConfiguration config, String location)
{
// Generate job id
Optional<String> jobId = state.params().getOptional(JOB_ID, String.class);
Expand Down Expand Up @@ -97,7 +97,7 @@ Job runJob(JobConfiguration config)
.withErrorMessage("BigQuery job status check failed: %s", canonicalJobId)
.run(s -> {
logger.info("Checking BigQuery job status: {}", canonicalJobId);
return bq.jobStatus(projectId, jobId.get());
return bq.jobStatus(projectId, jobId.get(), location);
});

// Done yet?
Expand Down