AWS Glue job minimum number of workers should be 1

[baolsen]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v1.1.6
on darwin_arm64

• provider registry.terraform.io/hashicorp/aws v3.49.0
• provider registry.terraform.io/hashicorp/http v2.1.0
• provider registry.terraform.io/hashicorp/null v3.1.0
• provider registry.terraform.io/hashicorp/time v0.7.2

---

Affected Resource(s)

aws_glue_job

Terraform Configuration Files

resource "aws_glue_job" "job" {

  name     = "test"
  role_arn = aws_iam_role.some_role.arn

  glue_version      = "2.0"
  number_of_workers = 1  # Causes error below.
  worker_type       = "Standard"

  timeout = 5

  command {
    script_location = "s3://some_script.py"
  }
}

Debug Output

Panic Output

Expected Behavior

Should be allowed to create Glue job with 1 worker as it is allowed in the AWS Console.
At least, I could edit an existing job that had 2 workers down to 1 worker using the console. The change was applied successfully in the console, verified by refreshing it.

Actual Behavior

│ Error: expected number_of_workers to be at least (2), got 1
│
│ with aws_glue_job.lakefs,
│ on glue.tf line 16, in resource "aws_glue_job" "job":
│ 16: number_of_workers = 1

Steps to Reproduce

Create Glue job above with number of workers = 1 instead of 2 (default).

Important Factoids

References

[baolsen]

Can be assigned to me :)

[bschaatsbergen]

How's this going @baolsen? :)

[baolsen]

I haven't started on this, not sure when I'd get a chance. Anyone is welcome to grab it :)

[bschaatsbergen]

Ok, I'll pick it up 👍

[baolsen]

I've done some more digging to get to the bottom of this this.

It appears to me that when using Worker types G1.X and G2.X the minimum number of workers must indeed be at least 2.

When the worker type is "Standard" then in the AWS Console it is possible to change the number of workers from 2 to 1, after the job was created by Terraform:

This job can then be run successfully (see below).

After changing the number of workers from 2 to 1 on a "Standard" worker type job, if you then try to change the worker type to G1.X or G2.X the following exceptions appear on the AWS Console:

Failed to update job
[gluestudio-service.eu-west-1.amazonaws.com] updateJob: InvalidInputException: To use Worker Type G.1X, minimum allowed value of Number of Workers is 2.

Failed to update job
[gluestudio-service.eu-west-1.amazonaws.com] updateJob: InvalidInputException: To use Worker Type G.2X, minimum allowed value of Number of Workers is 2.

So it appears that number of workers is allowed to be 1 ONLY IF the Standard worker type is used.

Currently the Terraform AWS provider has a simple validation done, which just insists that the minimum value is 2 regardless of the worker type. To close this issue, we could possibly change the provider to allow a minimum value of 1 if the worker type is Standard, but if it is G1.X or G2.X then 2 is the minimum.

Here is some evidence that a Standard worker type does currently run with only 1 worker (or at least, it appears that this works):

[baolsen]

I'm no Go expert but from what I can see in the provider code, ValidateFunc doesn't seem to have a way for one parameter's validation to depend on the value of another.

Maybe someone can point me in the right direction, if this is possible.

Something along these lines in internal/service/glue/job.go and internal/service/glue/dev_endpoint.go

	"number_of_workers": {
		Type:          schema.TypeInt,
		Optional:      true,
		ForceNew:      true,
		ValidateFunc:  validation.Any(
			validation.IntAtLeast(2),
			validation.All(
				**Somehow validate that worker type == Standard, then number_of_workers=1 is allowed**
				validation.IntAtLeast(1)
			)
		)
		ConflictsWith: []string{"number_of_nodes"},
	},

If it is not possible to validate one parameter based on the value of another, then I guess this issue can be closed as a won't do.

[kiki-ki]

I faced the same issue.

Are there few cases like this where you want to apply validation that depends on other Schemas?

If there are other cases like this one, we thought it would be necessary to implement a cross-parameter verification at the Resource layer, one level above.

ex.)

in: hashicorp/terraform-plugin-sdk/v2@v2.17.0/helper/schema/resource.go

type Resource struct {
	SchemaValidateFuncs []func(map[string]*Schema, string) ([]string, error) // add
	...
}

in: internal/service/glue/job.go

func ResourceJob() *schema.Resource {
	return &schema.Resource{
		...
		SchemaValidateFuncs: []func(map[string]*schema.Schema, string) ([]string, error){
			func(s map[string]*schema.Schema, k string) ([]string, error) {
				// Extract the value of s["worker_type"] and verify s["number_of_worker"] by that value.
				return warnings, errors
			},
		},
                ...
}

In addition, it is necessary to process the common use of SchemaValidateFunc as well as ValidateFunc.

I haven't figured out all the code yet, so sorry if I'm being reckless.

[rafaljanicki]

Hey guys, any news over here? It's quite an old bug and is still there

[bschaatsbergen]

Hey @rafaljanicki, thanks for reminding us. I've raised a pull request to change the validation for the number_of_workers to a minimum of 1.

[github-actions]

This functionality has been released in v5.42.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.