Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IPv6 clusters break because of netmask dependency in YAML #1520

Open
ZeidAqemia opened this issue Dec 6, 2024 · 3 comments
Open

IPv6 clusters break because of netmask dependency in YAML #1520

ZeidAqemia opened this issue Dec 6, 2024 · 3 comments
Labels
kind/bug Some behavior is incorrect or out of spec

Comments

@ZeidAqemia
Copy link

ZeidAqemia commented Dec 6, 2024 •
edited
Loading

What happened?

pulumi preview and pulumi up fail when creating IPv6 cluster in a IPv6 configured VPC in YAML.

I have attached my VPC and EKS stacks in the Context input field.

Thank you!

Example

❯ pulumi preview
[...]

error: eks:index:Cluster resource 'tests' has a problem: Invalid net address:
    error: Error: Invalid net address:
        at new Netmask (/snapshot/eks/node_modules/netmask/lib/netmask.js:150:15)
        at Netmask.contains (/snapshot/eks/node_modules/netmask/lib/netmask.js:166:14)
        at isPrivateCIDRBlock (/snapshot/eks/bin/nodegroup.js:923:22)
        at /snapshot/eks/bin/nodegroup.js:876:93
        at Array.find (<anonymous>)
        at /snapshot/eks/bin/nodegroup.js:876:63
        at Generator.next (<anonymous>)
        at fulfilled (/snapshot/eks/bin/nodegroup.js:18:58)
        at processTicksAndRejections (node:internal/process/task_queues:95:5)

Here are both of the stacks Pulumi.yaml files.

The config itself is quite small (omitting accounts and Roles)

config:
  aws:region: us-east-1
  aws-native:region: us-east-1
Pulumi.yaml for the VPC (CUElang generated) 

name: tests
description: Tests VPC
backend:
  url: s3://<REDACTED>
runtime: yaml
variables:
  prefix: tests
  subnetsRanges:
    fn::split:
      - "\n"
      - ${tests-ipv6CidrCommand.stdout}
outputs:
  vpcId: ${tests.id}
  vpcArn: ${tests.arn}
  natGatewayIds:
    - ${tests-nat-gateway-1.id}
    - ${tests-nat-gateway-2.id}
    - ${tests-nat-gateway-3.id}
  internetGatewayId: ${tests-internet-gateway.id}
  subnets: ${subnetsRanges}
  publicSubnetRanges:
    - ${subnetsRanges[0]}
    - ${subnetsRanges[1]}
    - ${subnetsRanges[2]}
  privateSubnetRanges:
    - ${subnetsRanges[3]}
    - ${subnetsRanges[4]}
    - ${subnetsRanges[5]}
  publicSubnetIds:
    - ${tests-public-1.id}
    - ${tests-public-2.id}
    - ${tests-public-3.id}
  privateSubnetIds:
    - ${tests-private-1.id}
    - ${tests-private-2.id}
    - ${tests-private-3.id}
  profile: |-
    [profile tests]
    source_profile = provision-aws
    role_arn = ${aws:assumeRole.roleArn}
resources:
  tests:
    type: aws:ec2:Vpc
    properties:
      tags:
        Name: tests
      cidrBlock: 10.0.0.0/16
      assignGeneratedIpv6CidrBlock: true
      enableDnsSupport: true
      enableDnsHostnames: true
  tests-ipv6CidrCommand:
    type: command:local:Command
    properties:
      create: cidr divide ${tests.ipv6CidrBlock} 256 | head -n 6
    options:
      parent: ${tests}
  tests-internet-gateway:
    type: aws:ec2:InternetGateway
    properties:
      tags:
        Name: tests-internet-gateway
      vpcId: ${tests.id}
    options:
      parent: ${tests}
  tests-public-1:
    type: aws:ec2:Subnet
    properties:
      tags:
        Name: tests-public-1
        kubernetes.io/cluster/tests: shared
      mapPublicIpOnLaunch: false
      assignIpv6AddressOnCreation: true
      cidrBlock: 10.0.1.0/24
      ipv6CidrBlock: ${subnetsRanges[0]}
      vpcId: ${tests.id}
      availabilityZone: us-east-1a
    options:
      parent: ${tests}
  tests-public-2:
    type: aws:ec2:Subnet
    properties:
      tags:
        Name: tests-public-2
        kubernetes.io/cluster/tests: shared
      mapPublicIpOnLaunch: false
      assignIpv6AddressOnCreation: true
      cidrBlock: 10.0.2.0/24
      ipv6CidrBlock: ${subnetsRanges[1]}
      vpcId: ${tests.id}
      availabilityZone: us-east-1b
    options:
      parent: ${tests}
  tests-public-3:
    type: aws:ec2:Subnet
    properties:
      tags:
        Name: tests-public-3
        kubernetes.io/cluster/tests: shared
      mapPublicIpOnLaunch: false
      assignIpv6AddressOnCreation: true
      cidrBlock: 10.0.3.0/24
      ipv6CidrBlock: ${subnetsRanges[2]}
      vpcId: ${tests.id}
      availabilityZone: us-east-1c
    options:
      parent: ${tests}
  tests-private-1:
    type: aws:ec2:Subnet
    properties:
      tags:
        Name: tests-private-1
        karpenter.sh/discovery: tests
        kubernetes.io/cluster/tests: shared
      mapPublicIpOnLaunch: false
      assignIpv6AddressOnCreation: true
      cidrBlock: 10.0.4.0/24
      ipv6CidrBlock: ${subnetsRanges[3]}
      vpcId: ${tests.id}
      availabilityZone: us-east-1a
    options:
      parent: ${tests}
  tests-private-2:
    type: aws:ec2:Subnet
    properties:
      tags:
        Name: tests-private-2
        karpenter.sh/discovery: tests
        kubernetes.io/cluster/tests: shared
      mapPublicIpOnLaunch: false
      assignIpv6AddressOnCreation: true
      cidrBlock: 10.0.5.0/24
      ipv6CidrBlock: ${subnetsRanges[4]}
      vpcId: ${tests.id}
      availabilityZone: us-east-1b
    options:
      parent: ${tests}
  tests-private-3:
    type: aws:ec2:Subnet
    properties:
      tags:
        Name: tests-private-3
        karpenter.sh/discovery: tests
        kubernetes.io/cluster/tests: shared
      mapPublicIpOnLaunch: false
      assignIpv6AddressOnCreation: true
      cidrBlock: 10.0.6.0/24
      ipv6CidrBlock: ${subnetsRanges[5]}
      vpcId: ${tests.id}
      availabilityZone: us-east-1c
    options:
      parent: ${tests}
  tests-nat-eip-1:
    type: aws:ec2:Eip
    properties:
      tags:
        Name: tests-nat-eip-1
      domain: vpc
    options:
      parent: ${tests}
  tests-nat-gateway-1:
    type: aws:ec2:NatGateway
    properties:
      tags:
        Name: tests-nat-gateway-1
      allocationId: ${tests-nat-eip-1.id}
      subnetId: ${tests-public-1.id}
    options:
      parent: ${tests}
  tests-natGateway-1-tagsCommand:
    type: command:local:Command
    properties:
      create: |
        aws ec2 create-tags --resources ${tests-nat-gateway-1.networkInterfaceId} --tags Key=Name,Value=${tests-nat-gateway-1.tags.Name} Key=project,Value=tests
      interpreter:
        - /bin/bash
        - -c
      environment:
        AWS_PROFILE: tests
        AWS_REGION: us-east-1
    options:
      parent: ${tests}
  tests-nat-eip-2:
    type: aws:ec2:Eip
    properties:
      tags:
        Name: tests-nat-eip-2
      domain: vpc
    options:
      parent: ${tests}
  tests-nat-gateway-2:
    type: aws:ec2:NatGateway
    properties:
      tags:
        Name: tests-nat-gateway-2
      allocationId: ${tests-nat-eip-2.id}
      subnetId: ${tests-public-2.id}
    options:
      parent: ${tests}
  tests-natGateway-2-tagsCommand:
    type: command:local:Command
    properties:
      create: |
        aws ec2 create-tags --resources ${tests-nat-gateway-2.networkInterfaceId} --tags Key=Name,Value=${tests-nat-gateway-2.tags.Name} Key=project,Value=tests
      interpreter:
        - /bin/bash
        - -c
      environment:
        AWS_PROFILE: tests
        AWS_REGION: us-east-1
    options:
      parent: ${tests}
  tests-nat-eip-3:
    type: aws:ec2:Eip
    properties:
      tags:
        Name: tests-nat-eip-3
      domain: vpc
    options:
      parent: ${tests}
  tests-nat-gateway-3:
    type: aws:ec2:NatGateway
    properties:
      tags:
        Name: tests-nat-gateway-3
      allocationId: ${tests-nat-eip-3.id}
      subnetId: ${tests-public-3.id}
    options:
      parent: ${tests}
  tests-natGateway-3-tagsCommand:
    type: command:local:Command
    properties:
      create: |
        aws ec2 create-tags --resources ${tests-nat-gateway-3.networkInterfaceId} --tags Key=Name,Value=${tests-nat-gateway-3.tags.Name} Key=project,Value=tests
      interpreter:
        - /bin/bash
        - -c
      environment:
        AWS_PROFILE: tests
        AWS_REGION: us-east-1
    options:
      parent: ${tests}
  tests-public-route-table-1:
    type: aws:ec2:RouteTable
    properties:
      tags:
        Name: tests-public-route-table-1
      vpcId: ${tests.id}
    options:
      parent: ${tests}
  tests-private-route-table-1:
    type: aws:ec2:RouteTable
    properties:
      tags:
        Name: tests-private-route-table-1
      vpcId: ${tests.id}
    options:
      parent: ${tests}
  tests-public-route-table-2:
    type: aws:ec2:RouteTable
    properties:
      tags:
        Name: tests-public-route-table-2
      vpcId: ${tests.id}
    options:
      parent: ${tests}
  tests-private-route-table-2:
    type: aws:ec2:RouteTable
    properties:
      tags:
        Name: tests-private-route-table-2
      vpcId: ${tests.id}
    options:
      parent: ${tests}
  tests-public-route-table-3:
    type: aws:ec2:RouteTable
    properties:
      tags:
        Name: tests-public-route-table-3
      vpcId: ${tests.id}
    options:
      parent: ${tests}
  tests-private-route-table-3:
    type: aws:ec2:RouteTable
    properties:
      tags:
        Name: tests-private-route-table-3
      vpcId: ${tests.id}
    options:
      parent: ${tests}
  tests-public-route-ipv4-1:
    type: aws:ec2:Route
    properties:
      routeTableId: ${tests-public-route-table-1.id}
      destinationCidrBlock: 0.0.0.0/0
      gatewayId: ${tests-internet-gateway.id}
    options:
      parent: ${tests-internet-gateway}
  tests-private-route-ipv4-1:
    type: aws:ec2:Route
    properties:
      routeTableId: ${tests-private-route-table-1.id}
      destinationCidrBlock: 0.0.0.0/0
      natGatewayId: ${tests-nat-gateway-1.id}
    options:
      parent: ${tests-nat-gateway-1}
  tests-public-route-ipv6-1:
    type: aws:ec2:Route
    properties:
      routeTableId: ${tests-public-route-table-1.id}
      destinationIpv6CidrBlock: ::/0
      gatewayId: ${tests-internet-gateway.id}
    options:
      parent: ${tests-internet-gateway}
  tests-public-route-ipv4-2:
    type: aws:ec2:Route
    properties:
      routeTableId: ${tests-public-route-table-2.id}
      destinationCidrBlock: 0.0.0.0/0
      gatewayId: ${tests-internet-gateway.id}
    options:
      parent: ${tests-internet-gateway}
  tests-private-route-ipv4-2:
    type: aws:ec2:Route
    properties:
      routeTableId: ${tests-private-route-table-2.id}
      destinationCidrBlock: 0.0.0.0/0
      natGatewayId: ${tests-nat-gateway-2.id}
    options:
      parent: ${tests-nat-gateway-2}
  tests-public-route-ipv6-2:
    type: aws:ec2:Route
    properties:
      routeTableId: ${tests-public-route-table-2.id}
      destinationIpv6CidrBlock: ::/0
      gatewayId: ${tests-internet-gateway.id}
    options:
      parent: ${tests-internet-gateway}
  tests-public-route-ipv4-3:
    type: aws:ec2:Route
    properties:
      routeTableId: ${tests-public-route-table-3.id}
      destinationCidrBlock: 0.0.0.0/0
      gatewayId: ${tests-internet-gateway.id}
    options:
      parent: ${tests-internet-gateway}
  tests-private-route-ipv4-3:
    type: aws:ec2:Route
    properties:
      routeTableId: ${tests-private-route-table-3.id}
      destinationCidrBlock: 0.0.0.0/0
      natGatewayId: ${tests-nat-gateway-3.id}
    options:
      parent: ${tests-nat-gateway-3}
  tests-public-route-ipv6-3:
    type: aws:ec2:Route
    properties:
      routeTableId: ${tests-public-route-table-3.id}
      destinationIpv6CidrBlock: ::/0
      gatewayId: ${tests-internet-gateway.id}
    options:
      parent: ${tests-internet-gateway}
  tests-private-route-ipv6-1:
    type: aws:ec2:Route
    properties:
      routeTableId: ${tests-private-route-table-1.id}
      destinationIpv6CidrBlock: "::/0"
      gatewayId: ${tests-internet-gateway.id}
    options:
      parent: ${tests-internet-gateway}
  tests-private-route-ipv6-2:
    type: aws:ec2:Route
    properties:
      routeTableId: ${tests-private-route-table-2.id}
      destinationIpv6CidrBlock: "::/0"
      gatewayId: ${tests-internet-gateway.id}
    options:
      parent: ${tests-internet-gateway}
  tests-private-route-ipv6-3:
    type: aws:ec2:Route
    properties:
      routeTableId: ${tests-private-route-table-3.id}
      destinationIpv6CidrBlock: "::/0"
      gatewayId: ${tests-internet-gateway.id}
    options:
      parent: ${tests-internet-gateway}
  tests-public-subnet-association-1:
    type: aws:ec2:RouteTableAssociation
    properties:
      subnetId: ${tests-public-1.id}
      routeTableId: ${tests-public-route-table-1.id}
    options:
      parent: ${tests-public-1}
  tests-public-subnet-association-2:
    type: aws:ec2:RouteTableAssociation
    properties:
      subnetId: ${tests-public-2.id}
      routeTableId: ${tests-public-route-table-2.id}
    options:
      parent: ${tests-public-2}
  tests-public-subnet-association-3:
    type: aws:ec2:RouteTableAssociation
    properties:
      subnetId: ${tests-public-3.id}
      routeTableId: ${tests-public-route-table-3.id}
    options:
      parent: ${tests-public-3}
  tests-private-subnet-association-1:
    type: aws:ec2:RouteTableAssociation
    properties:
      subnetId: ${tests-private-1.id}
      routeTableId: ${tests-private-route-table-1.id}
    options:
      parent: ${tests-private-1}
  tests-private-subnet-association-2:
    type: aws:ec2:RouteTableAssociation
    properties:
      subnetId: ${tests-private-2.id}
      routeTableId: ${tests-private-route-table-2.id}
    options:
      parent: ${tests-private-2}
  tests-private-subnet-association-3:
    type: aws:ec2:RouteTableAssociation
    properties:
      subnetId: ${tests-private-3.id}
      routeTableId: ${tests-private-route-table-3.id}
    options:
      parent: ${tests-private-3}

Pulumi.yaml for the EKS cluster (CUElang generated) 

name: tests
description: Tests
backend:
  url: s3://<REDACTED>
runtime: yaml
config:
  pulumi:tags:
    value:
      pulumi:template: aws-native-yaml
outputs:
  kubeconfig: ${cluster.kubeconfig}
  oidcIssuer: ${cluster.oidcIssuer}
  oidcProviderUrl: ${cluster.oidcProviderUrl}
  oidcProviderArn: ${cluster.oidcProviderArn}
  APIEndpoint: ${cluster.core.endpoint}
  clusterSecurityGroupId: ${cluster.clusterSecurityGroupId}
  nodeSecurityGroupId: ${cluster.nodeSecurityGroupId}
  core: ${cluster.core}
resources:
  vpc:
    type: pulumi:pulumi:StackReference
    properties:
      name: organization/tests/vpc
  cluster:
    type: eks:Cluster
    name: tests
    properties:
      authenticationMode: API
      defaultAddonsToRemove:
        - kube-proxy
        - coredns
        - vpc-cni
      clusterSecurityGroupTags:
        project: tests
      clusterTags:
        project: tests
      createOidcProvider: true
      desiredCapacity: 0
      enableConfigMapMutable: false
      enabledClusterLogTypes:
        - api
        - audit
        - authenticator
        - controllerManager
        - scheduler
      encryptionConfigKeyArn: ${secrets-encryption-key.arn}
      endpointPrivateAccess: true
      endpointPublicAccess: true
      fargate: true
      gpu: false
      instanceType: t2.medium
      ipFamily: ipv6
      maxSize: 0
      minSize: 0
      name: tests
      nodeAmiId: ami-08941e3bab61ce709
      nodeAssociatePublicIpAddress: false
      nodeRootVolumeEncrypted: true
      nodeRootVolumeSize: 100
      nodeSecurityGroupTags:
        karpenter.sh/discovery: tests
        project: tests
      privateSubnetIds: ${vpc.outputs["privateSubnetIds"]}
      providerCredentialOpts:
        profileName: tests
      publicSubnetIds: ${vpc.outputs["publicSubnetIds"]}
      skipDefaultNodeGroup: true
      tags:
        project: tests
      useDefaultVpcCni: true
      vpcId: ${vpc.outputs["vpcId"]}
  secrets-encryption-key:
    type: aws:kms:Key
    properties:
      description: KMS key for encrypting Kubernetes Secrets using envelope encryption
      deletionWindowInDays: 7
  secrets-encryption-keyAlias:
    type: aws:kms:Alias
    name: secrets-encryption-key
    properties:
      name: alias/k8s-secrets-encryption-key
      targetKeyId: ${secrets-encryption-key.keyId}

Output of pulumi about

CLI
Version      3.142.0
Go Version   go1.23.3
Go Compiler  gc

Plugins
KIND      NAME     VERSION
resource  aws      unknown
resource  command  unknown
resource  eks      unknown
language  yaml     unknown

Host
OS       darwin
Version  15.1.1
Arch     arm64

This project is written in yaml

Current Stack: organization/tests/eks

TYPE                                                 URN
pulumi:pulumi:Stack                                  urn:pulumi:eks::tests::pulumi:pulumi:Stack::tests-eks
pulumi:providers:pulumi                              urn:pulumi:eks::tests::pulumi:providers:pulumi::default
pulumi:providers:aws                                 urn:pulumi:eks::tests::pulumi:providers:aws::default
pulumi:pulumi:StackReference                         urn:pulumi:eks::tests::pulumi:pulumi:StackReference::vpc
aws:kms/key:Key                                      urn:pulumi:eks::tests::aws:kms/key:Key::secrets-encryption-key
aws:kms/alias:Alias                                  urn:pulumi:eks::tests::aws:kms/alias:Alias::secrets-encryption-key
pulumi:providers:eks                                 urn:pulumi:eks::tests::pulumi:providers:eks::default
pulumi:providers:aws                                 urn:pulumi:eks::tests::pulumi:providers:aws::default_6_47_0
eks:index:Cluster                                    urn:pulumi:eks::tests::eks:index:Cluster::tests
eks:index:ServiceRole                                urn:pulumi:eks::tests::eks:index:Cluster$eks:index:ServiceRole::tests-eksRole
eks:index:ServiceRole                                urn:pulumi:eks::tests::eks:index:Cluster$eks:index:ServiceRole::tests-podExecutionRole
aws:ec2/securityGroup:SecurityGroup                  urn:pulumi:eks::tests::eks:index:Cluster$aws:ec2/securityGroup:SecurityGroup::tests-eksClusterSecurityGroup
aws:iam/role:Role                                    urn:pulumi:eks::tests::eks:index:Cluster$eks:index:ServiceRole$aws:iam/role:Role::tests-eksRole-role
aws:iam/role:Role                                    urn:pulumi:eks::tests::eks:index:Cluster$eks:index:ServiceRole$aws:iam/role:Role::tests-podExecutionRole-role
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:eks::tests::eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::tests-eksClusterInternetEgressRule
aws:iam/rolePolicyAttachment:RolePolicyAttachment    urn:pulumi:eks::tests::eks:index:Cluster$eks:index:ServiceRole$aws:iam/rolePolicyAttachment:RolePolicyAttachment::tests-podExecutionRole-6ad441d9
aws:iam/rolePolicyAttachment:RolePolicyAttachment    urn:pulumi:eks::tests::eks:index:Cluster$eks:index:ServiceRole$aws:iam/rolePolicyAttachment:RolePolicyAttachment::tests-eksRole-4b490823
aws:eks/cluster:Cluster                              urn:pulumi:eks::tests::eks:index:Cluster$aws:eks/cluster:Cluster::tests-eksCluster
aws:ec2/securityGroup:SecurityGroup                  urn:pulumi:eks::tests::eks:index:Cluster$aws:ec2/securityGroup:SecurityGroup::tests-nodeSecurityGroup
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:eks::tests::eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::tests-eksExtApiServerClusterIngressRule
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:eks::tests::eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::tests-eksNodeInternetEgressRule
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:eks::tests::eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::tests-eksClusterIngressRule
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:eks::tests::eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::tests-eksNodeClusterIngressRule
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:eks::tests::eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::tests-eksNodeIngressRule
aws:iam/openIdConnectProvider:OpenIdConnectProvider  urn:pulumi:eks::tests::eks:index:Cluster$aws:iam/openIdConnectProvider:OpenIdConnectProvider::tests-oidcProvider
pulumi:providers:kubernetes                          urn:pulumi:eks::tests::eks:index:Cluster$pulumi:providers:kubernetes::tests-eks-k8s
aws:eks/addon:Addon                                  urn:pulumi:eks::tests::eks:index:Cluster$aws:eks/addon:Addon::tests-kube-proxy
aws:eks/fargateProfile:FargateProfile                urn:pulumi:eks::tests::eks:index:Cluster$aws:eks/fargateProfile:FargateProfile::tests-fargateProfile
aws:eks/addon:Addon                                  urn:pulumi:eks::tests::eks:index:Cluster$aws:eks/addon:Addon::tests-coredns
pulumi:providers:command                             urn:pulumi:eks::tests::pulumi:providers:command::default
command:local:Command                                urn:pulumi:eks::tests::command:local:Command::cleanupAWSResources
command:local:Command                                urn:pulumi:eks::tests::command:local:Command::queryServiceIpv6Cidr
command:local:Command                                urn:pulumi:eks::tests::command:local:Command::queryCoreDNSIP
aws:eks/accessEntry:AccessEntry                      urn:pulumi:eks::tests::aws:eks/accessEntry:AccessEntry::NodeRole
aws:eks/accessEntry:AccessEntry                      urn:pulumi:eks::tests::aws:eks/accessEntry:AccessEntry::KarpenterNodeRole


Found no pending operations associated with eks

Backend
Name           pc-36.home
URL            s3://pulumi-state-a194e259/tests?awssdk=v2&region=us-east-1&profile=pulumi
User           zeidmarouf
Organizations
Token type     personal

Dependencies:
NAME     VERSION
aws
command
eks
pulumi

Additional context

From https://pulumi-community.slack.com/archives/CRH5ENVDX/p1733409647746379

Trying to create an EKS cluster and I get a weird error when adding InternetGateway routes to private subnet route tables for IPv6.

The code related to this cf

pulumi-eks/nodejs/eks/nodegroup.ts

Lines 1632 to 1634 in 9c128c5

const hasInternetGatewayRoute =
routeTable.routes.find((r) => !!r.gatewayId && !isPrivateCIDRBlock(r.cidrBlock)) !==
undefined;

The function definition is here

pulumi-eks/nodejs/eks/nodegroup.ts

Lines 1671 to 1683 in 9c128c5

/**
* Returns true if the given CIDR block falls within a private range [1].
* [1] https://en.wikipedia.org/wiki/Private_network
*/
function isPrivateCIDRBlock(cidrBlock: string): boolean {
const privateA = new netmask.Netmask("10.0.0.0/8");
const privateB = new netmask.Netmask("172.16.0.0/12");
const privateC = new netmask.Netmask("192.168.0.0/16");
return (
privateA.contains(cidrBlock) || privateB.contains(cidrBlock) || privateC.contains(cidrBlock)
);
}

As you can see, the netmasks are hardcoded for IPv4 private ranges.
The netmask package only mentions IPv4 support cf its description:

The Netmask class parses and understands IPv4 CIDR blocks so they can be explored and compared.

Image

So hasInternetGatewayRoute errors because the route:

  • mentions an Internet Gateway (lefthand expression of the check)
  • has an IPv6 which breaks the netmask contains function call as it only supports IPv4

On Slack, we think we've managed to go all the way back to the root cause -> https://github.com/pulumi/pulumi-eks/blob/master/nodejs/eks/nodegroup.ts#L653

core is an instance of aws.eks.Cluster rather than an instance of CoreDataArgs

So there's no aws.eks.Cluster.privateSubnetIds

What's weird for me is that this happens in the default nodegroup creation flow but I have "skipDefaultNodeGroup": true and "fargate": true in my Pulumi.yaml as you can see above.

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).
@ZeidAqemia ZeidAqemia added kind/bug Some behavior is incorrect or out of spec needs-triage Needs attention from the triage team labels Dec 6, 2024
@ionicsolutions
Copy link

ionicsolutions commented Dec 7, 2024 •
edited
Loading

Since fargate is set to true, computeWorkerSubnets and in turn the isPrivateCIDRBlock helper function is invoked when the aws.eks.FargateProfile is created:

pulumi-eks/nodejs/eks/cluster.ts

Lines 1038 to 1043 in 9c128c5

subnetIds: pulumi.output(clusterSubnetIds).apply((subnets) => {
if (fargate.subnetIds?.length && fargate.subnetIds.length > 0) {
return computeWorkerSubnets(parent, fargate.subnetIds);
} else {
return computeWorkerSubnets(parent, subnets);
}

Since the function is incompatible with IPv6, this will always fail when there are IPv6 routes to an internet gateway in a subnet's route table.

@ZeidAqemia
Copy link
Author

ZeidAqemia commented Dec 7, 2024 •
edited
Loading

I unblocked myself by disabling the fargate property and creating the policy, role and profile manually.
I also switched to using Egress-only internet gateways in the private subnets, which makes more sense from a network design perspective.

But the problem technically still persists.
The logic probably needs to be reworked and it might be worth switching netmask for ip-toolkit as it supports IPv6.

@corymhall
Copy link
Contributor

Thanks for raising this issue with us, it sounds like we do not correctly support ipv6. This sounds like it will be a bigger lift to add support for ipv6 so we will most likely not be able to pick this up soon. Upvotes will definitely help us prioritize this.

@corymhall corymhall removed the needs-triage Needs attention from the triage team label Dec 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Some behavior is incorrect or out of spec
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ionicsolutions @corymhall @ZeidAqemia