Unmanaged LZ V2 client onboarding - minimum client-setup/core-landing-zone/client-landing-zone - with tier 3/4 workload customization

[obriensystems]

20231019 setup.sh script for kcc cluster delete/recreate and lz kpt apply/destroy in #446 (comment) under https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/gh446-hub/solutions/setup.sh

Onboard clean organization

• using obr*.indust*
• follow Day 0: non-managed client KCC onboarding (CC GKE cluster + (core-landing-zone + experimentation/core-landing-zone)) - dual bootstrap scripts #445

Start with base LZ (client-setup/core-landing-zone/client-landing-zone)

• https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit#quickstart

• with new publicly accessible endpoint change in https://github.com/ssc-spc-ccoe-cei/gcp-tools/commit/941d542e5024144b541136e19700b50cd8eaf895 for https://github.com/ssc-spc-ccoe-cei/gcp-tools/pull/53 in
  https://github.com/ssc-spc-ccoe-cei/gcp-tools/tree/main/scripts/bootstrap#readme

• https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/docs/landing-zone-v2#folder-structure-per-environment-dev-preprod-prod

[obriensystems]

Onboarding clean org

re-procedure from GoogleCloudPlatform/pbmm-on-gcp-onboarding#300
Combination of 3B1 and 1

Follow and edit/migrate to KCC https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/blob/main/docs/google-cloud-onboarding.md#onboarding-category-1-workspace-account---domain-hosted-on-google-domains

Follow and edit https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/blob/main/docs/google-cloud-onboarding.md#onboarding-category-3b1-3rd-party-email-account---3rd-party-aws-route53-domain-validation---reuse-existing-billing-account

Create 1b: Workspace, domain on 3rd party
https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/blob/main/docs/google-cloud-onboarding.md#onboarding-category-1b-workspace-account---domain-hosted-by-3rd-party

Requirements

• workspace business standard account (for emails without forwarding) - combination of 3
• Billing Account Administrator, Organization Administrator - single account for now

Use existing domain for new GCP account

Google Domains has moved to Squarespace - therefore use Route53 for now obr*.indust*

Start GCP

not https://workspace.google.com/signup/gcpidentity/welcome#0
use https://workspace.google.com/business/signup/welcome

Use incognito window

business account name

Add name + current email
Yes I have a domain

How you will sign in

default to business standard (will allow for Duet-AI)

billing for workspace first before gcp

new account (setup mx records later)

admin account

Activate gmail

set route53 dns

on route53

remove @

retry

2 min OK

skip new users for now

Start a new Chrome profile

launch cloud.google.com

accept credits

setup billing

setup org

prepay 50 on payment overview to enable billing quota increases

add cloud identity subscription in admin and disable automatic workspace user subscription association

accept free trial - enable full account

[obriensystems]

Setup base LZ V2 - kpt version - start with KCC cluster bootstrap

• https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit#quickstart
• with new publicly accessible endpoint change in https://github.com/ssc-spc-ccoe-cei/gcp-tools/commit/941d542e5024144b541136e19700b50cd8eaf895 for https://github.com/ssc-spc-ccoe-cei/gcp-tools/pull/53 in
  https://github.com/ssc-spc-ccoe-cei/gcp-tools/tree/main/scripts/bootstrap#readme
  then
  https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/docs/landing-zone-v2#folder-structure-per-environment-dev-preprod-prod

Create new bootstrap project (before running the script to create a kcc cluster project), assign super admin roles

Add all this to provisioning.sh

and add to https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/docs/landing-zone-v2/README.md#1-complete-the-bootstrap-procedure

# check IAM roles at the org level
export EMAIL=mich*...
export DOMAIN=obrien.industries

gcloud organizations list --format="get(name)" --filter=displayName=$DOMAIN
use later
ORG_ID=$(gcloud projects get-ancestors $BOOT_PROJECT_ID --format='get(id)' | tail -1)
export ORG_ID=459065....

so far only org admin

michael@cloudshell:~$ gcloud organizations get-iam-policy $ORG_ID --filter="bindings.members:$EMAIL" --flatten="bindings[].members" --format="table(bindings.role)"
ROLE: roles/resourcemanager.organizationAdmin

set additional (from another LZ)

root_@cloudshell:~ (kcc-kls)$ gcloud organizations get-iam-policy $ORG_ID --filter="bindings.members:ro...ems" --flatten="bindings[].members" --format="table(bindings.role)"
ROLE: roles/accesscontextmanager.policyAdmin
ROLE: roles/billing.admin
ROLE: roles/billing.projectManager
ROLE: roles/iam.securityAdmin
ROLE: roles/iam.serviceAccountTokenCreator
ROLE: roles/logging.admin
ROLE: roles/resourcemanager.folderAdmin
ROLE: roles/resourcemanager.organizationAdmin
ROLE: roles/resourcemanager.tagAdmin
ROLE: roles/securitycenter.admin
ROLE: roles/storage.admin

 gcloud organizations add-iam-policy-binding $ORG_ID  --member=user:$EMAIL --role=roles/iam.serviceAccountTokenCreator
 gcloud organizations add-iam-policy-binding $ORG_ID  --member=user:$EMAIL --role=roles/resourcemanager.folderAdmin
 gcloud organizations add-iam-policy-binding $ORG_ID  --member=user:$EMAIL --role=roles/orgpolicy.policyAdmin 
 gcloud organizations add-iam-policy-binding $ORG_ID  --member=user:$EMAIL --role=roles/resourcemanager.projectCreator
 gcloud organizations add-iam-policy-binding $ORG_ID  --member=user:$EMAIL --role=roles/billing.projectManager
 gcloud organizations add-iam-policy-binding $ORG_ID  --member=user:$EMAIL --role=roles/accesscontextmanager.policyAdmin
 gcloud organizations add-iam-policy-binding $ORG_ID  --member=user:$EMAIL --role=roles/billing.admin
 gcloud organizations add-iam-policy-binding $ORG_ID  --member=user:$EMAIL --role=roles/logging.admin
 gcloud organizations add-iam-policy-binding $ORG_ID  --member=user:$EMAIL --role=roles/resourcemanager.tagAdmin
 gcloud organizations add-iam-policy-binding $ORG_ID  --member=user:$EMAIL --role=roles/securitycenter.admin
 gcloud organizations add-iam-policy-binding $ORG_ID  --member=user:$EMAIL --role=roles/storage.admin

create bootstrap project

michael@cloudshell:~$ mkdir kcc-oi
michael@cloudshell:~$ cd kcc-oi
michael@cloudshell:~/kcc-oi$ mkdir github
michael@cloudshell:~/kcc-oi$ mkdir kpt
michael@cloudshell:~/kcc-oi$ cd github
michael@cloudshell:~/kcc-oi/github$ git clone https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git
michael@cloudshell:~/kcc-oi/github$ git clone https://github.com/ssc-spc-ccoe-cei/gcp-tools.git
michael@cloudshell:~/kcc-oi/github$ cd gcp-tools/scripts/bootstrap/
michael@cloudshell:~/kcc-oi/github/gcp-tools/scripts/bootstrap$ cp .env.sample kcc.env
michael@cloudshell:~/kcc-oi/github/gcp-tools/scripts/bootstrap$ 

create unique project name (your domain in reverse - under 30 char)
michael@cloudshell:~/kcc-oi/github/gcp-tools/scripts/bootstrap$ export PROJECT_ID=kcc-oi

michael@cloudshell:~/kcc-oi/github/gcp-tools/scripts/bootstrap$ gcloud projects create $PROJECT_ID --name="${PROJECT_ID}" --set-as-default
Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/kcc-oi].
Waiting for [operations/cp.7343501433229372585] to finish...done.                                                                                                                   
Enabling service [cloudapis.googleapis.com] on project [kcc-oi]...
Operation "operations/acat.p2-112532392452-da7d6cb0-3fcc-4a52-82da-a4bdc0943961" finished successfully.
Updated property [core/project] to [kcc-oi].

redundant
gcloud config set project "${PROJECT_ID}"

link billing id
if already set use
michael@cloudshell:~/kcc-oi/github/gcp-tools/scripts/bootstrap (kcc-oi)$ export BILLING_ID=$(gcloud alpha billing projects describe $PROJECT_ID '--format=value(billingAccountName)' | sed 's/.*\///')

other wise set it manually
gcloud beta billing projects link "$PROJECT_ID" --billing-account "$BILLING_ID"

create root folder for .env

michael@cloudshell:~/kcc-oi/github/gcp-tools/scripts/bootstrap (kcc-oi)$ export ROOT_FOLDER=kcc
michael@cloudshell:~/kcc-oi/github/gcp-tools/scripts/bootstrap (kcc-oi)$ gcloud resource-manager folders create --display-name=$ROOT_FOLDER --organization=$ORG_ID
Waiting for [operations/cf.5116850681862216947] to finish...done.                                                                                                                   
Created [<Folder
 createTime: '2023-09-27T16:41:23.820Z'
 displayName: 'kcc'
 lifecycleState: LifecycleStateValueValuesEnum(ACTIVE, 1)
 name: 'folders/96269513997'
 parent: 'organizations/459065442144'>].

edit env file

export CLUSTER=kcc-oi
export REGION=northamerica-northeast1
export PROJECT_ID=kcc-oi-cluster
export LZ_FOLDER_NAME=kcc-lz-20230927a
export NETWORK=kcc-oi-vpc
export SUBNET=kcc-oi-sn
export ORG_ID=45....144
export ROOT_FOLDER_ID=96..997
export BILLING_ID=0144...5
#export GIT_USERNAME=obriensystems
#export CONFIG_SYNC_REPO=<Repo for Config Sync> # tierX repo URL
#export CONFIG_SYNC_VERSION='HEAD'
#export CONFIG_SYNC_DIR=<Directory for config sync repo which syncs> # Should default to csync/deploy/<env>


run the bootstrap script - 1335

michael@cloudshell:~/kcc-oi/github/gcp-tools/scripts/bootstrap (kcc-oi)$ chmod 777 setup-kcc.sh 
michael@cloudshell:~/kcc-oi/github/gcp-tools/scripts/bootstrap (kcc-oi)$ ./setup-kcc.sh -af kcc.env

michael@cloudshell:~/kcc-oi/github/gcp-tools/scripts/bootstrap (kcc-oi)$ ./setup-kcc.sh -af kcc.env

##INFO - Update the logging for region

name: organizations/459065442144/settings
storageLocation: northamerica-northeast1

##INFO - create folder and project

Waiting for [operations/cf.6712241249467794709] to finish...working. 
Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/kcc-oi-cluster].
Waiting for [operations/cp.5994882289514299850] to finish...done.                                                                                                                   
Enabling service [cloudapis.googleapis.com] on project [kcc-oi-cluster]...
Operation "operations/acat.p2-116236192958-d0af6de3-66b6-43a4-a89b-1247b6c4e208" finished successfully.
Updated property [core/project] to [kcc-oi-cluster].

##INFO - Link billing account

billingAccountName: billingAccounts/014479-806359-2F5F85
billingEnabled: true
name: projects/kcc-oi-cluster/billingInfo
projectId: kcc-oi-cluster

##INFO - sleep 30s to allow for project creation before enabling services

Updated property [core/project].

##INFO - Enable services

Operation "operations/acf.p2-116236192958-a8990377-b9af-4caa-a9a0-426c502f5812" finished successfully.

##INFO - VPC

Created [https://www.googleapis.com/compute/v1/projects/kcc-oi-cluster/global/networks/kcc-oi-vpc].
NAME: kcc-oi-vpc
SUBNET_MODE: CUSTOM
BGP_ROUTING_MODE: REGIONAL
IPV4_RANGE: 
GATEWAY_IPV4: 

Instances on this network will not be reachable until firewall rules
are created. As an example, you can allow all internal traffic between
instances as well as SSH, RDP, and ICMP by running:

$ gcloud compute firewall-rules create <FIREWALL_NAME> --network kcc-oi-vpc --allow tcp,udp,icmp --source-ranges <IP_RANGE>
$ gcloud compute firewall-rules create <FIREWALL_NAME> --network kcc-oi-vpc --allow tcp:22,tcp:3389,icmp


##INFO - Subnet

Created [https://www.googleapis.com/compute/v1/projects/kcc-oi-cluster/regions/northamerica-northeast1/subnetworks/kcc-oi-sn].
NAME: kcc-oi-sn
REGION: northamerica-northeast1
NETWORK: kcc-oi-vpc
RANGE: 192.168.0.0/16
STACK_TYPE: IPV4_ONLY
IPV6_ACCESS_TYPE: 
INTERNAL_IPV6_PREFIX: 
EXTERNAL_IPV6_PREFIX: 

##INFO - Cloud router and Cloud NAT

Creating router [kcc-router]...done.                                                                                                                                                
NAME: kcc-router
REGION: northamerica-northeast1
NETWORK: kcc-oi-vpc
Creating NAT [kcc-router] in router [kcc-router]...done.                                                                                                                            

##INFO - enable logging for dns

Created Policy [https://dns.googleapis.com/dns/v1/projects/kcc-oi-cluster/policies/dnspolicy1].
{
  "description": "dns policy to enable logging",
  "enableInboundForwarding": false,
  "enableLogging": true,
  "id": "1183451581704200894",
  "kind": "dns#policy",
  "name": "dnspolicy1",
  "networks": [
    {
      "kind": "dns#policyNetwork",
      "networkUrl": "https://compute.googleapis.com/compute/v1/projects/kcc-oi-cluster/global/networks/kcc-oi-vpc"
    }
  ]
}

##INFO - private ip for apis

Created [https://www.googleapis.com/compute/v1/projects/kcc-oi-cluster/global/addresses/apis-private-ip].

##INFO - sleep 15s to allow for address to create


##INFO - private endpoint

Created [https://www.googleapis.com/compute/v1/projects/kcc-oi-cluster/global/forwardingRules/endpoint1].

##INFO - private dns zone for googleapis.com

Created [https://dns.googleapis.com/dns/v1/projects/kcc-oi-cluster/managedZones/googleapis].
NAME: googleapis.com.
TYPE: A
TTL: 300
DATA: 10.255.255.254
NAME: *.googleapis.com.
TYPE: CNAME
TTL: 300
DATA: googleapis.com.

##INFO - private dns zone for gcr.io

Created [https://dns.googleapis.com/dns/v1/projects/kcc-oi-cluster/managedZones/gcrio].
NAME: gcr.io.
TYPE: A
TTL: 300
DATA: 10.255.255.254
NAME: *.gcr.io.
TYPE: CNAME
TTL: 300
DATA: gcr.io.

##INFO - Allow egress to AZDO (optional)

Creating firewall...working..Cre
Creating firewall...done.                                                                                                                                                           
NAME: allow-egress-azure
NETWORK: kcc-oi-vpc
DIRECTION: EGRESS
PRIORITY: 5000
ALLOW: tcp:22,tcp:443
DENY: 
DISABLED: False

##INFO - Allow egress to Github (optional)

Creating firewall...working..Created [https://www.googleapis.com/compute/v1/projects/kcc-oi-cluster/global/firewalls/allow-egress-github].                                          
Creating firewall...done.                                                                                                                                                           
NAME: allow-egress-github
NETWORK: kcc-oi-vpc
DIRECTION: EGRESS
PRIORITY: 5001
ALLOW: tcp:22,tcp:443
DENY: 
DISABLED: False

##INFO - Allow egress to internal, peered vpc and secondary ranges

Creating firewall...working..Created [https://www.googleapis.com/compute/v1/projects/kcc-oi-cluster/global/firewalls/allow-egress-internal].                                        
Creating firewall...done.                                                                                                                                                           
NAME: allow-egress-internal
NETWORK: kcc-oi-vpc
DIRECTION: EGRESS
PRIORITY: 1000
ALLOW: all
DENY: 
DISABLED: False

##INFO - Deny egress to internet

Creating firewall...working.                        
Creating firewall...working..Created [https://www.googleapis.com/compute/v1/projects/kcc-oi-cluster/global/firewalls/deny-egress-internet].                                         
Creating firewall...done.                                                                                                                                                           
NAME: deny-egress-internet
NETWORK: kcc-oi-vpc
DIRECTION: EGRESS
PRIORITY: 65535
ALLOW: 
DENY: all
DISABLED: False

##INFO - sleep 30s to allow for policy to update


##INFO - Create Config controller

Create request issued for: [kcc-oi]
Waiting for operation [projects/kcc-oi-cluster/locations/northamerica-northeast1/operations/operation-1695836627727-6065abbcaf2b9-30345801-98711abe] to complete...working... 

1344 - expect under 15 min



Waiting for operation [projects/kcc-oi-cluster/locations/northamerica-northeast1/operations/operation-1695836627727-6065abbcaf2b9-30345801-98711abe] to complete...done.            
Created instance [kcc-oi].
Fetching cluster endpoint and auth data.
kubeconfig entry generated for krmapihost-kcc-oi.

##INFO - Config controller get credentials

Fetching cluster endpoint and auth data.
kubeconfig entry generated for krmapihost-kcc-oi.

##WARNING - configure-kcc-access.sh script should be run once connectivity to the cluster is established using bastion host / proxy.

increase billing quota before running the full LZ - before asking make sure you prepay $50 at least 24h before and mention this - also select only paid billing

via
https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/blob/main/docs/google-cloud-onboarding.md#billing-quota
and
https://support.google.com/code/contact/billing_quota_increase

[fmichaelobrien]

KCC cluster up via
https://github.com/ssc-spc-ccoe-cei/gcp-tools/blob/main/scripts/bootstrap/setup-kcc.sh

• proceeding to 3 package LZ deploy via kpt

TODO: check webhook visibility fix
This is new since my last cluster was up 6 weeks ago in #445

see #539 for spawned jira

This cluster has an admission webhook installed that is intercepting system critical requests in the last 24 hours. Intercepting these requests can impact availability of the GKE Control Plane. Learn more

https://cloud.google.com/kubernetes-engine/docs/how-to/optimize-webhooks?&_ga=2.215054544.-699491976.1695837480#unsafe-webhooks

gatekeeper-validating-webhook-configuration	Intercepting cluster-scoped system resources
gatekeeper-validating-webhook-configuration	Intercepting resources in the kube-node-lease namespace
gatekeeper-validating-webhook-configuration	Intercepting resources in the kube-system namespace

gatekeeper-validating-webhook-configuration Intercepting cluster-scoped system resources
gatekeeper-validating-webhook-configuration Intercepting resources in the kube-node-lease namespace
gatekeeper-validating-webhook-configuration Intercepting resources in the kube-system namespace

https://cloud.google.com/kubernetes-engine/docs/how-to/optimize-webhooks?&_ga=2.254246863.-699491976.1695837480#no-available-endpoints

workloads up

[obriensystems]

Raise core-landing-zone - then change one or add a new folder to start LZ customization
via
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/docs/landing-zone-v2#folder-structure-per-environment-dev-preprod-prod

2 - create your landing zone

michael@cloudshell:~$ gcloud config set project kcc-oi
Updated property [core/project].
michael@cloudshell:~ (kcc-oi)$ michael@cloudshell:~ (kcc-oi)$ cd kcc-oi/
michael@cloudshell:~/kcc-oi (kcc-oi)$ ls
github  kpt
michael@cloudshell:~/kcc-oi (kcc-oi)$ cd kpt/
michael@cloudshell:~/kcc-oi/kpt (kcc-oi)$ 

skip getting the latest package version - it needs to be fixed - just use main for now

michael@cloudshell:~/kcc-oi/kpt (kcc-oi)$ VERSION=main
michael@cloudshell:~/kcc-oi/kpt (kcc-oi)$ kpt pkg get https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git/${PACKAGE}@${VERSION}
Package "gatekeeper-policies":
Fetching https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit@main
From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
 * branch            main       -> FETCH_HEAD
 * [new branch]      main       -> origin/main
Adding package "solutions/gatekeeper-policies".

Fetched 1 package(s).

michael@cloudshell:~/kcc-oi/kpt (kcc-oi)$ PACKAGE="solutions/core-landing-zone"
michael@cloudshell:~/kcc-oi/kpt (kcc-oi)$ kpt pkg get https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git/${PACKAGE}@${VERSION}
Package "core-landing-zone":
Fetching https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit@main
From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
 * branch            main       -> FETCH_HEAD
Adding package "solutions/core-landing-zone".

Fetched 1 package(s).

michael@cloudshell:~/kcc-oi/kpt (kcc-oi)$ gcloud organizations list
DISPLAY_NAME: obrien.industries
ID: 45...44
DIRECTORY_CUSTOMER_ID: C03kdhrkc


edit setters.yaml


  org-id: "4590..144"
  lz-folder-id: '1080361268326'
  billing-id: "014...85"

  management-project-number: "116..58"
  # leave the kubernetes namespace as a default
  management-namespace: config-control
  #

  allowed-trusted-image-projects: |
    - "projects/cos-cloud"

  allowed-contact-domains: |
    - "@obri..g"
  #
  
  allowed-policy-domain-members: |
    - "C03kdhrkc"
  allowed-vpc-peering: |
    - "under:organizations/459..144"

  logging-project-id: logging-project-oi

  dns-project-id: dns-project-oi

kpt

michael@cloudshell:~/kcc-oi/kpt (kcc-oi)$ kpt live init core-landing-zone --namespace config-control
initializing "resourcegroup.yaml" data (namespace: config-control)...success
michael@cloudshell:~/kcc-oi/kpt (kcc-oi)$ kpt fn render core-landing-zone
Package "core-landing-zone": 
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.2"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.2" in 1.4s
  Results:
    [info] spec.folderRef.external: set field value to "1080361268326"
    [info] metadata.name: set field value to "security-log-bucket-oi"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi"
    [info] spec.projectRef.name: set field value to "logging-project-oi"
    ...(217 line(s) truncated, use '--truncate-output=false' to disable)

Triaging connection
older server working has 35 address in .kube/config

    server: https://35.203.120.71
  name: gke_kcc-kls-cluster3_northamerica-northeast1_krmapihost-kcc-kls3

newer server has private address

    server: https://172.16.0.130
  name: gke_kcc-oi-cluster_northamerica-northeast1_krmapihost-kcc-oi

Found above issue - forgot to add -p for public endpoint

ran./setup-kcc.sh -af kcc.env

for

https://github.com/ssc-spc-ccoe-cei/gcp-tools/commit/941d542e5024144b541136e19700b50cd8eaf895

[obriensystems]

Check the cluster - not working as usual

michael@cloudshell:~ (kcc-oi-cluster)$ gcloud anthos config controller get-credentials kcc-oi  --location northamerica-northeast1
Fetching cluster endpoint and auth data.
kubeconfig entry generated for krmapihost-kcc-oi.
michael@cloudshell:~ (kcc-oi-cluster)$ kubens config-control
E0928 19:20:41.674768     974 memcache.go:265] couldn't get current server API group list: Get "https://172.16.0.130/api?timeout=32s": dial tcp 172.16.0.130:443: i/o timeout
^C
michael@cloudshell:~ (kcc-oi-cluster)$ kubectl get pods --all-namespaces
E0928 19:21:26.018017    1006 memcache.go:265] couldn't get current server API group list: Get "https://172.16.0.130/api?timeout=32s": dial tcp 172.16.0.130:443: i/o timeout

kubectl is timing out - kpt won't deploy as usual as expectedmichael@cloudshell:~ (kcc-oi-cluster)$ kubectl get pods --all-namespaces
E0928 19:21:26.018017    1006 memcache.go:265] couldn't get current server API group list: Get "https://172.16.0.130/api?timeout=32s": dial tcp 172.16.0.130:443: i/o timeout

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-cluster)$ kpt live apply core-landing-zone --reconcile-timeout=2m --output=table
W0928 19:25:14.080087    1087 factory.go:66] Failed to query apiserver to check for flow control enablement: %vmaking /livez/ping request: context deadline exceeded

As a workaround I will use a standard cluster and retest - avoided standard as there was the odd timeout between 15-30 min

however my older kcc cluster on kcc.landing.systems even with webhook error is ok

root_@cloudshell:~ (kcc-kls-cluster3)$ kubectl get pods --all-namespaces
NAMESPACE                         NAME                                                       READY   STATUS    RESTARTS        AGE
cnrm-system                       cnrm-controller-manager-3fo6phebqgg23knqq5qq-0             1/1     Running   0               4d2h
cnrm-system                       cnrm-controller-manager-7c4rehlik7xgxc2utq6a-0             1/1     Running   0               4d1h

something off with my clean KCC env obrien.industries (will check the diffs on setup-kcc.sh) - as my older 6 week kcc.landing.systems - even with the admission errors will edit the yaml no problem

root_@cloudshell:~ (kcc-kls-cluster3)$ kubectl edit validatingwebhookconfiguration/gatekeeper-validating-webhook-configuration
Edit cancelled, no changes made.
root_@cloudshell:~ (kcc-kls-cluster3)$

rerun setup-kcc.sh with -p public ip option

see

export CLUSTER=kcc-oi2
export REGION=northamerica-northeast1
export PROJECT_ID=kcc-oi2-cluster
export LZ_FOLDER_NAME=kcc-lz-20230928b
export NETWORK=kcc-oi2-vpc
export SUBNET=kcc-oi2-sn

michael@cloudshell:~/kcc-oi/github/gcp-tools/scripts/bootstrap (kcc-oi)$ ./setup-kcc.sh -afp kcc.env

1644 - estimate 1700 kcc-oi2 cluster up
##INFO - Create Config controller

Create request issued for: [kcc-oi2]
Waiting for operation [projects/kcc-oi2-cluster/locations/northamerica-northeast1/operations/operation-1695933801715-606715bd057e8-f452780e-92d1cb2e] to complete...working..

fix

michael@cloudshell:~/kcc-oi/github/gcp-tools/scripts/bootstrap (kcc-oi)$ ./setup-kcc.sh -afp kcc.env
aiting for operation [projects/kcc-oi2-cluster/locations/northamerica-northeast1/operations/operation-1695933801715-606715bd057e8-f452780e-92d1cb2e] to complete...done.                                    
Created instance [kcc-oi2].
Fetching cluster endpoint and auth data.
kubeconfig entry generated for krmapihost-kcc-oi2.

##INFO - Config controller get credentials

Fetching cluster endpoint and auth data.
kubeconfig entry generated for krmapihost-kcc-oi2.

##WARNING - configure-kcc-access.sh script should be run once connectivity to the cluster is established using bastion host / proxy.
ichael@cloudshell:~/kcc-oi/github/gcp-tools/scripts/bootstrap (kcc-oi2-cluster)$ kubectl get nodes
NAME                                                STATUS   ROLES    AGE     VERSION
gk3-krmapihost-kcc-oi2-default-pool-6fc83c0e-ss20   Ready    <none>   9m12s   v1.27.3-gke.100
gk3-krmapihost-kcc-oi2-pool-1-28f0e374-tzw8         Ready    <none>   3m43s   v1.27.3-gke.100
gk3-krmapihost-kcc-oi2-pool-1-ae2f0850-4kmt         Ready    <none>   7m32s   v1.27.3-gke.100
gk3-krmapihost-kcc-oi2-pool-1-c9c2a582-9sdc         Ready    <none>   2m47s   v1.27.3-gke.100

cluster up with no admissions endpoint (has both public and private endpoints)