CloudBuild GKE Autopilot Private Cluster on New VPC

Note

This post translated By AI and the original content was written in Traditional Chinese.

This method is to create a brand new VPC, dedicated for Cloud Build Private Pool usage, aiming to provide a fixed external IP for Cloud Build Private Pool without interfering with the existing VPC.

Scenario

• Ensure all network traffic is internal, but want the convenience of Autopilot GKE, so Private Cluster is needed
• If you want to connect to CloudSQL with a private IP without using CloudSQL Auth Proxy, then Autopilot GKE needs to use Private Cluster
• The red box in the diagram below is the scope of this implementation. GKE uses a private cluster, CloudBuild below uses a private pool with an independent VPC, while CloudBuild above without a red box uses a public pool for CI. Details can be found in CloudBuild triggered by Pub/Sub

Process and Explanation

1. Create a VPC network and enable Subnet PGA and NAT Gateway, so the pods in GKE will use the NAT Gateway for external networking
2. Create a GKE autopilot Private Cluster

   Tip

   • When creating GKE, choose Public endpoint access enabled, authorized networks enabled to support both public IP with authorized network and private IP, adding flexibility
   • If the organization’s policy completely disallows external connections, do not enable public access
3. (Optional) Use GCE as a bastion host, the service account must have Kubernetes Engine Developer permissions to get the GKE credentials
4. Create Artifact Registry so GKE Private Cluster can directly connect through PGA
5. Create a VPC network (red box in the diagram above) dedicated to CloudBuild and the necessary routes

• Create Cloud Build and use a Private Pool to ensure data flow remains within the private network with a fixed IP, thus an independent VM as a bastion from CloudBuild to GKE is needed
  • The process becomes CloudBuild to internal IP -> VM (static internal IP & static public IP) -> GKE(public IP)
  • When setting up the private pool for CloudBuild, do not check the option for external IP to ensure traffic stays within the VPC, but this means it cannot access the internet. For installing packages, it is recommended to separate private pool execution

6. Reserve the following two network segments to be unused by any services:

• CloudBuild reserves 192.168.10.0/24
• Docker bridge reserves 172.17.0.0/16 (GCP must avoid using this segment)

Execution Steps

Create GKE VPC

This VPC is to provide external access for GKE Private cluster and is used for running internal services, not for Cloud Build Private Pool external access.

Enter the network name and choose Custom in Subnets

1. Enter a name
2. Enter a custom subnet
3. Enable PGA
4. Done

Click Create to create the VPC

Set up IAP Firewall

Ensure IAP can pass through the firewall by creating a rule to allow 35.235.240.0/20.

Click Create

Set up NAT Gateway

Create a NAT Gateway so all traffic can go out through a specific IP.

1. Enter the gateway name
2. Select VPC
3. Choose the region
4. Create a router

Enter the router name and click Create

Finally, create Cloud NAT

Create Artifact Registry

Click + Create

Enter the name, choose Docker format, and select the region to create.

Create GKE autopilot Private Cluster

Create GKE

1. Enter the name
2. Select the region
3. Set up the network

1. Select the network and subnet
2. Choose private cluster
3. Check to enable external connections
4. Check to allow different regions to connect, and to lock specific external IPs to connect to the control plane
5. Create

(Optional) Create Bastion VM

If you want the control plane to have no external access, you can use a VM in the same subnet as a bastion, or use a VPN in a production environment to connect from on-premise to the private cluster control plane.

Create Service Account

To allow the bastion VM to obtain and operate GKE, create a service account and provide permissions.

Enter the name, click Create and continue.

Grant the following permissions to complete:

• Kubernetes Engine Developer
• Logs Writer
• Monitoring Metric Writer
• Monitoring Viewer
• Stackdriver Resource Metadata Writer

(Optional) Provide Artifact Registry Permissions to Bastion VM service account

• Add the previously created service account to Artifact Registry with the Artifact Registry Writer role.
• Check the previously created repo, click show info panel, and then click ADD PRINCIPAL

1. Paste the previously created service account
2. Add the Artifact Registry Writer permission
3. Save

Create GCE VM

Create a VM within the same internal network as the GKE Cluster without an external IP.

Enter the name and choose an appropriate VM size

1. Choose the operating system, for instance Ubuntu 22.04
2. Select the previously created service account

Expand Advanced options and Networking

1. Select the VPC and subnet created earlier
2. Disable external IP
3. Done

Create the VM

Test

Click SSH

Install gcloud

Refer to this document for installing gcloud commands.

GCP Bastion VM

sudo su -
sudo apt-get update
sudo apt-get install apt-transport-https ca-certificates gnupg curl sudo
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
sudo apt-get update && sudo apt-get install google-cloud-cli
# Install kubectl and related packages
apt install kubectl google-cloud-sdk-gke-gcloud-auth-plugin -y

Connect to the GKE cluster using the internal IP

GCP Bastion VM

gcloud container clusters get-credentials CLUSTER_NAME --project=PROJECT_NAME --region=asia-east1 --internal-ip
# View kube config
cat ~/.kube/config

# Use kubectl to connect to the cluster; theoretically, there should be nothing, as no resources are created
kubectl get no -o wide

Install docker and push image to Artifact Registry

Follow this guide to install Docker.

GCP Bastion VM

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

# Install Packages
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Docker push image to Registry

Click the repo that was created in Artifact Registry

Copy the path, then click SETUP INSTRUCTIONS

Copy configure docker command

Paste the command in the console

GCP Bastion VM

gcloud auth configure-docker asia-east1-docker.pkg.dev

Expected result; during the process, input Y to agree:

GCP Bastion VM

Adding credentials for: asia-east1-docker.pkg.dev
After update, the following will be written to your Docker config file located at
[/root/.docker/config.json]:
{
 "credHelpers": {
   "asia-east1-docker.pkg.dev": "gcloud"
 }
}

Do you want to continue (Y/n)?  Y

Docker configuration file updated.

Pull Ubuntu image and push it to Artifact Registry

GCP Bastion VM

docker pull ubuntu

# Tag with the copied path and name + tag
docker tag ubuntu:latest asia-east1-docker.pkg.dev/PROJECT_NAME/REPO_NAME/demo:latest

# Push to the repo
docker push asia-east1-docker.pkg.dev/PROJECT_NAME/REPO_NAME/demo:latest

Now, the image should be stored in Artifact Registry.

Deploy image to GKE

Use the following yaml for deployment and change the image path to the one used earlier for push.

GCP Bastion VM

echo 'apiVersion: v1
kind: Pod
metadata:
  name: ubuntu
spec:
  containers:
  - name: ubuntu
    image: asia-east1-docker.pkg.dev/PROJECT_NAME/REPO_NAME/demo:latest
    # Just spin & wait forever
    command: [ "/bin/bash", "-c", "--" ]
    args: [ "while true; do sleep 30000; done;" ]' > sleep.yaml

Deploy to GKE autopilot, it can take several minutes to complete the deployment

GCP Bastion VM

kubectl apply -f sleep.yaml
kubectl get po

Expected result:

GCP Bastion VM

NAME     READY   STATUS    RESTARTS   AGE
ubuntu   1/1     Running   0          4m21s

Check if the external IP is the NAT Gateway IP

GCP Bastion VM

kubectl exec ubuntu -it -- apt update > /dev/null && apt install -y curl > /dev/null && curl https://api.ipify.org

After testing, remove the pod

GCP Bastion VM

kubectl delete -f sleep.yaml

Create CloudBuild Private Pool VPC

The following steps are based on this and this document.

Create new VPC

Same method as creating “GKE VPC”, configure as follows:

• VPC Name: build-network
• Subnet Name: nat-subnet
• IP: 10.1.0.0/24
• Region: asia-east1

Create PSA

Open VPC to create IP RANGE

Allocate IP range

After creation, ensure the firewall allows this segment

Create Private connections to services

Click Create connection

Select Google Cloud Platform and check the newly created subnet

Open Export custom route

Firewall Settings

• Name: allow-pool-to-nat
• Direction of traffic: Ingress
• Action on match: Allow
• Targets: Specified target tags
• Target tags: nat-gateway
• Source IPv4 Ranges: 192.168.124.0/24
• Protocols and ports: Allow all

Create CloudBuild

Set Cloud Build Network Environment

Click me to enable the necessary API and refer to this document to create Private Pool.

Create a worker pool.

1. Enter name and select region
2. Choose Private network
3. Select the created VPC network (build-network)
4. Do not assign an external IP to ensure connections through private VPC
5. Create, using default settings of e2-medium and providing 100GB disk space

Create Bridging VM

Refer to this document to ensure CloudBuild uses a fixed IP when connecting to GKE.

Create VM, enter the name, choose the region and an appropriate size, e.g., e2-micro

1. Expand advanced settings for the network setup
2. Input network tags: direct-gateway-access, nat-gateway to be used later
3. Enable IP forwarding

1. Select the created VPC network (build-network) and its subnetwork
2. Reserve static IP
3. Done

1. Enter the name for the reserved IP
2. Customize the IP or let it auto-allocate
3. Reserve

Also, reserve a public IP

Paste the following startup script in the startup script field

Bridging VM startup script

sysctl -w net.ipv4.ip_forward=1
iptables -t nat -A POSTROUTING -o $(ip addr show scope global | head -1 | awk -F: '{print $2}') -j MASQUERADE

Create VPC Routes

To ensure all outgoing connections use fixed IP, create 4 routes. The main reason is to avoid the default 0.0.0.0/0. Thus, use 0.0.0.0/1 and 128.0.0.0/1 to prioritize.

• Redirect all traffic to the Bridging VM, priority 100
  • 0.0.0.0/1 -> VM
  • 128.0.0.0/1 -> VM
• Redirect traffic from the Bridging VM to the Default internet gateway (i.e., the VM’s public IP), priority 10 (higher than the first two routes)
  • VM (0.0.0.0/1) -> Default internet gateway
  • VM (128.0.0.0/1) -> Default internet gateway

Open VPC Route and create

1. Select Routes
2. Click ROUTE MANAGEMENT
3. Create ROUTE

The first two routes redirect all traffic to the VM as follows

1. Name them through-nat, through-nat2 respectively
2. Select the VPC (build-network)
3. Enter 0.0.0.0/1 and 128.0.0.0/1 respectively
4. Set priority as 100
5. Specify the next hop for all outgoing traffic and enter the previously created VM internal static IP

Click Create at the bottom and create two routes.

The next two routes allow traffic from the VM to the external IP. Configure as follows:

1. Name them direct-to-gateway1 and direct-to-gateway2
2. Select the VPC (build-network)
3. Enter 0.0.0.0/1 and 128.0.0.0/1 respectively
4. Set priority as 10 (lower than the first two routes)
5. Enter the tag: direct-gateway-access
6. Next hop: Default internet gateway

Click Create at the bottom and create two routes.

Set GKE Authorized Network

Configure to allow connection from the previously created Bridging VM. Click on the created GKE Cluster.

Click Control plane authorized networks and Edit

1. Add a new entry
2. Enter name and Bridging VM public static IP
3. Save

Test External Connectivity

Open Cloud Shell and create a cloudbuild.yaml file. Paste the following content and modify the purple parameters.

This script will verify if the outgoing IP is the Bridging VM’s public static IP.

cloudbuild.yaml

steps:
- name: alpine
  args:
    - sh
    - -exc
    - |
      apk update
      apk add bind-tools
      dig @resolver1.opendns.com myip.opendns.com
options:
  pool:
    name: 'projects/YOUR_PROJECT_NAME/locations/asia-east1/workerPools/YOUR_POOL_NAME'

Run the following command

Shell

gcloud builds submit --no-source

Expected result; the highlighted part is the VM’s external IP

Shell

Created [https://cloudbuild.googleapis.com/v1/projects/YOUR_PROJECT_NAME/locations/asia-east1/builds/f25e0644-3d37-4a8a-89ac-xxxxxxx].
Logs are available at [ https://console.cloud.google.com/cloud-build/builds;region=asia-east1/f25e0644-3d37-4a8a-89ac-xxxxxxx?project=xxxxxxx ].
----------------------------------------------------------- REMOTE BUILD OUTPUT -----------------------------------------------------------
starting build "f25e0644-3d37-4a8a-89ac-xxxxxxx"

FETCHSOURCE
BUILD
Pulling image: alpine
Using default tag: latest
latest: Pulling from library/alpine
Digest: sha256:51b67269f354137895d43f3b3d810bfacd39454xxxxxxx
Status: Downloaded newer image for alpine:latest
docker.io/library/alpine:latest
+ apk update
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
v3.19.0-148-g1780794db9c [https://dl-cdn.alpinelinux.org/alpine/v3.19/main]
v3.19.0-149-gf57fb478059 [https://dl-cdn.alpinelinux.org/alpine/v3.19/community]
OK: 22981 distinct packages available
+ apk add bind-tools
(1/14) Installing fstrm (0.6.1-r4)
(2/14) Installing krb5-conf (1.0-r2)
(3/14) Installing libcom_err (1.47.0-r5)
(4/14) Installing keyutils-libs (1.6.3-r3)
(5/14) Installing libverto (0.3.2-r2)
(6/14) Installing krb5-libs (1.21.2-r0)
(7/14) Installing json-c (0.17-r0)
(8/14) Installing nghttp2-libs (1.58.0-r0)
(9/14) Installing protobuf-c (1.4.1-r7)
(10/14) Installing libuv (1.47.0-r0)
(11/14) Installing xz-libs (5.4.5-r0)
(12/14) Installing libxml2 (2.11.6-r0)
(13/14) Installing bind-libs (9.18.19-r1)
(14/14) Installing bind-tools (9.18.19-r1)
Executing busybox-1.36.1-r15.trigger
OK: 15 MiB in 29 packages
+ dig @resolver1.opendns.com myip.opendns.com

; <<>> DiG 9.18.19 <<>> @resolver1.opendns.com myip.opendns.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 52347
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;myip.opendns.com.              IN      A

;; ANSWER SECTION:
myip.opendns.com.       0       IN      A       x.x.x.x

;; Query time: 28 msec
;; SERVER: 208.67.222.222#53(resolver1.opendns.com) (UDP)
;; WHEN: Fri Dec 29 01:31:47 UTC 2023
;; MSG SIZE  rcvd: 61

PUSH
DONE
-------------------------------------------------------------------------------------------------------------------------------------------
ID: f25e0644-3d37-4a8a-89ac-xxxxxxx
CREATE_TIME: 2023-12-29T01:30:57+00:00
DURATION: 9S
SOURCE: -
IMAGES: -
STATUS: SUCCESS

Cloudbuild yaml file example

In the substitutions section, replace it with your environment variables. This example deploys a specific container image to GKE and integrates with CloudBuild triggered by Pub/Sub for GKE deployment.

cloudbuild.yaml

---
steps:
  - name: gcr.io/cloud-builders/gcloud
    entrypoint: gcloud
    args: ["container", "clusters", "get-credentials", "$_CLUSTER_NAME", "--region", "$_REGION_NAME", "--project", "$_PROJECT_ID"]

  - name: gcr.io/cloud-builders/gcloud
    entrypoint: sh
    args: ["-c", "cat $_DEPLOY.template |sed -e 's%{{IMAGE}}%$_REGISTRY_NAME/$_PROJECT_ID/$_REGISTRY_REPO_NAME/$_REPO_NAME:$_TAG_NAME%g' > $_DEPLOY"]

  - name: gcr.io/cloud-builders/gcloud
    entrypoint: kubectl
    args: ["apply", "-f", "$_DEPLOY", "-n", "$_NAMESPACE_NAME"]

options:
  env:
  - 'KUBECONFIG=/workspace/kubeconfig'
  logging: CLOUD_LOGGING_ONLY
  pool:
    name: 'projects/$_PROJECT_ID/locations/$_REGION_NAME/workerPools/$_WORKER_POOL_NAME'

substitutions:
  _PROJECT_ID: YOUR_PROJECT_ID
  _REGISTRY_NAME: YOUR_REGISTRY_LOCATION
  _REGISTRY_REPO_NAME: YOUR_REGISTRY_REPO_NAME
  _REPO_NAME: "YOUR_REPO_NAME"
  _TAG_NAME: "latest"
  _DEPLOY: kube-hello-change.yaml # YOUR K8S YAML
  _CLUSTER_NAME: YOUR_GKE_CLUSTER_NAME
  _REGION_NAME: asia-east1
  _NAMESPACE_NAME: default
  _WORKER_POOL_NAME: YOUR_WORKER_POOL_NAME

timeout: 6000s

Example

cloudbuild.yaml

---
steps:

  - name: gcr.io/cloud-builders/gcloud
    entrypoint: gcloud
    args: ["container", "clusters", "get-credentials", "$_CLUSTER_NAME", "--region", "$_REGION_NAME", "--project", "$_PROJECT_ID"]

  - name: gcr.io/cloud-builders/gcloud
    entrypoint: sh
    args: ["-c", "cat deploy/$_DEPLOY.template |sed -e 's%{{IMAGE}}%$_REGISTRY_NAME/$_PROJECT_ID/$_REGISTRY_REPO_NAME/$_REPO_NAME:$_TAG_NAME%g' > $_DEPLOY"]

  - name: gcr.io/cloud-builders/gcloud
    entrypoint: kubectl
    args: ["apply", "-f", "$_DEPLOY", "-n", "$_NAMESPACE_NAME"]

options:
  logging: CLOUD_LOGGING_ONLY
  env:
  - 'KUBECONFIG=/workspace/kubeconfig'
  pool:
    name: 'projects/$_PROJECT_ID/locations/$_REGION_NAME/workerPools/$_WORKER_POOL_NAME'
substitutions:
  _PROJECT_ID: OOOOOOXXXXXXXX
  _REGISTRY_NAME: asia-east1-docker.pkg.dev
  _REGISTRY_REPO_NAME: docker-repo
  _REPO_NAME: "test-api"
  _TAG_NAME: "latest"
  _DIR_NAME: "./"
  _DEPLOY: kube-hello-change.yaml
  _CLUSTER_NAME: dev-cluster
  _REGION_NAME: asia-east1
  _NAMESPACE_NAME: default
  _WORKER_POOL_NAME: test-build

timeout: 6000s

kube-hello-change.yaml.template

apiVersion: v1
kind: Service
metadata:
  name: hello-test-service
  annotations:
    networking.gke.io/load-balancer-type: "Internal"
spec:
  type: LoadBalancer
  externalTrafficPolicy: Cluster
  selector:
    app: hello-test
  ports:
  - protocol: "TCP"
    port: 8080
    targetPort: 9999

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-test
spec:
  selector:
    matchLabels:
      app: hello-test
  replicas: 1
  template:
    metadata:
      labels:
        app: hello-test
    spec:
      containers:
      - name: test-pod
        image: {{IMAGE}}
        imagePullPolicy: Always
        ports:
        - name: http-server
          containerPort: 9999
        livenessProbe:
          httpGet:
            path: /actuator/health/liveness
            port: http-server
          initialDelaySeconds: 60
          periodSeconds: 5
        resources:
          limits:
            cpu: 500m
            ephemeral-storage: 1Gi
            memory: 2Gi
          requests:
            cpu: 500m
            ephemeral-storage: 1Gi
            memory: 2Gi
        readinessProbe:
          httpGet:
            path: /actuator/health/readiness
            port: http-server
          initialDelaySeconds: 60
          periodSeconds: 5