GKE Private Cluster Connect to CloudSQL by Internal IP

Note

This post translated By AI and the original content was written in Traditional Chinese.

Image source

When using GKE and CloudSQL in an all-private manner, it’s important to pay attention to whether the GKE Control Plane and the kubectl user connection endpoint are peering.

Scenario

The following network segments need to be planned and carefully considered to avoid overlap, which could cause connection issues.

• GKE
  • Control Plane
  • Node
  • Pod
  • Service
• CloudSQL
  • Private Service Access

Implementation Method

• The private cluster uses a completely closed approach, requiring an internal network jump box VM for connection
• When connecting to the jump box VM, environment variables need to be re-pasted to obtain the cluster credential
• Create a pod with mysqlclient for connection testing

Create GKE private cluster and network

• Use command-line instructions to create a VPC network and GKE private cluster
• Remember to adjust the following environment variables

CloudShell

export NETWORK_NAME="ricky-private-gke"
export SUBNET_NAME="subnet-us-east"
export PROJECT_ID="YOUR_PROJECT_ID"
export REGION="us-east4"
export ZONE="us-east4-c"
export GKE_CLUSTER_NAME="ricky-nat-test-cluster"
export BASTION_VM_NAME="ricky-bastion"
export ROUTER_NAME="ricky-nat-router"
export DB_NAME="ricky-mysql"
export DB_IP_RANGE="10.0.101.0"
export DB_IP_PREFIX="24"
export CP_IP_RANGE="10.0.100.0/28"
export NODE_IP_RANGE="10.0.1.0/24"
export POD_IP_RANGE="10.1.0.0/16"
export SERVICE_IP_RANGE="10.2.0.0/24"

# create network
gcloud compute networks create ${NETWORK_NAME} --subnet-mode custom
gcloud compute networks subnets create ${SUBNET_NAME} \
    --network ${NETWORK_NAME} \
    --region ${REGION} \
    --range ${NODE_IP_RANGE} \
    --secondary-range my-pods=${POD_IP_RANGE},my-services=${SERVICE_IP_RANGE} \
    --enable-private-ip-google-access

gcloud compute firewall-rules create allow-ssh \
    --network ${NETWORK_NAME} \
    --source-ranges 35.235.240.0/20 \
    --allow tcp:22


gcloud compute addresses create google-managed-services-${NETWORK_NAME} \
--global \
--purpose=VPC_PEERING \
--addresses=${DB_IP_RANGE} \
--prefix-length=${DB_IP_PREFIX} \
--network=projects/${PROJECT_ID}/global/networks/${NETWORK_NAME}

gcloud services vpc-peerings connect \
--service=servicenetworking.googleapis.com \
--ranges=google-managed-services-${NETWORK_NAME} \
--network=${NETWORK_NAME} \
--project=${PROJECT_ID}


# create private cluster
gcloud container clusters create ${GKE_CLUSTER_NAME} \
    --zone ${ZONE} \
    --cluster-version "latest" \
    --machine-type "n2-standard-2" \
    --disk-type "pd-standard" \
    --disk-size "100" \
    --scopes "https://www.googleapis.com/auth/compute","https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" \
    --num-nodes "1" \
    --enable-private-nodes \
    --enable-private-endpoint \
    --master-ipv4-cidr ${CP_IP_RANGE} \
    --enable-ip-alias \
    --network ${NETWORK_NAME} \
    --subnetwork ${SUBNET_NAME} \
    --max-nodes-per-pool "110" \
    --enable-master-authorized-networks \
    --addons HorizontalPodAutoscaling,HttpLoadBalancing \
    --enable-autoupgrade \
    --enable-autorepair \
    --cluster-secondary-range-name my-pods \
    --services-secondary-range-name my-services \
    --enable-network-policy

# create bastion
gcloud compute instances create ${BASTION_VM_NAME} \
    --project=${PROJECT_ID} \
    --zone=${ZONE} \
    --machine-type=e2-micro \
    --network-interface=network-tier=PREMIUM,stack-type=IPV4_ONLY,subnet=${SUBNET_NAME} \
    --maintenance-policy=MIGRATE \
    --provisioning-model=STANDARD \
    --create-disk=auto-delete=yes,boot=yes,device-name=ricky-bastion,image=projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230616,mode=rw,size=10,type=projects/${PROJECT_ID}/zones/${ZONE}/diskTypes/pd-balanced \
    --no-shielded-secure-boot \
    --shielded-vtpm \
    --shielded-integrity-monitoring \
    --labels=goog-ec-src=vm_add-gcloud \
    --reservation-affinity=any


# add bastion ip to GKE control plane allowlist (for external ip only)
export BASTION_IP=$(gcloud compute instances describe ${BASTION_VM_NAME} --zone ${ZONE} --format='get(networkInterfaces[0].networkIP)')

gcloud container clusters update ${GKE_CLUSTER_NAME} --zone=${ZONE} \
    --enable-master-authorized-networks \
    --master-authorized-networks ${BASTION_IP}/32

Configure the jump box

Install necessary packages, which will take about 5 minutes

CloudShell

gcloud compute ssh ${BASTION_VM_NAME} --zone ${ZONE} --tunnel-through-iap -- "sudo snap remove google-cloud-cli && \
sudo apt-get update && \
sudo apt-get install apt-transport-https ca-certificates gnupg curl -y && \
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg && \
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && \
sudo apt-get update && sudo apt-get install google-cloud-cli kubectl google-cloud-cli-gke-gcloud-auth-plugin -y"

Log in to the gcloud account on the jump box

CloudShell

gcloud compute ssh ${BASTION_VM_NAME} --zone ${ZONE} --tunnel-through-iap -- "gcloud auth login"

Create CloudSQL

Use the following command to create CloudSQL, adjust the password as needed

CloudShell

gcloud beta sql instances create ${DB_NAME} \
--database-version=MYSQL_8_0 \
--tier db-f1-micro \
--region=${REGION} \
--network ${NETWORK_NAME} \
--no-assign-ip \
--allocated-ip-range-name google-managed-services-${NETWORK_NAME} \
--storage-size 10

gcloud sql users set-password root \
--host=% \
--instance ${DB_NAME} \
--password 1qaz2wsx

Connection test

Use mysql-client for connection testing

CloudShell

gcloud compute ssh ${BASTION_VM_NAME} --zone ${ZONE} --tunnel-through-iap
kubectl run mysqlclient --image imega/mysql-client -- sleep 86400
DB_IP=$(gcloud sql instances describe ricky-mysql --format='value(ipAddresses[0].ipAddress)')
kubectl exec mysqlclient -- mysql --host=${DB_IP} --user=root --password=1qaz2wsx --execute='show databases;'
# Remove the pod
kubectl delete po mysqlclient

Expected result, showing all current databases

Database
information_schema
mysql
performance_schema
sys