GKE Private Cluster Connect to CloudSQL by Internal IP

GKE Control Plane Peering 圖片來源

採用 all private 方式使用 GKE 與 CloudSQL，需留意 GKE Control Plane 與使用 kubectl 使用者連線端是否 peering。

Scenario

須規劃並留意以下網段並避免 overlap，造成連線上的問題。

GKE
- Control Plane
- Node
- Pod
- Service
CloudSQL
- Private Service Access

實做方式

private cluster 使用完全封閉式，須透過內網跳板 VM 才可連接
連入跳板 VM 時，環境變數須再重貼一次，才可取得 cluster credential
建立一個具有 mysqlclient 的 pod 進行連線測試

建立 GKE private cluster and network

使用指令方式建立 VPC 網路與 GKE private cluster
記得將以下環境變數進行調整

1
export NETWORK_NAME="ricky-private-gke"
2
export SUBNET_NAME="subnet-us-east"
3
export PROJECT_ID="YOUR_PROJECT_ID"
4
export REGION="us-east4"
5
export ZONE="us-east4-c"
6
export GKE_CLUSTER_NAME="ricky-nat-test-cluster"
7
export BASTION_VM_NAME="ricky-bastion"
8
export ROUTER_NAME="ricky-nat-router"
9
export DB_NAME="ricky-mysql"
10
export DB_IP_RANGE="10.0.101.0"
11
export DB_IP_PREFIX="24"
12
export CP_IP_RANGE="10.0.100.0/28"
13
export NODE_IP_RANGE="10.0.1.0/24"
14
export POD_IP_RANGE="10.1.0.0/16"
15
export SERVICE_IP_RANGE="10.2.0.0/24"
16

17
# create network
18
gcloud compute networks create ${NETWORK_NAME} --subnet-mode custom
19
gcloud compute networks subnets create ${SUBNET_NAME} \
20
    --network ${NETWORK_NAME} \
21
    --region ${REGION} \
22
    --range ${NODE_IP_RANGE} \
23
    --secondary-range my-pods=${POD_IP_RANGE},my-services=${SERVICE_IP_RANGE} \
24
    --enable-private-ip-google-access
25

26
gcloud compute firewall-rules create allow-ssh \
27
    --network ${NETWORK_NAME} \
28
    --source-ranges 35.235.240.0/20 \
29
    --allow tcp:22
30

31

32
gcloud compute addresses create google-managed-services-${NETWORK_NAME} \
33
--global \
34
--purpose=VPC_PEERING \
35
--addresses=${DB_IP_RANGE} \
36
--prefix-length=${DB_IP_PREFIX} \
37
--network=projects/${PROJECT_ID}/global/networks/${NETWORK_NAME}
38

39
gcloud services vpc-peerings connect \
40
--service=servicenetworking.googleapis.com \
41
--ranges=google-managed-services-${NETWORK_NAME} \
42
--network=${NETWORK_NAME} \
43
--project=${PROJECT_ID}
44

45

46
# create private cluster
47
gcloud container clusters create ${GKE_CLUSTER_NAME} \
48
    --zone ${ZONE} \
49
    --cluster-version "latest" \
50
    --machine-type "n2-standard-2" \
51
    --disk-type "pd-standard" \
52
    --disk-size "100" \
53
    --scopes "https://www.googleapis.com/auth/compute","https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" \
54
    --num-nodes "1" \
55
    --enable-private-nodes \
56
    --enable-private-endpoint \
57
    --master-ipv4-cidr ${CP_IP_RANGE} \
58
    --enable-ip-alias \
59
    --network ${NETWORK_NAME} \
60
    --subnetwork ${SUBNET_NAME} \
61
    --max-nodes-per-pool "110" \
62
    --enable-master-authorized-networks \
63
    --addons HorizontalPodAutoscaling,HttpLoadBalancing \
64
    --enable-autoupgrade \
65
    --enable-autorepair \
66
    --cluster-secondary-range-name my-pods \
67
    --services-secondary-range-name my-services \
68
    --enable-network-policy
69

70
# create bastion
71
gcloud compute instances create ${BASTION_VM_NAME} \
72
    --project=${PROJECT_ID} \
73
    --zone=${ZONE} \
74
    --machine-type=e2-micro \
75
    --network-interface=network-tier=PREMIUM,stack-type=IPV4_ONLY,subnet=${SUBNET_NAME} \
76
    --maintenance-policy=MIGRATE \
77
    --provisioning-model=STANDARD \
78
    --create-disk=auto-delete=yes,boot=yes,device-name=ricky-bastion,image=projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230616,mode=rw,size=10,type=projects/${PROJECT_ID}/zones/${ZONE}/diskTypes/pd-balanced \
79
    --no-shielded-secure-boot \
80
    --shielded-vtpm \
81
    --shielded-integrity-monitoring \
82
    --labels=goog-ec-src=vm_add-gcloud \
83
    --reservation-affinity=any
84

85

86
# add bastion ip to GKE control plane allowlist (for external ip only)
87
export BASTION_IP=$(gcloud compute instances describe ${BASTION_VM_NAME} --zone ${ZONE} --format='get(networkInterfaces[0].networkIP)')
88

89
gcloud container clusters update ${GKE_CLUSTER_NAME} --zone=${ZONE} \
90
    --enable-master-authorized-networks \
91
    --master-authorized-networks ${BASTION_IP}/32

設定跳板機

安裝必要套件，會需要等待 5 分鐘左右

1
gcloud compute ssh ${BASTION_VM_NAME} --zone ${ZONE} --tunnel-through-iap -- "sudo snap remove google-cloud-cli && \
2
sudo apt-get update && \
3
sudo apt-get install apt-transport-https ca-certificates gnupg curl -y && \
4
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg && \
5
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && \
6
sudo apt-get update && sudo apt-get install google-cloud-cli kubectl google-cloud-cli-gke-gcloud-auth-plugin -y"

登入跳板機的 gcloud 帳號

1
gcloud compute ssh ${BASTION_VM_NAME} --zone ${ZONE} --tunnel-through-iap -- "gcloud auth login"

建立 CloudSQL

使用以下指令建立 CloudSQL，password 部分可依照需求調整

1
gcloud beta sql instances create ${DB_NAME} \
2
--database-version=MYSQL_8_0 \
3
--tier db-f1-micro \
4
--region=${REGION} \
5
--network ${NETWORK_NAME} \
6
--no-assign-ip \
7
--allocated-ip-range-name google-managed-services-${NETWORK_NAME} \
8
--storage-size 10
9

10
gcloud sql users set-password root \
11
--host=% \
12
--instance ${DB_NAME} \
13
--password 1qaz2wsx

連線測試

使用 mysql-client 進行連測試

1
gcloud compute ssh ${BASTION_VM_NAME} --zone ${ZONE} --tunnel-through-iap
2
kubectl run mysqlclient --image imega/mysql-client -- sleep 86400
3
DB_IP=$(gcloud sql instances describe ricky-mysql --format='value(ipAddresses[0].ipAddress)')
4
kubectl exec mysqlclient -- mysql --host=${DB_IP} --user=root --password=1qaz2wsx --execute='show databases;'
5
# 移除 pod
6
kubectl delete po mysqlclient

預期結果，會出現目前所有的資料庫

1
Database
2
information_schema
3
mysql
4
performance_schema
5
sys