GKE Private Cluster NAT and Network Policy

Scenario

建立 GKE private cluster，並使用 bastion 以內網私有 ip 連線至 control plane
所有主機透過 NAT 對外進行連線
使用 GKE network policy 進行連線管制

流程與說明

建立 GKE Private Cluster 與跳板機
設定跳板機，安裝必要套件
建立 CloudNAT 以提供對外網路連線
測試 network policy 連線管理與 NAT 對外功能

執行步驟

建立 GKE Private Cluster 與跳板機

1
export PROJECT_ID="YOUR_PROJECT_ID"
2
export NETWORK_NAME="YOUR_NETWORK_NAME"
3
export SUBNET_NAME="YOUR_SUBNET_NAME"
4
export REGION="YOUR_REGIOIN"
5
export ZONE="YOUR_ZONE"
6
export GKE_CLUSTER_NAME="YOUR_GKE_CLUSTER_NAME"
7
export BASTION_VM_NAME="YOUR_BASTION_VM_NAME"
8
export ROUTER_NAME="YOUR_ROUTER_NAME"
9

10
# create network
11
gcloud compute networks create ${NETWORK_NAME} --subnet-mode custom
12
gcloud compute networks subnets create ${SUBNET_NAME} \
13
   --network ${NETWORK_NAME} \
14
   --region ${REGION} \
15
   --range 192.168.1.0/24
16
gcloud compute firewall-rules create allow-ssh \
17
    --network ${NETWORK_NAME} \
18
    --source-ranges 35.235.240.0/20 \
19
    --allow tcp:22
20

21
# create private cluster
22
gcloud container clusters create ${GKE_CLUSTER_NAME} \
23
    --zone ${ZONE} \
24
    --cluster-version "latest" \
25
    --machine-type "n2-standard-2" \
26
    --disk-type "pd-standard" \
27
    --disk-size "100" \
28
    --scopes "https://www.googleapis.com/auth/compute","https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" \
29
    --num-nodes "1" \
30
    --enable-private-nodes \
31
    --enable-private-endpoint \
32
    --master-ipv4-cidr "172.16.0.0/28" \
33
    --enable-ip-alias \
34
    --network "projects/${PROJECT_ID}/global/networks/${NETWORK_NAME}" \
35
    --subnetwork "projects/${PROJECT_ID}/regions/${REGION}/subnetworks/${SUBNET_NAME}" \
36
    --max-nodes-per-pool "110" \
37
    --enable-master-authorized-networks \
38
    --addons HorizontalPodAutoscaling,HttpLoadBalancing \
39
    --enable-autoupgrade \
40
    --enable-autorepair \
41
    --enable-network-policy
42

43
# create bastion
44
gcloud compute instances create ${BASTION_VM_NAME} \
45
    --project=${PROJECT_ID} \
46
    --zone=${ZONE} \
47
    --machine-type=e2-micro \
48
    --network-interface=network-tier=PREMIUM,stack-type=IPV4_ONLY,subnet=${SUBNET_NAME} \
49
    --maintenance-policy=MIGRATE \
50
    --provisioning-model=STANDARD \
51
    --create-disk=auto-delete=yes,boot=yes,device-name=ricky-bastion,image=projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240801,mode=rw,size=10,type=projects/${PROJECT_ID}/zones/${ZONE}/diskTypes/pd-balanced \
52
    --no-shielded-secure-boot \
53
    --shielded-vtpm \
54
    --shielded-integrity-monitoring \
55
    --labels=goog-ec-src=vm_add-gcloud \
56
    --reservation-affinity=any
57
export BASTION_IP=$(gcloud compute instances describe ${BASTION_VM_NAME} --zone ${ZONE} --format='get(networkInterfaces[0].networkIP)')
58

59
# add bastion private to GKE authorized networks
60
gcloud container clusters update ${GKE_CLUSTER_NAME} --zone=${ZONE} \
61
    --enable-master-authorized-networks \
62
    --master-authorized-networks ${BASTION_IP}/32

設定跳板機

安裝必要套件，會需要等待 5 分鐘左右

1
gcloud compute ssh ${BASTION_VM_NAME} --zone ${ZONE} --tunnel-through-iap -- "sudo snap remove google-cloud-cli && \
2
sudo apt-get update && \
3
sudo apt-get install apt-transport-https ca-certificates gnupg curl -y && \
4
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg && \
5
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && \
6
sudo apt-get update && sudo apt-get install google-cloud-cli kubectl google-cloud-cli-gke-gcloud-auth-plugin -y"

登入跳板機的 gcloud 帳號

1
gcloud compute ssh ${BASTION_VM_NAME} --zone ${ZONE} --tunnel-through-iap -- "gcloud auth login"

建立測試用 pod，會針對 test pod 加上 label 以便後續限制對外連線

1
gcloud compute ssh ${BASTION_VM_NAME} --zone ${ZONE} --tunnel-through-iap -- "gcloud container clusters get-credentials ${GKE_CLUSTER_NAME} --zone=${ZONE} && \
2
kubectl get no && \
3
kubectl run test --image nginx -l net=limited && \
4
kubectl run test2 --image nginx && \
5
kubectl get po"

進入 pod 測試確認是否可對外連線

1
gcloud compute ssh ${BASTION_VM_NAME} --zone ${ZONE} --tunnel-through-iap -- "kubectl exec -it test -- apt update"

建立 CloudNAT

1
gcloud compute routers create ${ROUTER_NAME} \
2
    --network ${NETWORK_NAME} \
3
    --region ${REGION}
4
gcloud compute routers nats create nat-config \
5
    --router-region ${REGION} \
6
    --router ${ROUTER_NAME} \
7
    --nat-all-subnet-ip-ranges \
8
    --auto-allocate-nat-external-ips

(Optional) 啟用 GKE Network Policy 功能

如果為既有 GKE cluster 可輸入以下指令進行啟用

1
gcloud container clusters update ${GKE_CLUSTER_NAME} --zone=${ZONE} --update-addons=NetworkPolicy=ENABLED
2
gcloud container clusters update ${GKE_CLUSTER_NAME} --zone=${ZONE}  --enable-network-policy

測試

對外與對內連線測試

進入 pod 測試確認是否可對外連線

1
gcloud compute ssh ${BASTION_VM_NAME} --zone ${ZONE} --tunnel-through-iap -- "kubectl exec -it test -- apt update"

1
gcloud compute ssh ${BASTION_VM_NAME} --zone ${ZONE} --tunnel-through-iap -- 'TEST2_IP=$(kubectl get po test2 -o jsonpath='{.status.podIP}') && \
2
kubectl exec -it test -- curl ${TEST2_IP} && \
3
kubectl exec -it test -- curl https://google.com'

預期結果


29 collapsed lines
1
<!DOCTYPE html>
2
<html>
3
<head>
4
<title>Welcome to nginx!</title>
5
<style>
6
html { color-scheme: light dark; }
7
body { width: 35em; margin: 0 auto;
8
font-family: Tahoma, Verdana, Arial, sans-serif; }
9
</style>
10
</head>
11
<body>
12
<h1>Welcome to nginx!</h1>
13
<p>If you see this page, the nginx web server is successfully installed and
14
working. Further configuration is required.</p>
15

16
<p>For online documentation and support please refer to
17
<a href="http://nginx.org/">nginx.org</a>.<br/>
18
Commercial support is available at
19
<a href="http://nginx.com/">nginx.com</a>.</p>
20

21
<p><em>Thank you for using nginx.</em></p>
22
</body>
23
</html>
24
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
25
<TITLE>301 Moved</TITLE></HEAD><BODY>
26
<H1>301 Moved</H1>
27
The document has moved
28
<A HREF="https://www.google.com/">here</A>.
29
</BODY></HTML>

建立 Network Policy 限制 test

1
gcloud compute ssh ${BASTION_VM_NAME} --zone ${ZONE} --tunnel-through-iap -- "echo 'apiVersion: networking.k8s.io/v1
2
kind: NetworkPolicy
3
metadata:
4
  name: default-deny-egress
5
  namespace: default
6
spec:
7
    podSelector:
8
      matchLabels:
9
        net: limited
10
    policyTypes:
11
    - Egress
12
    egress:
13
      - to:
14
          - ipBlock:
15
              cidr: 10.0.0.0/8
16
          - ipBlock:
17
              cidr: 172.16.0.0/12
18
          - ipBlock:
19
              cidr: 192.168.0.0/16' > deny-all.yaml && \
20
kubectl apply -f deny-all.yaml"

test pod 對 test2 pod 進行連線測試，可成功連線

1
gcloud compute ssh ${BASTION_VM_NAME} --zone ${ZONE} --tunnel-through-iap -- 'TEST2_IP=$(kubectl get po test2 -o jsonpath='{.status.podIP}') && \
2
kubectl exec -it test -- curl ${TEST2_IP}'

test pod 對外網進行連線測試，會發現舞法對外

1
gcloud compute ssh ${BASTION_VM_NAME} --zone ${ZONE} --tunnel-through-iap -- "kubectl exec -it test -- curl https://google.com"

test2 pod 對外網進行連線測試，但可以正式對外

1
gcloud compute ssh ${BASTION_VM_NAME} --zone ${ZONE} --tunnel-through-iap -- "kubectl exec -it test2 -- curl https://google.com"

Clean Up

1
gcloud compute instances delete ${BASTION_VM_NAME} --zone=${ZONE} --quiet
2
gcloud container clusters delete ${GKE_CLUSTER_NAME} --zone=${ZONE} --quiet
3
gcloud compute routers delete ${ROUTER_NAME} --region ${REGION} --quiet
4
gcloud compute firewall-rules delete allow-ssh --quiet
5
gcloud compute networks subnets delete ${SUBNET_NAME} --region ${REGION} --quiet
6
gcloud compute networks delete ${NETWORK_NAME} --quiet