Skip to content
Ricky's Tech Explorer Log

GKE Private Cluster NAT and Network Policy

Scenario

  • Create a GKE private cluster and use a bastion to connect to the control plane via internal private IP
  • All hosts connect externally through NAT
  • Use GKE network policy for connection control

Process and Description

  1. Create GKE Private Cluster and bastion host
  2. Configure the bastion host, install necessary packages
  3. Create CloudNAT to provide external network connections
  4. Test network policy connection management and NAT external functionality

Execution Steps

Create GKE Private Cluster and Bastion Host

CloudShell
1
export PROJECT_ID="YOUR_PROJECT_ID"
2
export NETWORK_NAME="YOUR_NETWORK_NAME"
3
export SUBNET_NAME="YOUR_SUBNET_NAME"
4
export REGION="YOUR_REGION"
5
export ZONE="YOUR_ZONE"
6
export GKE_CLUSTER_NAME="YOUR_GKE_CLUSTER_NAME"
7
export BASTION_VM_NAME="YOUR_BASTION_VM_NAME"
8
export ROUTER_NAME="YOUR_ROUTER_NAME"
9


10
# create network
11
gcloud compute networks create ${NETWORK_NAME} --subnet-mode custom
12
gcloud compute networks subnets create ${SUBNET_NAME} \
13
   --network ${NETWORK_NAME} \
14
   --region ${REGION} \
15
   --range 192.168.1.0/24
16
gcloud compute firewall-rules create allow-ssh \
17
    --network ${NETWORK_NAME} \
18
    --source-ranges 35.235.240.0/20 \
19
    --allow tcp:22
20


21
# create private cluster
22
gcloud container clusters create ${GKE_CLUSTER_NAME} \
23
    --zone ${ZONE} \
24
    --cluster-version "latest" \
25
    --machine-type "n2-standard-2" \
26
    --disk-type "pd-standard" \
27
    --disk-size "100" \
28
    --scopes "https://www.googleapis.com/auth/compute","https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" \
29
    --num-nodes "1" \
30
    --enable-private-nodes \
31
    --enable-private-endpoint \
32
    --master-ipv4-cidr "172.16.0.0/28" \
33
    --enable-ip-alias \
34
    --network "projects/${PROJECT_ID}/global/networks/${NETWORK_NAME}" \
35
    --subnetwork "projects/${PROJECT_ID}/regions/${REGION}/subnetworks/${SUBNET_NAME}" \
36
    --max-nodes-per-pool "110" \
37
    --enable-master-authorized-networks \
38
    --addons HorizontalPodAutoscaling,HttpLoadBalancing \
39
    --enable-autoupgrade \
40
    --enable-autorepair \
41
    --enable-network-policy
42


43
# create bastion
44
gcloud compute instances create ${BASTION_VM_NAME} \
45
    --project=${PROJECT_ID} \
46
    --zone=${ZONE} \
47
    --machine-type=e2-micro \
48
    --network-interface=network-tier=PREMIUM,stack-type=IPV4_ONLY,subnet=${SUBNET_NAME} \
49
    --maintenance-policy=MIGRATE \
50
    --provisioning-model=STANDARD \
51
    --create-disk=auto-delete=yes,boot=yes,device-name=ricky-bastion,image=projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240801,mode=rw,size=10,type=projects/${PROJECT_ID}/zones/${ZONE}/diskTypes/pd-balanced \
52
    --no-shielded-secure-boot \
53
    --shielded-vtpm \
54
    --shielded-integrity-monitoring \
55
    --labels=goog-ec-src=vm_add-gcloud \
56
    --reservation-affinity=any
57
export BASTION_IP=$(gcloud compute instances describe ${BASTION_VM_NAME} --zone ${ZONE} --format='get(networkInterfaces[0].networkIP)')
58


59
# add bastion private to GKE authorized networks
60
gcloud container clusters update ${GKE_CLUSTER_NAME} --zone=${ZONE} \
61
    --enable-master-authorized-networks \
62
    --master-authorized-networks ${BASTION_IP}/32

Configure Bastion Host

Install necessary packages, this will take about 5 minutes

CloudShell
1
gcloud compute ssh ${BASTION_VM_NAME} --zone ${ZONE} --tunnel-through-iap -- "sudo snap remove google-cloud-cli && \
2
sudo apt-get update && \
3
sudo apt-get install apt-transport-https ca-certificates gnupg curl -y && \
4
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg && \
5
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && \
6
sudo apt-get update && sudo apt-get install google-cloud-cli kubectl google-cloud-cli-gke-gcloud-auth-plugin -y"

Log in to the gcloud account on the bastion host

CloudShell
1
gcloud compute ssh ${BASTION_VM_NAME} --zone ${ZONE} --tunnel-through-iap -- "gcloud auth login"

Create test pods, add a label to the test pod for later restriction of external connections

CloudShell
1
gcloud compute ssh ${BASTION_VM_NAME} --zone ${ZONE} --tunnel-through-iap -- "gcloud container clusters get-credentials ${GKE_CLUSTER_NAME} --zone=${ZONE} && \
2
kubectl get no && \
3
kubectl run test --image nginx -l net=limited && \
4
kubectl run test2 --image nginx && \
5
kubectl get po"

Enter the pod to test and confirm if external connections are possible

CloudShell
1
gcloud compute ssh ${BASTION_VM_NAME} --zone ${ZONE} --tunnel-through-iap -- "kubectl exec -it test -- apt update"

Create CloudNAT

CloudShell
1
gcloud compute routers create ${ROUTER_NAME} \
2
    --network ${NETWORK_NAME} \
3
    --region ${REGION}
4
gcloud compute routers nats create nat-config \
5
    --router-region ${REGION} \
6
    --router ${ROUTER_NAME} \
7
    --nat-all-subnet-ip-ranges \
8
    --auto-allocate-nat-external-ips

(Optional) Enable GKE Network Policy Feature

If it’s an existing GKE cluster, you can enter the following command to enable it

CloudShell
1
gcloud container clusters update ${GKE_CLUSTER_NAME} --zone=${ZONE} --update-addons=NetworkPolicy=ENABLED
2
gcloud container clusters update ${GKE_CLUSTER_NAME} --zone=${ZONE}  --enable-network-policy

Testing

External and Internal Connection Testing

Enter the pod to test and confirm if external connections are possible

CloudShell
1
gcloud compute ssh ${BASTION_VM_NAME} --zone ${ZONE} --tunnel-through-iap -- "kubectl exec -it test -- apt update"
CloudShell
1
gcloud compute ssh ${BASTION_VM_NAME} --zone ${ZONE} --tunnel-through-iap -- 'TEST2_IP=$(kubectl get po test2 -o jsonpath='{.status.podIP}') && \
2
kubectl exec -it test -- curl ${TEST2_IP} && \
3
kubectl exec -it test -- curl https://google.com'

Expected result

CloudShell
29 collapsed lines
1
<!DOCTYPE html>
2
<html>
3
<head>
4
<title>Welcome to nginx!</title>
5
<style>
6
html { color-scheme: light dark; }
7
body { width: 35em; margin: 0 auto;
8
font-family: Tahoma, Verdana, Arial, sans-serif; }
9
</style>
10
</head>
11
<body>
12
<h1>Welcome to nginx!</h1>
13
<p>If you see this page, the nginx web server is successfully installed and
14
working. Further configuration is required.</p>
15


16
<p>For online documentation and support please refer to
17
<a href="http://nginx.org/">nginx.org</a>.<br/>
18
Commercial support is available at
19
<a href="http://nginx.com/">nginx.com</a>.</p>
20


21
<p><em>Thank you for using nginx.</em></p>
22
</body>
23
</html>
24
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
25
<TITLE>301 Moved</TITLE></HEAD><BODY>
26
<H1>301 Moved</H1>
27
The document has moved
28
<A HREF="https://www.google.com/">here</A>.
29
</BODY></HTML>

Create Network Policy to Restrict Test

CloudShell
1
gcloud compute ssh ${BASTION_VM_NAME} --zone ${ZONE} --tunnel-through-iap -- "echo 'apiVersion: networking.k8s.io/v1
2
kind: NetworkPolicy
3
metadata:
4
  name: default-deny-egress
5
  namespace: default
6
spec:
7
    podSelector:
8
      matchLabels:
9
        net: limited
10
    policyTypes:
11
    - Egress
12
    egress:
13
      - to:
14
          - ipBlock:
15
              cidr: 10.0.0.0/8
16
          - ipBlock:
17
              cidr: 172.16.0.0/12
18
          - ipBlock:
19
              cidr: 192.168.0.0/16' > deny-all.yaml && \
20
kubectl apply -f deny-all.yaml"

Test connection from test pod to test2 pod, should be successful

CloudShell
1
gcloud compute ssh ${BASTION_VM_NAME} --zone ${ZONE} --tunnel-through-iap -- 'TEST2_IP=$(kubectl get po test2 -o jsonpath='{.status.podIP}') && \
2
kubectl exec -it test -- curl ${TEST2_IP}'

Test external connection from test pod, should fail

CloudShell
1
gcloud compute ssh ${BASTION_VM_NAME} --zone ${ZONE} --tunnel-through-iap -- "kubectl exec -it test -- curl https://google.com"

Test external connection from test2 pod, should succeed

CloudShell
1
gcloud compute ssh ${BASTION_VM_NAME} --zone ${ZONE} --tunnel-through-iap -- "kubectl exec -it test2 -- curl https://google.com"

Clean Up

CloudShell
1
gcloud compute instances delete ${BASTION_VM_NAME} --zone=${ZONE} --quiet
2
gcloud container clusters delete ${GKE_CLUSTER_NAME} --zone=${ZONE} --quiet
3
gcloud compute routers delete ${ROUTER_NAME} --region ${REGION} --quiet
4
gcloud compute firewall-rules delete allow-ssh --quiet
5
gcloud compute networks subnets delete ${SUBNET_NAME} --region ${REGION} --quiet
6
gcloud compute networks delete ${NETWORK_NAME} --quiet

REF