跳到內容
Ricky's Tech Explorer Log

GCP Secret Manager on Google Kubernetes Engine

secret store csi driver diagram 圖片來源

Scenario

  • 如上圖所示，其原理為使用 GCP Secret Manager 服務儲存 Kubernetes secret 資訊
  • 使用以下兩種其中一種 Secrets Store CSI Driver 綁定權限方式進行設定
    • Kubernetes service account 加入 GCP Secret Manager 特定 secret 賦予權限
    • Kubernetes service account 至 GCP IAM service account，再由 IAM service account 加入 GCP Secret Manager 特定 secret 中賦予權限
    • 以上兩種方式，後者耦合性較低，適用於 multi-cluster 或 multi-project 情境，前者則適用於 single-cluster 或 single-project 情境
  • 其中需要使用到 Secrets Store CSI Driver，支援三朵雲的 provider 外，還有其他第三方 provider

流程與說明

  1. 建立 VPC 網路
  2. 建立 GKE cluster 並開啟 enable-secret-manager 與 workload-pool 功能
  3. 建立 Kubernetes namespace
  4. 設定 service account 以下做法則一即可
  • 使用 Kubernetes service account
  • 使用 IAM service account
  1. 設定 SecretManager
  • 建立 Secret
  • 將 Kubernetes or GCP IAM 之 service account 賦予 secretAccessor 權限並綁訂於所建立之 secret
  • 建立 SecretProviderClasses
  1. 測試
  2. 資源清除

執行步驟

前置作業

建立環境變數

  • 可依照需求調整，操作主要使用 GCP CloudShell
GCP CloudShell
1
export NETWORK_NAME="ricky-demo-gke"
2
export SUBNET_NAME="subnet-us-east"
3
export PROJECT_ID=${DEVSHELL_PROJECT_ID}
4
export PROJECT_NUMBER=$(gcloud projects list --filter PROJECT_ID=${PROJECT_ID} --format="value(PROJECT_NUMBER)")
5
export REGION="us-east4"
6
export ZONE="us-east4-c"
7
export GKE_CLUSTER_NAME="ricky-demo-cluster"
8
export NODE_IP_RANGE="10.0.1.0/24"
9
export POD_IP_RANGE="10.1.0.0/16"
10
export SERVICE_IP_RANGE="10.2.0.0/24"
11
export SECRET_NS="demo"
12
export K8S_SA_NAME="ricky-demo"
13
export SECRET_NAME="ricky-demo-secret"

建立 GKE VPC

GCP CloudShell
1
gcloud compute networks create ${NETWORK_NAME} --subnet-mode custom
2
gcloud compute networks subnets create ${SUBNET_NAME} \
3
    --network ${NETWORK_NAME} \
4
    --region ${REGION} \
5
    --range ${NODE_IP_RANGE} \
6
    --secondary-range my-pods=${POD_IP_RANGE},my-services=${SERVICE_IP_RANGE} \
7
    --enable-private-ip-google-access
8


9
gcloud compute firewall-rules create allow-ssh \
10
    --network ${NETWORK_NAME} \
11
    --source-ranges 35.235.240.0/20 \
12
    --allow tcp:22

建立 GKE cluster

使用前面所建立之 VPC 網路與 Subnet 建立 GKE cluster

GCP CloudShell
1
# machine-type at least need 8 GB memory
2
gcloud beta container clusters create ${GKE_CLUSTER_NAME} \
3
    --zone ${ZONE} \
4
    --cluster-version "latest" \
5
    --machine-type "n2-standard-2" \
6
    --disk-type "pd-standard" \
7
    --disk-size "100" \
8
    --scopes "https://www.googleapis.com/auth/compute","https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" \
9
    --num-nodes "1" \
10
    --enable-ip-alias \
11
    --network ${NETWORK_NAME} \
12
    --subnetwork ${SUBNET_NAME} \
13
    --max-nodes-per-pool "110" \
14
    --enable-master-authorized-networks \
15
    --addons HorizontalPodAutoscaling,HttpLoadBalancing \
16
    --enable-autoupgrade \
17
    --enable-autorepair \
18
    --cluster-secondary-range-name my-pods \
19
    --services-secondary-range-name my-services \
20
    --enable-secret-manager \
21
    --workload-pool=${PROJECT_ID}.svc.id.goog \
22
    --enable-network-policy

取得 GKE cluster 憑證與連線資訊

GCP CloudShell
1
# connect to the GKE cluster and add create a new Kubernetes ServiceAccount
2
gcloud container clusters get-credentials --zone=${ZONE} ${GKE_CLUSTER_NAME}

確認所有系統服務均已正常啟動，特別是 csi-secrets-store 相關服務

GCP CloudShell
1
# check the kube-system running normal
2
kubectl get po -n kube-system
3
NAME                                                           READY   STATUS    RESTARTS   AGE
4
calico-node-t45r4                                              1/1     Running   0          20m
5
calico-node-vertical-autoscaler-85f58fc987-x7srd               1/1     Running   0          22m
6
calico-typha-8d48dbbcf-zs8cb                                   1/1     Running   0          20m
7
calico-typha-horizontal-autoscaler-5cd46b4fc7-9r74b            1/1     Running   0          22m
8
calico-typha-vertical-autoscaler-69cdfcdcf6-j4bzw              1/1     Running   0          22m
9
csi-secrets-store-gke-vg5fv                                    3/3     Running   0          21m
10
csi-secrets-store-provider-gke-w5c6g                           1/1     Running   0          21m
11
event-exporter-gke-766bc76558-lwzgd                            2/2     Running   0          23m
12
fluentbit-gke-bm7x5                                            3/3     Running   0          21m
13
gke-metadata-server-9cgs6                                      1/1     Running   0          21m
14
gke-metrics-agent-jqhnk                                        3/3     Running   0          21m
15
ip-masq-agent-d86hv                                            1/1     Running   0          21m
16
konnectivity-agent-544c896c46-st55p                            2/2     Running   0          23m
17
konnectivity-agent-autoscaler-67d4f7d5f-j9qfd                  1/1     Running   0          23m
18
kube-dns-69f7bb4564-fqj6t                                      5/5     Running   0          23m
19
kube-dns-autoscaler-79b96f5cb-v8kjd                            1/1     Running   0          23m
20
kube-proxy-gke-ricky-demo-cluster-default-pool-829d2ebc-s788   1/1     Running   0          20m
21
l7-default-backend-6484dd554-w48hr                             1/1     Running   0          22m
22
metrics-server-v0.7.1-6b8d6d8c46-knkwx                         2/2     Running   0          20m
23
netd-ml54w                                                     2/2     Running   0          21m
24
pdcsi-node-jqt4n                                               2/2     Running   0          21m

建立 Kubernetes namespace

GCP CloudShell
1
kubectl create ns ${SECRET_NS}

設定 service account

選擇 Kubernetes service account 或 IAM service account，以下兩種方法則一即可

使用 Kubernetes service account

建立 Kubernetes service account

GCP CloudShell
1
echo "apiVersion: v1
2
kind: ServiceAccount
3
metadata:
4
  name: ${K8S_SA_NAME}
5
  namespace: ${SECRET_NS}" |
6
tee sa.yaml | kubectl apply -f -

使用 IAM service account

修改環境變數後建立 IAM service account

GCP CloudShell
1
export IAM_SA_NAME="ricky-demo-secretmanager"
2
gcloud iam service-accounts create ${IAM_SA_NAME}

預期結果

GCP CloudShell
1
Created service account [ricky-demo-secretmanager].

設定 SecretManager

建立 Secret

GCP CloudShell
1
echo "top1-secret" > mysecret.txt
2
gcloud secrets create ${SECRET_NAME} --replication-policy=automatic --data-file=mysecret.txt

使用 GUI 操作方式如下

建立 Secret

create secret

輸入 secret value

enter secret value

賦予 secretAccessor 權限

依照前面設定，Kubernetes service account 與 GCP IAM service account 權限綁定二擇一

Kubernetes service account 綁定權限
GCP CloudShell
1
gcloud secrets add-iam-policy-binding ${SECRET_NAME} \
2
    --role=roles/secretmanager.secretAccessor \
3
    --member=principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${PROJECT_ID}.svc.id.goog/subject/ns/${SECRET_NS}/sa/${K8S_SA_NAME}

預期結果

GCP CloudShell
1
Updated IAM policy for secret [ricky-demo-secret].
2
bindings:
3
- members:
4
  - principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/demo/sa/ricky-demo
5
  role: roles/secretmanager.secretAccessor
6
etag: BwYd-TXpp6I=
7
version: 1
GCP IAM service account 綁定權限

先綁定 kubernetes service account 到前面建立的 GCP IAM service account

GCP CloudShell
1
# Allow "namespace/serviceAccount" to act as the new service account
2
gcloud iam service-accounts add-iam-policy-binding \
3
    --role roles/iam.workloadIdentityUser \
4
    --member "serviceAccount:${PROJECT_ID}.svc.id.goog[${SECRET_NS}/${K8S_SA_NAME}]" \
5
    ${IAM_SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com

預期結果

GCP CloudShell
1
Updated IAM policy for serviceAccount [ricky-demo-secretmanager@YOUR_PROJECT_ID.iam.gserviceaccount.com].
2
bindings:
3
- members:
4
  - serviceAccount:YOUR_PROJECT_ID.svc.id.goog[demo/ricky-demo]
5
  role: roles/iam.workloadIdentityUser
6
etag: BwYd-WoF1eQ=
7
version: 1

再將 GCP IAM service account 綁定 secret

GCP CloudShell
1
# bind IAM policy to access secret manager
2
gcloud secrets add-iam-policy-binding ${SECRET_NAME} \
3
    --role=roles/secretmanager.secretAccessor \
4
    --member=serviceAccount:${IAM_SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com

預期結果

GCP CloudShell
1
Updated IAM policy for secret [ricky-demo-secret].
2
bindings:
3
- members:
4
  - serviceAccount:ricky-demo-secretmanager@YOUR_PROJECT_ID.iam.gserviceaccount.com
5
  role: roles/secretmanager.secretAccessor
6
etag: BwYd-WrNntc=
7
version: 1

確認權限 第 4 步中，上方權限為直接綁定 Kubernetes service account，下方則為綁定 GCP IAM service account，兩者擇一即可 check secret permission

建立 SecretProviderClasses

GCP CloudShell
1
echo "apiVersion: secrets-store.csi.x-k8s.io/v1
2
kind: SecretProviderClass
3
metadata:
4
  name: demo-spc
5
  namespace: ${SECRET_NS}
6
spec:
7
  provider: gke
8
  parameters:
9
    secrets: |
10
      - resourceName: "projects/${PROJECT_ID}/secrets/${SECRET_NAME}/versions/1"
11
        path: "demo-spc.txt"" |
12
tee app-secrets.yaml | kubectl apply -f -

預期結果

GCP CloudShell
1
secretproviderclass.secrets-store.csi.x-k8s.io/demo-spc created

確認 SecretProviderClasses 是否建立成功

GCP CloudShell
1
# check SecretProviderClasses
2
kubectl get SecretProviderClasses -n ${SECRET_NS}
3
NAME       AGE
4
demo-spc   10s

測試

建立一個 pod 掛載該 Secret 進行測試確認

GCP CloudShell
1
echo "apiVersion: v1
2
kind: Pod
3
metadata:
4
  name: demo-pod
5
  namespace: ${SECRET_NS}
6
spec:
7
  serviceAccountName: ${K8S_SA_NAME}
8
  containers:
9
  - image: nginx
10
    imagePullPolicy: IfNotPresent
11
    name: demo
12
    stdin: true
13
    stdinOnce: true
14
    terminationMessagePath: /dev/termination-log
15
    terminationMessagePolicy: File
16
    tty: true
17
    volumeMounts:
18
      - mountPath: "/var/secrets"
19
        name: mysecret
20
  volumes:
21
  - name: mysecret
22
    csi:
23
      driver: secrets-store-gke.csi.k8s.io
24
      readOnly: true
25
      volumeAttributes:
26
        secretProviderClass: demo-spc" |
27
tee app.yaml | kubectl apply -f -

進入 pod 中確認 secret 是否為正確 value

GCP CloudShell
1
kubectl -n ${SECRET_NS} exec -it demo-pod -- bash
2
cat /var/secrets/demo-spc.txt

資源清除

GCP CloudShell
1
gcloud container clusters delete --zone=${ZONE} ${GKE_CLUSTER_NAME} --quiet
2
gcloud secrets delete ${SECRET_NAME} --quiet
3
# if used iam service-accounts
4
gcloud iam service-accounts remove-iam-policy-binding \
5
    --role roles/iam.workloadIdentityUser \
6
    --member "serviceAccount:${PROJECT_ID}.svc.id.goog[${SECRET_NS}/${K8S_SA_NAME}]" \
7
    ${IAM_SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com
8
gcloud iam service-accounts delete ${IAM_SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com --quiet
9
gcloud compute networks delete ${NETWORK_NAME} --quiet
10
gcloud compute firewall-rules delete allow-ssh --quiet
11
gcloud compute networks subnets delete ${SUBNET_NAME} --region=${REGION} --quiet
12
gcloud compute networks delete ${NETWORK_NAME} --quiet

FAQ

以下為常見問題情境

SecretProviderClasses 無建立完成

使用 kubectl -n demo describe po demo-pod 確認狀態會出現以下錯誤

GCP CloudShell
1
Events:
2
  Type     Reason       Age                From               Message
3
  ----     ------       ----               ----               -------
4
  Normal   Scheduled    80s                default-scheduler  Successfully assigned demo/demo-pod to gke-ricky-demo-cluster-default-pool-829d2ebc-s788
5
  Warning  FailedMount  48s (x7 over 80s)  kubelet            MountVolume.SetUp failed for volume "mysecret" : rpc error: code = Unknown desc = failed to get secretproviderclass demo/demo-spc, error: SecretProviderClass.secrets-store.csi.x-k8s.io "demo-spc" not found
6
  Normal   Pulled       16s                kubelet            Container image "nginx" already present on machine
7
  Normal   Created      16s                kubelet            Created container demo
8
  Normal   Started      16s                kubelet            Started container demo

K8S service account 無權限存取權

無存取 GCP Secret Manager 中 Secret 權現時，使用 kubectl -n demo describe po demo-pod 確認狀態會出現以下錯誤

GCP CloudShell
1
Events:
2
  Type     Reason       Age              From               Message
3
  ----     ------       ----             ----               -------
4
  Normal   Scheduled    5s               default-scheduler  Successfully assigned demo/demo-pod to gke-ricky-demo-cluster-default-pool-829d2ebc-s788
5
  Warning  FailedMount  3s (x3 over 5s)  kubelet            MountVolume.SetUp failed for volume "mysecret" : rpc error: code = Internal desc = failed to mount secrets store objects for pod demo/demo-pod, err: rpc error: code = Internal desc = rpc error: code = PermissionDenied desc = Permission 'secretmanager.versions.access' denied for resource 'projects/YOUR_PROJECT_ID/secrets/ricky-demo-secret/versions/1' (or it may not exist).

K8S node 資源不足

使用 kubectl -n demo describe po demo-pod 確認狀態會出現以下錯誤

GCP CloudShell
1
Events:
2
Type     Reason       Age               From               Message
3
----     ------       ----              ----               -------
4
Normal   Scheduled    22s               default-scheduler  Successfully assigned demo/demo-pod to gke-ricky-demo-cluster-default-pool-fe62ce24-ns0z
5
Warning  FailedMount  7s (x6 over 22s)  kubelet            MountVolume.SetUp failed for volume "mysecret" : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name secrets-store-gke.csi.k8s.io not found in the list of registered CSI drivers
  • 可發現 csi-secrets-store 服務無法正常啟用
GCP CloudShell
1
kubectl get po -n kube-system | grep secret
2
csi-secrets-store-gke-qzx55                                    0/3     Pending            0             6m31s
3
csi-secrets-store-provider-gke-xq9z6                           0/1     Pending            0             6m38s

參考資料