Enhancing Cloud Run Security with Binary Authorization in GCP multi Project

Note

This post translated By AI and the original content was written in Traditional Chinese.

Scenario

Distribute different roles across multiple projects with their own management or usage permissions.

The roles can be divided into 4 project roles as follows:

• KMS: Store keys, usually managed by the security unit
• attestor: Place attestors created by KMS public key, usually managed by the security unit
• deployer: Create and use attestator policy and deploy workload
• attestation: Perform signing and store images

Permission Mapping Table

• user is the user account
• ooo-sa is the service account used by systems or services

Project	Account	Permission
kms (CloudKMS key)	[email protected]	Cloud KMS CryptoKey Public Key Viewer
kms (CloudKMS key)	[email protected]	Cloud KMS CryptoKey Signer Cloud KMS CryptoKey Public Key Viewer
attestor (attestor)	[email protected]	Binary Authorization Attestor Image Verifier
attestor (attestor)	[email protected]	Binary Authorization Attestor Viewer
attestor (attestor)	[email protected]	Binary Authorization Attestor Viewer
attestor (IAM)	[email protected]	Container Analysis Notes Attacher Container Analysis Occurrences for Notes Viewer
attestation (Artifact)	[email protected]	Artifact Registry Reader

Project API Mapping Table

Project	API
kms	cloudkms.googleapis.com
attestor	containeranalysis.googleapis.com binaryauthorization.googleapis.com
attestation	containeranalysis.googleapis.com binaryauthorization.googleapis.com cloudkms.googleapis.com artifactregistry.googleapis.com
deployer	container.googleapis.com artifactregistry.googleapis.com binaryauthorization.googleapis.com

Steps

Configure KMS Project

The preliminary steps are the same as setting up KMS for single-project Attestations However, the Key must provide read permission to the Attestor Project to import the public attestor, configured as follows

Key Permission Setting

Select the previously created key in KMS, then click PERMISSIONS and GRANT ACCESS

Provide Cloud KMS CryptoKey Public Key Viewer permission to the user account managing the attestor project

Configure Attestor Project

The steps are the same as setting up Attestor for a single project, just be mindful of the KMS Key permissions

Set Attestor Usage Permissions

First, go to the deployer project to get the gcp-sa-binaryauthorization.iam.gserviceaccount.com information

1. Enter IAM
2. Click PERMISSIONS
3. Check Include Google-provided role grants
4. Click VIEW BY PRINCIPALS
5. Enter CloudRun Service Agent to filter
6. Copy this service account for use in the attestor project

Return to the attestor project to set permissions

1. Open Binary Authorization
2. Click ATTESTORS
3. Check the previously created Attestor
4. Confirm PANEL is open
5. Add PRINCIPAL

Provide the deployer binary authorization service account with Binary Authorization Attestor Image Verifier permission

1. Paste the deployer binary authorization service account
2. Select Binary Authorization Attestor Image Verifier permission

Provide the deployer project user with Binary Authorization Attestor Viewer permission to add this attestor

1. Enter the deployer project user account
2. Provide Binary Authorization Attestor Viewer permission

Configure Attestation Project

Enable Attestation API

You can use the following command in CloudShell to enable it, or use the previously mentioned GUI interface

CloudShell

gcloud services --project=${ATTESTATION_PROJECT_ID} \
    enable containeranalysis.googleapis.com \
    binaryauthorization.googleapis.com \
    cloudkms.googleapis.com

(Optional) Create Test Image

Create Artifact registry and upload nginx image

CloudShell

export PROJECT_ID=${DEVSHELL_PROJECT_ID}
gcloud artifacts repositories create attestation-demo \
--repository-format=docker \
--location=asia-east1
docker pull nginx
docker tag nginx:latest asia-east1-docker.pkg.dev/${PROJECT_ID}/attestation-demo/demo-nginx:latest
gcloud auth configure-docker asia-east1-docker.pkg.dev
docker push asia-east1-docker.pkg.dev/${PROJECT_ID}/attestation-demo/demo-nginx:latest

Authorize Deployer Project to Use Artifact Registry Container Image in Attestation Project

Copy the CloudRun Service Agent service account from the Deployer Project IAM

1. Enter IAM
2. Click PERMISSIONS
3. Check Include Google-provided role grants
4. Click VIEW BY PRINCIPALS
5. Enter CloudRun Service Agent to filter
6. Copy this service account for use in the Attestation project

Add the CloudRun Service Agent service account from the Deployer Project to the Artifact Registry in the attestation project and provide Artifact Registry Reader permission

1. Select Repositories
2. Check the repo
3. Display panel
4. Add permission

Add the CloudRun Service Agent service account and provide Artifact Registry Reader permission

Configure Deployer Project

Policy Setting

Use CloudShell to enable the API with the following command, or use the previously mentioned UI to enable the API

CloudShell

gcloud --project=${DEPLOYER_PROJECT_ID} \
  services enable\
  container.googleapis.com\
  artifactregistry.googleapis.com\
  binaryauthorization.googleapis.com

Only the project needs to select attestor, the rest is the same as setting up Binary Authorization for single-project Attestations

Policy Testing

Since no attestation has been performed on any image yet, deploying any CloudRun will result in a warning. You can refer to enabling Binary Authorization on a new CloudRun service for testing

Perform Attestation

Permission settings

• Set the following permissions for the attestation project user account in the attestor project
  • Specify that the attestor has Binary Authorization Attestor Viewer
  • Add the following permissions to IAM
    • Container Analysis Notes Attacher
    • Container Analysis Occurrences for Notes Viewer
• Add the following permissions to the Cloud KMS key in the kms project for the attestation project user account
  • Cloud KMS CryptoKey Signer
  • Cloud KMS CryptoKey Public Key Viewer

After configuration is complete, you can refer to using Attestations for a single Project to use Attestation, and perform operations in the attestation project

Troubleshooting

Insufficient Attestor Permission

ERROR: (gcloud.beta.container.binauthz.attestations.sign-and-create) PERMISSION_DENIED: permission binaryauthorization.attestors.get denied on resource projects/ATTESTOR_PROJECT_ID/attestors/demo-attstor

Solution

Confirm that the attestation project user account has the Binary Authorization Attestor Viewer permission set for the specified attestor in the attestor project

Insufficient Cloud KMS Key Permission

ERROR: (gcloud.beta.container.binauthz.attestations.sign-and-create) PERMISSION_DENIED: Permission 'cloudkms.cryptoKeyVersions.viewPublicKey' denied on resource 'projects/KMS_PROJECT_ID/locations/global/keyRings/image-keys/cryptoKeys/binary-auth-key/cryptoKeyVersions/1' (or it may not exist).

Solution

Confirm that the Cloud KMS key in the kms project has added the attestation project user account permission and granted the following permissions

• Cloud KMS CryptoKey Signer
• Cloud KMS CryptoKey Public Key Viewer

Insufficient Container Analysis Notes Permission

ERROR: (gcloud.beta.container.binauthz.attestations.sign-and-create) User [user@ooo.com] does not have permission to access projects instance [ricky-attestations] (or it may not exist): permission "containeranalysis.notes.attachOccurrence" denied for project "ATTESTOR_PROJECT_ID", entity ID "demo-attstor-note"

Solution

Confirm that the attestation project user account has the following permissions added in the IAM of the attestor project

• Container Analysis Notes Attacher
• Container Analysis Occurrences for Notes Viewer

CloudRun Org Policy Setting

You can specify a particular Project / Folder / Org to forcibly enable Authorization Policies in CloudRun

Enable

This example uses a folder to enable, and all projects under that folder will enable Authorization Policies

CloudShell

gcloud resource-manager org-policies allow run.allowedBinaryAuthorizationPolicies default --folder=xxxxxxxxxxxxx

Expected result

constraint: constraints/run.allowedBinaryAuthorizationPolicies
etag: CM_3iKYGEOiJjTM=
listPolicy:
  allowedValues:
  - default
updateTime: '2023-07-27T09:41:35.107169Z'

Confirm

List folder status

CloudShell

gcloud resource-manager org-policies allow run.allowedBinaryAuthorizationPolicies default --folder=xxxxxxxxxxxxx

Expected result

constraint: constraints/run.allowedBinaryAuthorizationPolicies
etag: CM_3iKYGEOiJjTM=
listPolicy:
  allowedValues:
  - default
updateTime: '2023-07-27T09:41:35.107169Z'

Delete

Delete rule

CloudShell

gcloud resource-manager org-policies describe run.allowedBinaryAuthorizationPolicies --folder=xxxxxxxxxxxxx

Expected result

constraint: constraints/run.allowedBinaryAuthorizationPolicies
etag: BwVUSr8Q7Ng=