GCP Best Practices & Checklist
A comprehensive checklist and best practices guide for securing your GCP environment against AI agent mishaps. Use this as your go-to reference when setting up or auditing guardrails.
Complete GCP Guardrails Checklist
Work through this checklist for every GCP environment where AI agents operate:
IAM & Identity
| Check | Action | Priority |
|---|---|---|
| Dedicated service account | Create a separate SA for each AI agent | Critical |
| Custom roles only | Never assign primitive roles (Editor, Owner) to agents | Critical |
| No delete permissions | Exclude all *.delete permissions from agent roles | Critical |
| IAM Deny Policies | Create deny policies blocking destructive operations | Critical |
| Workload Identity Federation | Use WIF instead of SA key files for external agents | High |
| IAM Conditions | Apply time-based and resource-based conditions | High |
| No SA key creation | Enable iam.disableServiceAccountKeyCreation org policy | High |
| Regular IAM audit | Review agent permissions quarterly | Medium |
Organization Policies
| Check | Action | Priority |
|---|---|---|
| Prevent project deletion | Custom constraint blocking project DELETE method | Critical |
| Prevent critical resource deletion | Custom constraints for Compute, Cloud SQL, GKE | Critical |
| Tag-based protection | Tag critical resources and apply conditional constraints | High |
| Restrict service usage | Limit which GCP services agents can access | High |
| Block SA key creation | Enforce iam.disableServiceAccountKeyCreation | High |
| No org policy modification | Ensure agents cannot set or modify org policies | Critical |
Resource Protection
| Check | Action | Priority |
|---|---|---|
| Project Liens | Apply liens to all production projects | Critical |
| Compute deletion protection | Enable on all production instances | Critical |
| Cloud SQL deletion protection | Enable on all production databases | Critical |
| GKE deletion protection | Enable on all production clusters | Critical |
| GCS versioning | Enable on all critical buckets | Critical |
| GCS retention policies | Set retention period on critical buckets | High |
| Terraform prevent_destroy | Add lifecycle rule to all critical resources | High |
Monitoring & Alerting
| Check | Action | Priority |
|---|---|---|
| Audit log monitoring | Log-based metrics for delete operations by agents | Critical |
| Alert policies | Alerts for destructive operations (both blocked and successful) | Critical |
| Real-time notifications | Pub/Sub + Cloud Functions + Slack pipeline | High |
| Log sink to BigQuery | Long-term audit log storage for analysis | High |
| Permission denied alerts | Alert when agents hit permission boundaries | Medium |
Backup & Recovery
| Check | Action | Priority |
|---|---|---|
| Automated disk snapshots | Daily snapshot schedules with 30-day retention | Critical |
| Cloud SQL automated backups | Daily backups with PITR enabled | Critical |
| GCS versioning | Object versioning on all data buckets | Critical |
| Cross-region backups | Multi-region snapshot storage and SQL replicas | High |
| DR testing | Quarterly disaster recovery drills | High |
| Enforced retention | Backup vault with minimum enforced retention | High |
Project-Level Isolation Strategy
Isolate AI agent workloads in dedicated projects to contain the blast radius of any incident:
Organization
|
+-- Folder: Production (strict org policies, liens on all projects)
| +-- Project: prod-web-app (humans only, no agent access)
| +-- Project: prod-data-pipeline (humans only, no agent access)
| +-- Project: prod-databases (humans only, no agent access)
|
+-- Folder: Agent-Managed (agent-specific policies)
| +-- Project: agent-workspace (agent can create/read/update, no delete)
| +-- Project: agent-staging (agent can deploy here)
|
+-- Folder: Development (relaxed policies)
| +-- Project: dev-sandbox (agents have more freedom here)
|
+-- Folder: Shared Services (no agent access)
+-- Project: shared-networking
+-- Project: audit-logging
+-- Project: backup-storage
Resource Hierarchy Best Practices
- Environment separation: Use folders to separate production, staging, development, and agent-managed environments
- Policy inheritance: Set the strictest policies at the organization level and relax only where necessary at lower levels
- Tagging strategy: Tag all resources with
managed-by(agent vs human),environment(prod, staging, dev), andprotection-level(critical, standard) - Budget alerts: Set billing budgets on agent projects to prevent runaway cost from agent-created resources
- Network isolation: Use Shared VPC with strict firewall rules to prevent agent projects from accessing production networks
VPC Service Controls
VPC Service Controls create a security perimeter around GCP resources, preventing data exfiltration and unauthorized access even if IAM is misconfigured:
# Create an access policy gcloud access-context-manager policies create \ --organization=123456789 \ --title="AI Agent Security Policy" # Create a service perimeter for production resources gcloud access-context-manager perimeters create prod-perimeter \ --policy=POLICY_ID \ --title="Production Resource Perimeter" \ --resources="projects/111111111,projects/222222222" \ --restricted-services="storage.googleapis.com,bigquery.googleapis.com,sqladmin.googleapis.com" \ --access-levels="" # The agent project is NOT in the perimeter # So the agent cannot access production storage, BigQuery, or Cloud SQL # even if it somehow obtains IAM permissions
roles/owner on a production project, VPC Service Controls will block the API calls if the request originates from outside the perimeter. This provides defense independent of IAM.Assured Workloads
For highly regulated environments, Assured Workloads provides additional compliance guardrails:
- Data residency: Ensure all data stays in specific regions
- Personnel controls: Restrict who can access data (including Google support)
- Compliance frameworks: FedRAMP, HIPAA, PCI DSS, SOC 2
- Automatic org policies: Assured Workloads automatically applies organization policies that enforce compliance requirements
While Assured Workloads is primarily for compliance, the guardrails it enforces also benefit AI agent safety by restricting what operations are possible.
Emergency Response Procedures
When an AI agent causes an incident, follow this response playbook:
Isolate the Agent (0-5 minutes)
Immediately revoke the agent's service account IAM bindings. Disable the service account. This stops the agent from taking any further actions.
Emergency: Disable agent service account# Disable the service account immediately gcloud iam service-accounts disable \ ai-agent-worker@my-project.iam.gserviceaccount.com # Remove all IAM bindings for the service account gcloud projects remove-iam-policy-binding my-project \ --member="serviceAccount:ai-agent-worker@my-project.iam.gserviceaccount.com" \ --role="projects/my-project/roles/agentComputeNoDelete"
Assess the Damage (5-15 minutes)
Query Cloud Audit Logs to identify exactly what the agent did. List all resources that were modified or deleted. Determine the scope of the incident.
Initiate Recovery (15-60 minutes)
Begin restoring from backups. Prioritize by business impact: databases first, then VMs, then storage. Use the recovery procedures documented in the Backup & Recovery lesson.
Verify Recovery (1-4 hours)
Validate that all recovered resources are functional. Run application health checks. Verify data integrity. Confirm that all services are operational.
Post-Incident Review (Within 24 hours)
Conduct a blameless post-mortem. Identify the root cause. Document what guardrails failed and why. Update your guardrails configuration to prevent recurrence.
Frequently Asked Questions
Can Organization Policies completely prevent resource deletion?
Custom organization policy constraints can block DELETE method calls on specific resource types across your entire organization hierarchy. Combined with IAM Deny Policies and resource-level deletion protection, you can create a very robust defense. However, no single mechanism is foolproof - always use defense in depth with multiple layers.
What is the difference between IAM Deny Policies and Organization Policies?
IAM Deny Policies operate at the identity level: they block specific permissions for specific principals (like a service account). Organization Policies operate at the resource level: they constrain what operations can be performed on specific resource types regardless of who is making the request. Use both for maximum protection.
Should AI agents ever have delete permissions?
In general, no. AI agents should create, read, and update resources but not delete them. If deletion is occasionally needed (for example, cleaning up temporary resources), implement a human-in-the-loop approval workflow: the agent requests deletion, a human approves it, and a separate process with elevated permissions performs the actual deletion.
How do I handle Terraform destroy with AI agents?
Use prevent_destroy = true in Terraform lifecycle blocks for all critical resources. Additionally, restrict the agent's Terraform backend permissions so it cannot modify state files. Consider using Terraform Cloud or Atlantis with a mandatory approval workflow for any plan that includes resource destruction.
What happens if an agent deletes a project?
GCP has a 30-day grace period for project deletion. During this window, you can restore the project with gcloud projects undelete. After 30 days, the project and all resources are permanently destroyed. This is why Project Liens are so important - they prevent the deletion request in the first place.
How much do these guardrails cost?
Most guardrails are free: IAM policies, Organization Policies, Project Liens, deletion protection flags, and Admin Activity audit logs all cost nothing. The primary costs come from Data Access audit log storage, backup storage (disk snapshots, Cloud SQL backups), and the BigQuery storage for audit log exports. Budget $50-200/month for a typical setup, less for small environments.
Can an agent bypass all these guardrails?
If properly configured, an agent with a custom service account cannot bypass IAM Deny Policies, Organization Policies, deletion protection, and Project Liens simultaneously. The agent would need roles/owner or roles/resourcemanager.organizationAdmin to modify these controls, which should never be granted to an agent. The defense-in-depth approach makes bypass practically impossible for a properly restricted agent.
How do I audit my current guardrails setup?
Use the Policy Analyzer in the Google Cloud console to review effective IAM policies. Run gcloud org-policies list to check organization constraints. Use gcloud iam policies list --kind=denypolicies to review deny policies. Check deletion protection flags on each resource. This course's checklist (above) can serve as your audit template.
Ready to Go Deeper?
Live instructor-led courses from our partners. Affiliate disclosure.
AI & ML Courses - 30% Off
Live instructor-led AI, machine learning, data science, and cloud courses for working professionals. Use code Limited30 at checkout.
EdurekaDataCamp - AI & Data Science
Hands-on Python, machine learning, and AI courses with interactive exercises and real projects.
DataCampedX - Top AI Courses
University-level AI courses from MIT, Harvard, Stanford. Earn certificates that employers recognize.
edX