CloudOps Architect Benchmark

Prompt

You are the Senior DevOps / Platform Engineer at a high-throughput, low-latency AdTech company operating a real-time bidding platform. The engineering team has completed the application layer: approximately 20 containerized microservices written in Java, Go, and Python. The platform processes hundreds of thousands of requests per second and has strict latency requirements because bid responses must be returned within tens of milliseconds. The company has chosen Google Cloud Platform and Google Kubernetes Engine (GKE), but you are starting with an essentially empty GCP project and an empty Kubernetes cluster. Design a detailed, step-by-step plan for taking the environment from an empty cluster to a production-grade microservices platform. Your design should address: * GCP project and network architecture * VPCs, subnets, routing, NAT, private networking, and firewall rules * GKE cluster architecture and configuration * Node pools, machine types, autoscaling, and workload placement * Kubernetes namespaces and resource organization * IAM, Workload Identity, RBAC, and least-privilege access * Secrets and configuration management * Container registry and image security * Ingress, load balancing, DNS, TLS, and service exposure * Internal service-to-service communication * Horizontal and vertical scaling * Pod disruption budgets, topology spread, anti-affinity, and high availability * CPU/memory requests and limits * Health probes and graceful shutdown * Observability: metrics, logs, tracing, dashboards, and alerting * CI/CD and deployment strategy * Canary or progressive deployments * Infrastructure as Code * Disaster recovery and backup strategy * Multi-zone and potentially multi-region architecture * Security hardening * Cost optimization without compromising latency * Capacity planning for traffic spikes * Production readiness testing Assume: * Traffic can suddenly increase by 5–10× during major advertising events. * Some services are extremely latency-sensitive. * Some workloads consume Kafka/Pub/Sub-style event streams. * Redis is heavily used for low-latency state. * PostgreSQL is used for transactional workloads. * Deployments occur multiple times per day. * A bad deployment must be recoverable within minutes. * The platform must tolerate node and zone failures without customer-visible downtime. Do not simply list technologies. Explain the order in which you would implement the platform, why each architectural decision is being made, important trade-offs, and what you would verify before moving to the next stage. Where multiple valid approaches exist, compare them and state which one you would choose for this specific workload.