Cutting a cloud bill by 70% without a finance meeting

Most cloud bills are not high because the cloud is expensive. They are high because nobody can see who is spending what, so nobody feels the number. The fix is not a procurement process. It is three engineering moves: make cost visible per owner, then automate the two savings that are always there, then let the data decide the rest.

This is a generic worked example on Google Cloud with Kubernetes. The same three moves exist on every provider; only the resource names change.

reduction (example)

~70%

GPU nodes when idle

0

non-prod switched off

12 h / day

dashboard everyone reads

1

Make cost visible per owner

Turn on the billing export to BigQuery, and label everything you create with env and team. That second part is the whole game; an unlabelled resource is a cost nobody owns. Terraform makes it mechanical:

# every module receives the same labels and passes them to every resource
locals {
  labels = { env = var.env, team = var.team, managed_by = "terraform" }
}
resource "google_container_node_pool" "gpu" {
  # …
  node_config { labels = local.labels }
}

Then one query is the dashboard. Group by the labels, not by SKU, and net out credits so the number is what you actually pay:

SELECT
  DATE(usage_start_time)                                              AS day,
  (SELECT value FROM UNNEST(labels) WHERE key = 'env')                AS env,
  (SELECT value FROM UNNEST(labels) WHERE key = 'team')               AS team,
  ROUND(SUM(cost) + SUM(IFNULL((SELECT SUM(c.amount) FROM UNNEST(credits) c), 0)), 2) AS net_cost
FROM `billing.gcp_billing_export_resource_v1_XXXXXX`
WHERE usage_start_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
GROUP BY day, env, team
ORDER BY day DESC, net_cost DESC;

Put it on a page people open anyway. The first week it runs, two things become obvious in every organisation I have seen it in: GPU capacity that is paid for around the clock and used for a few hours, and non-production environments that cost the same at 3 a.m. as at 3 p.m.

GPU capacity that scales to zero

Inference and training jobs are bursty. The node pool that runs them should not exist when they are not running. On GKE that is an autoscaled pool with a minimum of zero, a taint so nothing else lands on it, and workloads that ask for it explicitly:

resource "google_container_node_pool" "gpu" {
  name       = "gpu-l4"
  cluster    = google_container_cluster.main.id
  location   = var.region
  autoscaling { min_node_count = 0  max_node_count = 4 }
  node_config {
    machine_type = "g2-standard-8"
    guest_accelerator { type = "nvidia-l4"  count = 1 }
    taint { key = "gpu"  value = "true"  effect = "NO_SCHEDULE" }
    labels = merge(local.labels, { pool = "gpu" })
  }
}

# the inference deployment: tolerate the taint, request the GPU, and nothing else fits here
spec:
  template:
    spec:
      nodeSelector: { cloud.google.com/gke-accelerator: nvidia-l4 }
      tolerations: [{ key: gpu, operator: Equal, value: "true", effect: NoSchedule }]
      containers:
        - name: inference
          resources: { limits: { nvidia.com/gpu: 1 } }

When the last GPU pod is deleted, the autoscaler removes the node after its cooldown, about ten minutes. When the next job is scheduled, it creates one. That creation is the price: expect three to six minutes from “pod pending” to “running” for a GPU node with driver install and a multi-gigabyte image. For batch work that is nothing. For an interactive endpoint it is a product decision, and the answer is usually one warm replica on a small pool plus scale-to-zero for the rest.

Environments that switch themselves off

Non-production environments are the second obvious saving: development and staging are used twelve hours a day, five days a week, and billed for a hundred and sixty-eight. Instead of a custom scaler, use an annotation-driven downscaler that already exists and is well understood:

# each non-prod namespace declares its own hours; the downscaler scales deployments to zero outside them
apiVersion: v1
kind: Namespace
metadata:
  name: staging
  annotations:
    downscaler/uptime: "Mon-Fri 08:00-20:00 America/Argentina/Buenos_Aires"

# somebody needs staging at 22:00? one annotation, expires on its own
kubectl annotate namespace staging downscaler/force-uptime="true" --overwrite
kubectl annotate namespace staging downscaler/force-uptime- # back to schedule

Wire the second command to a button in the team chat and nobody will ever complain about the schedule. Node pools follow the pods: when the deployments are at zero the autoscaler removes the nodes, and the environment costs its persistent disks and nothing else.

Let the data decide the rest

With the two automations running, the dashboard shows a new shape: a flat baseline and bursts. Now, and only now, committed-use discounts make sense. Commit to the baseline, never to the bursts:

-- the steady state you can safely commit to: the 10th percentile of daily vCPU-hours over 90 days
SELECT APPROX_QUANTILES(daily_vcpu_hours, 100)[OFFSET(10)] AS p10_vcpu_hours
FROM (
  SELECT DATE(usage_start_time) AS day, SUM(usage.amount_in_pricing_units) AS daily_vcpu_hours
  FROM `billing.gcp_billing_export_resource_v1_XXXXXX`
  WHERE sku.description LIKE '%Core running%' AND usage_start_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 90 DAY)
  GROUP BY day
);

What it costs

Cold starts are the honest price, and they are paid by whoever hits the environment first. Make that visible too: a “warming up” state in the app beats a timeout. The other cost is ownership. The dashboard, the downscaler and the labels each need a name next to them, or the savings quietly reverse over a year. The example number in the title came from exactly these three moves and nothing exotic; the exotic tools come later, if the data asks for them.