Comprehensive GKE Metrics & Kubernetes Alerting Catalog

This catalog compiles public Google Kubernetes Engine (GKE) telemetry metrics from the Google Cloud Managed Service for Prometheus Documentation, the GKE Observability Documentation, and the Google Cloud Monitoring Pricing Documentation, along with critical alerting rules from the open-source Awesome Prometheus Alerts for Kubernetes catalog.

It serves as the reference catalog for the gke-alert-configuration skill when generating evaluation suites and production alerting policies in Google Cloud Managed Service for Prometheus and Cloud Monitoring.

Table of Contents

• Telemetry Decision Hierarchy and KSM Cost Guardrails (~line 61)
  • Classification Tiers and Cost Optimization (~line 84)
  • Tier 2 Allowlist Template (PodMonitoring) (~line 92)
• GKE Public Metrics Catalog (kubernetes.io/) (~line 119)
  • Container and Pod Core Metrics (~line 123)
  • Node and Infrastructure Core Metrics (~line 143)
  • Autoscaling and Hardware Accelerator Metrics (~line 157)
• Open Source Kubernetes Alerts Catalog (awesome-prometheus-alerts) (~line 169)
  • Node Health and Capacity Alerts (~line 173)
  • Workload and Container Reliability Alerts (~line 184)
  • Jobs, CronJobs, and Storage Alerts (~line 199)
  • API Server and Control Plane Alerts (~line 210)
• Non-KSM Native Alternative Implementations (~line 221)
  • Container Memory Saturation and OOM Risk (~line 227)
  • Pod CrashLoopBackOff and Restarts (~line 257)
  • Node Readiness and Health (~line 277)
  • Multi-Window Multi-Burn-Rate SLO Error Rate Alerts (~line 287)
  • Traffic Outage Detection (absent() / default 0) (~line 310)

Telemetry Decision Hierarchy and KSM Cost Guardrails

[!IMPORTANT] Cost Warning on kube-state-metrics (KSM): In Google Cloud Managed Service for Prometheus, billing is based on metric samples ingested. Deploying open-source kube-state-metrics unmodified scrapes state for every object across the cluster, generating high time-series volume and significant ingestion cost.

Mandatory Agent Guardrail: When a user requests an alert that requires kube-state-metrics, you must alert the user that the policy requires KSM, ask for explicit permission before assuming or enabling KSM ingestion, and prefer Non-KSM Native Alternatives where possible.

flowchart TD
    A[User Request for Alert Policy] --> B{Can alert be served via Native GKE or cAdvisor?}
    B -->|Yes - Tier 1| C[Use Native GKE / cAdvisor Metric with Zero KSM Surcharge]
    B -->|No - Requires KSM| D[Alert User on KSM Cost and Ask Permission]
    D --> E{User Approved KSM?}
    E -->|No| F[Fall back to closest Native API / cAdvisor Approximation]
    E -->|Yes - Tier 2| G[Generate KSM Alert Policy and Recommend Filtered PodMonitoring Allowlist]

Classification Tiers and Cost Optimization

• Tier 1: Native GKE Built-ins / cAdvisor (Zero KSM Cost): Metrics collected natively by GKE or cAdvisor without kube-state-metrics.
• Tier 2: Filtered KSM (Requires Consent & Allowlist): Rules requiring Kubernetes control-plane state (such as Deployment spec versus ready replicas) that require kube-state-metrics.

Tier 2 Allowlist Template (PodMonitoring with metricRelabeling)

When KSM is required and authorized, always recommend configuring a filtered PodMonitoring resource to scrape only the required metrics and drop all others:

apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
  name: kube-state-metrics-filtered
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: kube-state-metrics
  endpoints:
  - port: http-metrics
    interval: 30s
    metricRelabeling:
    - action: keep
      sourceLabels: [__name__]
      regex: "^(kube_deployment_.*|kube_statefulset_.*|kube_daemonset_.*|kube_cronjob_.*|kube_job_.*|kube_pod_status_phase|kube_pod_info|kube_pod_container_status_.*|kube_persistentvolume_status_phase)$"

GKE Public Metrics Catalog (kubernetes.io/)

Below are the public native GKE telemetry metrics available in Cloud Monitoring.

Container and Pod Core Metrics

Node and Infrastructure Core Metrics

Autoscaling and Hardware Accelerator Metrics

Open Source Kubernetes Alerts Catalog (awesome-prometheus-alerts)

Compiled from the open-source community catalog.

Node Health and Capacity Alerts

Workload and Container Reliability Alerts

Jobs, CronJobs, and Storage Alerts

API Server and Control Plane Alerts

Non-KSM Native Alternative Implementations

When a user requests open-source catalog alerts that normally rely on kube-state-metrics, the skill must first offer these Tier 1 (Non-KSM) equivalents to avoid customer ingestion costs:

Container Memory Saturation and OOM Risk (Primary Saturation Alert)

Instead of container_memory_working_set_bytes / kube_pod_container_resource_limits, use native cAdvisor limits:

sum(
  container_memory_working_set_bytes{
    cluster="${var.cluster_name}",
    namespace="${var.namespace}",
    container!=""
  }
) by (pod, container)
/
sum(
  container_spec_memory_limit_bytes{
    cluster="${var.cluster_name}",
    namespace="${var.namespace}",
    container!=""
  }
) by (pod, container) > 0.90

[!NOTE] Memory versus CPU Saturation: In Kubernetes, CPU is compressible and throttled by the CFS scheduler without killing containers. Memory is uncompressible and triggers fatal OOMKilled events. Therefore, cluster saturation alerting focuses on Memory Saturation, and CPU saturation alerts (and container_cpu_usage_seconds_total) are excluded from the recommended cluster alert suite.

Pod CrashLoopBackOff and Restarts (Non-KSM Alternative)

Use native GKE Cloud Monitoring metric kubernetes.io/container/restart_count or cAdvisor restart counter:

sum(
  increase(
    kube_pod_container_status_restarts_total{
      cluster="${var.cluster_name}",
      namespace="${var.namespace}"
    }[15m]
  )
) by (pod, container) > 3

• Duration Rule: Set Terraform duration = "0s" (or "60s"). The [15m] lookback window already smooths transient spikes; adding duration = "300s" delays critical crashloop alerts by 20 minutes total.

Node Readiness and Health (Non-KSM Alternative)

kube_node_status_condition{
  cluster="${var.cluster_name}",
  condition="Ready",
  status="true"
} == 0

Multi-Window Multi-Burn-Rate (MWMBR) SLO Error Rate Alerts

For service error rate monitoring, use multi-window burn rate alerts instead of simple static ratios:

(
  (
    sum(rate(http_requests_total{cluster="${var.cluster_name}", namespace="${var.namespace}", status=~"5.."}[5m])) by (service)
    /
    sum(rate(http_requests_total{cluster="${var.cluster_name}", namespace="${var.namespace}"}[5m])) by (service)
  ) > (1 - 0.99) * 14.4
)
and
(
  (
    sum(rate(http_requests_total{cluster="${var.cluster_name}", namespace="${var.namespace}", status=~"5.."}[1h])) by (service)
    /
    sum(rate(http_requests_total{cluster="${var.cluster_name}", namespace="${var.namespace}"}[1h])) by (service)
  ) > (1 - 0.99) * 14.4
)

Traffic Outage Detection (absent() / default 0)

Avoid brittle == 0 checks. Use default 0 or absent() to detect total traffic loss:

(sum(rate(http_requests_total{cluster="${var.cluster_name}", namespace="${var.namespace}"}[5m])) by (service) default 0) == 0
and
(sum(rate(http_requests_total{cluster="${var.cluster_name}", namespace="${var.namespace}"}[5m] offset 1w)) by (service)) > 1