CloudWatch Metric Guide
cluster_failed_node_countAmazon EKS CloudWatch metric
cluster_failed_node_count reports the number of worker nodes in the cluster currently suffering from a Kubernetes node condition — per AWS documentation, a node is considered failed if it is suffering from any node condition (e.g. NotReady, MemoryPressure, DiskPressure, or PIDPressure).
What it measures
About cluster_failed_node_count
cluster_failed_node_count reports the number of worker nodes in the cluster currently suffering from a Kubernetes node condition — per AWS documentation, a node is considered failed if it is suffering from any node condition (e.g. NotReady, MemoryPressure, DiskPressure, or PIDPressure).
| Namespace | ContainerInsights |
| Metric name | cluster_failed_node_count |
| Unit | Count |
| AWS docs | Official Amazon EKS metrics reference |
Why this metric matters
cluster_failed_node_count > 0 is an emergency signal in the same category as EC2's StatusCheckFailed, not a gradual stress metric. A failed node condition means the node's kubelet has stopped reporting healthy status to the control plane — the node may be unreachable, out of resources, or crashed outright. Pods already scheduled to that node don't move immediately: by default Kubernetes waits out the pod-eviction-timeout (5 minutes) before rescheduling them elsewhere, so a failed node can mean real, sustained downtime for the workloads on it, not just a brief blip.
The failure also compounds at the cluster level. Losing a node reduces total cluster capacity right as the scheduler needs to place the displaced pods somewhere else — remaining healthy nodes absorb both their existing load and the failed node's pods simultaneously, which is exactly the kind of event that pushes node_cpu_utilization and pod_memory_utilization on the surviving nodes into alarm range right afterward. Catching cluster_failed_node_count at the first non-zero reading, rather than waiting for the downstream capacity alarms to fire, is what gives you time to act before the cascade.
Recommended alarm threshold for cluster_failed_node_count
Recommended threshold
> 0 — alarm immediately
Per AWS documentation, a node is counted as failed if it is suffering from any Kubernetes node condition — there is no partial or graduated state published for this metric, similar to EC2's StatusCheckFailed. Zero tolerance (Nuberio recommendation) is correct for production clusters: set a short evaluation period (1–2 datapoints) to minimize time-to-alert, since every minute a node sits failed is a minute its pods may be unreachable or awaiting the default 5-minute pod-eviction-timeout to be rescheduled.
Is your cluster_failed_node_count alarm already set up correctly?
The free Nuberio Audit scans your CloudWatch setup and flags missing or misconfigured alarms — including cluster_failed_node_count — in 5 minutes.
Common failures that show up in cluster_failed_node_count
When cluster_failed_node_count reaches an alarm threshold, these are the most common root causes — in order of how often Nuberio sees them across customer AWS accounts.
Underlying EC2 instance hardware or hypervisor failure taking the node's kubelet offline — the same failure class as EC2's StatusCheckFailed
Kubelet crash or resource exhaustion (DiskPressure from a full node disk) causing the node to stop reporting a Ready condition
CNI or networking failure on the node (e.g. ENI/IP exhaustion) preventing it from communicating with the control plane
Node group launch template, AMI, or bootstrap script misconfiguration causing new nodes to fail to join the cluster healthy
Spot Instance interruption reclaiming the underlying EC2 instance before the node has been fully drained
How the Nuberio Audit checks cluster_failed_node_count
The free Audit scans every Amazon EKS resource in your account and checks whether a cluster_failed_node_count alarm exists, whether its threshold and evaluation period are configured sensibly, and flags it if it's missing entirely. Every finding comes with the exact CLI command to fix it — read-only access, no credit card, results in about 5 minutes.
Nuberio Audit
Scans your CloudWatch setup for missing or misconfigured cluster_failed_node_count alarms. Free, 5-minute read-only scan.
Related Amazon EKS metrics
cluster_failed_node_count rarely fails in isolation. These metrics tend to correlate — monitor them together for complete Amazon EKS coverage.
FAQ
Frequently asked questions about cluster_failed_node_count
Common questions about setting up CloudWatch alarms for cluster_failed_node_count in Amazon EKS.
What is the recommended CloudWatch alarm threshold for cluster_failed_node_count?+
> 0 — alarm immediately. Per AWS documentation, a node is counted as failed if it is suffering from any Kubernetes node condition — there is no partial or graduated state published for this metric, similar to EC2's StatusCheckFailed. Zero tolerance (Nuberio recommendation) is correct for production clusters: set a short evaluation period (1–2 datapoints) to minimize time-to-alert, since every minute a node sits failed is a minute its pods may be unreachable or awaiting the default 5-minute pod-eviction-timeout to be rescheduled.
Which CloudWatch namespace does cluster_failed_node_count belong to?+
cluster_failed_node_count is published in the ContainerInsights namespace with a unit of Count. You can find it in the CloudWatch console under "Metrics" → "ContainerInsights". See the Amazon EKS CloudWatch metrics reference in the AWS documentation.
Does Nuberio automatically create CloudWatch alarms for cluster_failed_node_count?+
No — the free Nuberio Audit does not create alarms for you. It scans your CloudWatch setup and tells you which Amazon EKS resources are missing a cluster_failed_node_count alarm, then gives you the exact CLI command to create it yourself.
Can I use Nuberio without already having a cluster_failed_node_count alarm set up?+
Yes. The free Audit works whether or not you have any cluster_failed_node_count alarm configured — it reads the resource directly from CloudWatch and tells you which resources need one, plus the copy-paste AWS CLI command to create it.
This page is part of the CloudWatch metric guide — thresholds and debugging guidance for every metric across RDS, Lambda, ECS, ALB, EC2, DynamoDB, SQS, API Gateway, S3, CloudFront, ElastiCache, and EKS. To find which Amazon EKS alarms your account is missing — including cluster_failed_node_count — run the free CloudWatch alarm audit. The scan takes under 5 minutes and requires no account.