Teleport Kubernetes Health Checks

Report an issue with this page

This documentation provides an overview of health checks to Kubernetes clusters with Teleport. Teleport Kubernetes Services periodically check the connectivity and permissions of enrolled Kubernetes clusters, and is available in Teleport version 18.4 and later.

Why check the health of a Kubernetes cluster?

• Observability: Discover network and permission issues before users do. Unhealthy Kubernetes clusters are visible in a Teleport UI, command line tool, or Prometheus metrics.
• High Availability: Automatically route and distribute connections to healthy Kubernetes clusters in a high-availability configuration.

What's checked?

Kubernetes permissions and a Kubernetes health endpoint are checked to determine whether a Kubernetes cluster is both up and usable with Teleport.

Four Kubernetes RBAC permissions are routinely checked with the Kubernetes SelfSubjectAccessReview API. The permissions are part of minimum requirements for Teleport to work with a Kubernetes cluster. The checked permissions are:

• Impersonate users
• Impersonate groups
• Impersonate service accounts
• Get pods

If a permission can't be checked, the Kubernetes cluster's /readyz endpoint is called to further distinguish connection errors and Kubernetes component errors.

Health states

A Kubernetes cluster is in a healthy, unhealthy, or unknown state.

• healthy indicates a Kubernetes cluster's health was checked and is fine
• unhealthy indicates a Kubernetes cluster's health was checked and is not usable for some reason
• unknown indicates a Kubernetes cluster has been excluded from health checks, or the first health check is initializing

warning

unknown Kubernetes clusters may be unhealthy.

unknown Kubernetes clusters are not checked for health due to:

• Running a pre-18.4 version of Teleport Kubernetes Service
• Explicitly configuring health_check_config labels to exclude a Kubernetes cluster

Viewing health

Kubernetes cluster health is viewed through the Teleport web UI, desktop Connect UI, tctl CLI tool, or Prometheus metrics.

Teleport Web & Connect UI

Click the refresh icon to get the latest health of Kubernetes clusters.

Teleport tctl CLI

Run tctl get kube_server/<your-kube-server-name> for an overview of Kubernetes cluster health for a specific Kubernetes service.

kind: kube_server
...
status:
  target_health:
    address: 192.168.106.2:58458
    message: 1 health check passed
    protocol: http
    status: healthy
    transition_reason: threshold_reached
    transition_timestamp: "2025-10-13T19:26:58.842855Z"
version: v3

Teleport Prometheus metrics

Health check metrics offer a high-level view of Kubernetes cluster health. The total number of Kubernetes clusters in a healthy, unhealthy, or unknown state are monitored with gauge metrics teleport_resources_health_status_healthy, teleport_resources_health_status_unhealthy, and teleport_resources_health_status_unknown.

• teleport_resources_health_status_healthy{type="kubernetes"} is the total number of healthy Kubernetes clusters
• teleport_resources_health_status_unhealthy{type="kubernetes"} is the total number of unhealthy Kubernetes clusters
• teleport_resources_health_status_unknown{type="kubernetes"} is the total number of Kubernetes clusters in an unknown state

A PromQL expression may be used to determine the total number of Kubernetes clusters.

teleport_resources_health_status_healthy{type="kubernetes"} +
teleport_resources_health_status_unhealthy{type="kubernetes"} +
teleport_resources_health_status_unknown{type="kubernetes"}

A PromQL expression may be used to detect the presence of unhealthy Kubernetes clusters.

teleport_resources_health_status_unhealthy{type="kubernetes"} > 0

note

Notice that Prometheus metrics don't distinguish which cluster is unhealthy.

Use a Teleport UI, or tctl kube ls --query 'health.status == "unhealthy"' to determine which cluster is unhealthy.

The Teleport tctl top command displays Kubernetes health check metrics.

Steps to view health check metrics with tctl:

• Run tctl top
• Navigate to the "Raw Metrics" tab by pressing the right arrow → key several times
• Enter search mode by pressing the forward slash / key
• Type or paste teleport_resources_health_status

Health check metrics may also be viewed with the Teleport diagnostic endpoint http://<diagnostic-address>/metrics.

# HELP teleport_resources_health_status_healthy Number of healthy resources
# TYPE teleport_resources_health_status_healthy gauge
teleport_resources_health_status_healthy{type="kubernetes"} 99972
# HELP teleport_resources_health_status_unhealthy Number of unhealthy resources
# TYPE teleport_resources_health_status_unhealthy gauge
teleport_resources_health_status_unhealthy{type="kubernetes"} 3
# HELP teleport_resources_health_status_unknown Number of resources in an unknown health state
# TYPE teleport_resources_health_status_unknown gauge
teleport_resources_health_status_unknown{type="kubernetes"} 25

Configuring health checks

The Teleport tctl CLI tool enables reading, adding, editing, and deleting health_check_config resources.

health_check_config resources offer a way to configure and selectively apply health checks to Kubernetes clusters.

An example health_check_config.

kind: health_check_config
version: v1
metadata:
  name: example
  description: Example healthcheck configuration
spec:
  # interval is the time between each health check. Default 30s.
  interval: 30s
  # timeout is the health check connection establishment timeout. Default 5s.
  timeout: 5s
  # healthy_threshold is the number of consecutive passing health checks
  # after which a target's health status becomes "healthy". Default 2.
  healthy_threshold: 2
  # unhealthy_threshold is the number of consecutive failing health checks
  # after which a target's health status becomes "unhealthy". Default 1.
  unhealthy_threshold: 1
  # match is used to select Kubernetes clusters that apply these settings.
  # Kubernetes clusters are matched by label selectors and at least one label selector
  # must be set.
  # If multiple `health_check_config` resources match the same Kubernetes cluster,
  # the matching health check configs are sorted by name and only the first
  # config applies.
  match:
    # kubernetes_labels matches Kubernetes cluster labels. An empty value is ignored.
    # If kubernetes_labels_expression is also set, then the match result is the logical
    # AND of both.
    kubernetes_labels:
      - name: env
        values:
          - dev
          - staging
    # kubernetes_labels_expression is a label predicate expression to match Kubernetes clusters.
    # An empty value is ignored.
    # If kubernetes_labels is also set, then the match result is the logical AND of both.
    kubernetes_labels_expression: 'labels["owner"] == "platform-team"'

The default health_check_config enables all Kubernetes clusters to participate in health checks from version 18.4 onward.

kind: health_check_config
metadata:
  description: Enables all health checks by default
  labels:
    teleport.internal/resource-type: preset
  name: default
  namespace: default
spec:
  match:
    kubernetes_labels:
    - name: '*'
      values:
      - '*'
version: v1

Multiple different health_check_config resources may be created for different groups of Kubernetes clusters. When multiple health_check_config match the same Kubernetes cluster, configs are sorted in ascending order by name, and only the first config applies (e.g., the name "00-my-config" has greater precedence than "10-my-config").

tctl health check commands

Read the default health check config with tctl get:

$ tctl get health_check_config/default

Create a new health check config with tctl create:

$ tctl create health_check_config.yaml

Update an existing config interactively with tctl edit:

$ tctl edit health_check_config/default

Delete a health check config with tctl rm:

$ tctl rm health_check_config/example

Teleport Kubernetes Services are notified of changes to health_check_config, and reevaluate whether a Kubernetes cluster participates in health checks, applying any changes.

FAQ

How do I see unhealthy Kubernetes clusters?

The Teleport web and Connect UI highlight unhealthy Kubernetes clusters.

Clicking a highlighted Kubernetes cluster shows details of an unhealthy Kubernetes cluster.

It may take approximately 5m for a health change to be reported.

Click the circular arrow refresh icon to get the latest health status.

The Teleport tctl CLI tool also searches and displays unhealthy Kubernetes clusters.

tctl kube ls --query 'health.status == "unhealthy"' 

What guidance is there for troubleshooting unhealthy Kubernetes clusters?

See the Kubernetes Service troubleshooting guide for specific errors returned by health checks.

How do I disable Kubernetes health checks?

Add a filtering label to exclude Kubernetes clusters with the CLI command tctl edit health_check_config/default. If you have multiple health_check_config, such as health_check_config/example-a, health_check_config/example-b, each config would be adjusted.

Adding any label to health_check_config tells Teleport to filter out Kubernetes clusters which don't match the label.

For example, adding healthcheck and enabled to kubernetes_labels in the following example would exclude Kubernetes clusters without the label.

kind: health_check_config
metadata:
  description: Enables all health checks by default
  labels:
    teleport.internal/resource-type: preset
  name: default
  namespace: default
  revision: 040e3839-c248-4b9d-898f-4ae88b1d4752
spec:
  match:
    kubernetes_labels:
      - name: 'healthcheck'
        values:
          - 'enabled'
version: v1

The default is to match all enrolled Kubernetes clusters.

spec:
  match:
    kubernetes_labels:
      - name: '*'
        values:
          - '*'

Do health check metrics show which Kubernetes cluster is unhealthy?

No. A specific Kubernetes cluster's health cannot be determined from the teleport_resources_health_status_* health metrics.

The quantity of unhealthy Kubernetes clusters are available from metrics.

How do I configure health checks for high-availability?

No additional configuration is needed.

Health-based connection routing is automatic when multiple Teleport Kubernetes Services are proxying to the same Kubernetes cluster.

Configuring high-availability with multiple Teleport Kubernetes Services proxying the same Kubernetes cluster would be needed.