Skip to content

🐛 Sync kubeconfig secret changes in ClusterCache and refresh rotated bearer tokens in place - #13956

Open
miltalex wants to merge 1 commit into
kubernetes-sigs:mainfrom
miltalex:feat/clustercache
Open

🐛 Sync kubeconfig secret changes in ClusterCache and refresh rotated bearer tokens in place#13956
miltalex wants to merge 1 commit into
kubernetes-sigs:mainfrom
miltalex:feat/clustercache

Conversation

@miltalex

@miltalex miltalex commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

Today the ClusterCache only picks up a changed kubeconfig Secret after the health
probe fails (consecutive failures or unauthorized). This has two problems:

  1. If the old credentials keep working (e.g. a relay proxy address change where
    existing connections survive but new connections fail), the change is never
    picked up. Consumers of GetRESTConfig (e.g. the KCP etcd client) keep getting
    a stale server address (ClusterCache doesn't pick latest kubeconfig secret proactively #12399).
  2. If credentials are rotated frequently (e.g. the EKS token in the CAPA kubeconfig
    Secret rotates every ~15m), the connection is torn down and all informers for the
    workload cluster are recreated on every rotation, with "connection to the workload
    cluster is down" error spam in between (MachineHealthcheck controller fails to get cluster connection from cache #12363).

This PR makes the ClusterCache sync the kubeconfig Secret on every reconcile of a
connected cluster, before the health probe runs:

  • The connection tracks the kubeconfig Secret's resourceVersion and raw kubeconfig,
    so an unchanged Secret is a cheap no-op (single read via the (ideally caching)
    SecretClient, per-field comparison only when the resourceVersion changed). No watch
    on Secrets is added.
  • If only the bearer token of the current AuthInfo changed, the token is refreshed
    in place: connections use a dynamic bearer token RoundTripper that reads the
    current token from a shared holder on every request, so the existing client, cache
    and informers keep working across rotations. GetRESTConfig consumers get a copied
    rest.Config with the new token (previously handed out configs also pick up the new
    token). Request-specific Authorization headers are preserved. This only applies to
    static bearer tokens; token file / exec plugin credentials are already refreshed by
    client-go.
  • For any other effective change (server, CA, certs, auth method, ...), the accessor
    is disconnected and the reconcile requeues immediately to reconnect with the new
    kubeconfig.
  • Irrelevant changes are ignored: other Secret keys, tokens of unused AuthInfos, and,
    when the controller runs on the workload cluster, server/CA changes (those fields
    are replaced with in-cluster values anyway; runningOnCluster is captured at connect
    time and not re-checked).
  • Errors reading or parsing the Secret never tear down a working connection; the
    health probe remains the fallback.
  • Adds a capi_cluster_cache_kubeconfig_sync_total metric (results: token_refreshed,
    reconnect), cleaned up on Cluster deletion like the existing metrics.

I followed the approach discussed in
#12363 (comment)
and supersedes #12400.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #12399
Fixes #12363

@kubernetes-prow kubernetes-prow Bot added the do-not-merge/needs-area PR is missing an area label label Jul 22, 2026
@kubernetes-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign chrischdi for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubernetes-prow

Copy link
Copy Markdown
Contributor

This PR is currently missing an area label, which is used to identify the modified component when generating release notes.

Area labels can be added by org members by writing /area ${COMPONENT} in a comment

Please see the labels list for possible areas.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kubernetes-prow kubernetes-prow Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Jul 22, 2026
@kubernetes-prow
kubernetes-prow Bot requested a review from vincepri July 22, 2026 06:34
@kubernetes-prow kubernetes-prow Bot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jul 22, 2026
@miltalex
miltalex force-pushed the feat/clustercache branch from 45f6cb5 to 8e55866 Compare July 22, 2026 07:36
@kubernetes-prow kubernetes-prow Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 23, 2026
Signed-off-by: Miltiadis Alexis <alexmiltiadis@gmail.com>
@miltalex
miltalex force-pushed the feat/clustercache branch from 8e55866 to 981fad9 Compare July 24, 2026 08:20
@kubernetes-prow kubernetes-prow Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-area PR is missing an area label size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ClusterCache doesn't pick latest kubeconfig secret proactively MachineHealthcheck controller fails to get cluster connection from cache

1 participant