🐛 Sync kubeconfig secret changes in ClusterCache and refresh rotated bearer tokens in place - #13956
🐛 Sync kubeconfig secret changes in ClusterCache and refresh rotated bearer tokens in place#13956miltalex wants to merge 1 commit into
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
This PR is currently missing an area label, which is used to identify the modified component when generating release notes. Area labels can be added by org members by writing Please see the labels list for possible areas. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
45f6cb5 to
8e55866
Compare
Signed-off-by: Miltiadis Alexis <alexmiltiadis@gmail.com>
8e55866 to
981fad9
Compare
What this PR does / why we need it:
Today the ClusterCache only picks up a changed kubeconfig Secret after the health
probe fails (consecutive failures or unauthorized). This has two problems:
existing connections survive but new connections fail), the change is never
picked up. Consumers of
GetRESTConfig(e.g. the KCP etcd client) keep gettinga stale server address (ClusterCache doesn't pick latest kubeconfig secret proactively #12399).
Secret rotates every ~15m), the connection is torn down and all informers for the
workload cluster are recreated on every rotation, with "connection to the workload
cluster is down" error spam in between (MachineHealthcheck controller fails to get cluster connection from cache #12363).
This PR makes the ClusterCache sync the kubeconfig Secret on every reconcile of a
connected cluster, before the health probe runs:
resourceVersionand raw kubeconfig,so an unchanged Secret is a cheap no-op (single read via the (ideally caching)
SecretClient, per-field comparison only when the resourceVersion changed). No watch
on Secrets is added.
in place: connections use a dynamic bearer token RoundTripper that reads the
current token from a shared holder on every request, so the existing client, cache
and informers keep working across rotations.
GetRESTConfigconsumers get a copiedrest.Configwith the new token (previously handed out configs also pick up the newtoken). Request-specific Authorization headers are preserved. This only applies to
static bearer tokens; token file / exec plugin credentials are already refreshed by
client-go.
is disconnected and the reconcile requeues immediately to reconnect with the new
kubeconfig.
when the controller runs on the workload cluster, server/CA changes (those fields
are replaced with in-cluster values anyway; runningOnCluster is captured at connect
time and not re-checked).
health probe remains the fallback.
capi_cluster_cache_kubeconfig_sync_totalmetric (results:token_refreshed,reconnect), cleaned up on Cluster deletion like the existing metrics.I followed the approach discussed in
#12363 (comment)
and supersedes #12400.
Which issue(s) this PR fixes (optional, in
fixes #<issue number>(, fixes #<issue_number>, ...)format, will close the issue(s) when PR gets merged):Fixes #12399
Fixes #12363