Skip to content

Dependent Kubernetes resources (serviceaccount, configmap) are not always recreated during a force refresh. #6559

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bobdoah opened this issue May 29, 2025 · 0 comments
Labels
bug Something isn't working

Comments

@bobdoah
Copy link

bobdoah commented May 29, 2025

Expected Behavior

Force refresh should recreate dependent resources (service accounts, configmaps) prior to deployment.

Current Behavior

When hitting the force refresh button, it's possible that dependent resources are not recreated, such as in the example described below. Pods from the deployment will fail to start.

STEP 1/2 — Force update
     Beginning force update
     Deleting kubernetes objects:
     → Deployment/my-app
     → ServiceAccount/my-service-account
Beginning garbage collecting Kubernetes objects
Deleting kubernetes objects:
→ Deployment/my-app
→ ServiceAccount/my-service-account

STEP 2/2 — Deploying
     Applying YAML to cluster
     Objects applied to cluster:
       → my-service-account:serviceaccount
       → my-app:deployment

     Step 1 - 0.08s (Force update)
     Step 2 - 0.14s (Deploying)
     DONE IN: 0.22s 

[event: replicaset my-app-6fdfcf9477] Error creating: pods "my-app-6fdfcf9477-" is forbidden: error looking up service account default/my-service-account: serviceaccount "my-service-account" not found
[event: replicaset my-app-6fdfcf9477] Error creating: pods "my-app-6fdfcf9477-" is forbidden: error looking up service account default/my-service-account: serviceaccount "my-service-account" not found
[event: replicaset my-app-6fdfcf9477] Error creating: pods "my-app-6fdfcf9477-" is forbidden: error looking up service account default/my-service-account: serviceaccount "my-service-account" not found

querying the cluster shows the serviceaccount has not been created:

% kubectl get sa
NAME      SECRETS   AGE
default   0         7d2h
% kubectl get deploy
NAME     READY   UP-TO-DATE   AVAILABLE   AGE
my-app   0/1     0            0           2m54s

Steps to Reproduce

I have put together a simple example reproduction here: https://github.com/bobdoah/tilt-delete-race-condition

Context

This seems to be related to this change to make deletion asynchronous/two-step. I built a version with this change reverted. Using that, I am unable to reproduce the issue.

tilt doctor Output

Tilt: v0.34.2, built 2025-04-11
System: linux-amd64

Docker

  • Host: unix:///var/run/docker.sock
  • Server Version: 28.0.4
  • API Version: 1.48
  • Builder: 2
  • Compose Version: v2.34.0

Kubernetes

  • Env: kind
  • Context: kind-tilt
  • Cluster Name: kind-tilt
  • Namespace: default
  • Container Runtime: containerd
  • Version: v1.29.2
  • Cluster Local Registry: &RegistryHosting{Host:localhost:46139,HostFromClusterNetwork:ctlptl-registry:5000,HostFromContainerRuntime:,Help:https://github.com/tilt-dev/ctlptl,SingleName:,}

Thanks for seeing the Tilt Doctor!
Please send the info above when filing bug reports. 💗

The info below helps us understand how you're using Tilt so we can improve,
but is not required to ask for help.

Analytics Settings
--> (These results reflect your personal opt in/out status and may be overridden by an analytics_settings call in your Tiltfile)

  • User Mode: opt-in
  • Machine: 5247e78a689a305f4392f2b8e5bdfe12
  • Repo: vPUwqCdPvBm6dJXroBYigg==

#### About Your Use Case
This is impacting our ability to use newer versions of Tilt, with this change in.  
<!--- How has this issue affected you? What are you trying to accomplish? -->
<!--- Providing context helps us come up with a solution that is most useful in the real world -->
@bobdoah bobdoah added the bug Something isn't working label May 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant