Skip to content

[k8s] unable to SSH into cluster on K8s that use ipv6 #5354

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
alex000kim opened this issue Apr 24, 2025 · 5 comments · Fixed by #5413
Closed

[k8s] unable to SSH into cluster on K8s that use ipv6 #5354

alex000kim opened this issue Apr 24, 2025 · 5 comments · Fixed by #5413

Comments

@alex000kim
Copy link
Contributor

  1. I launched a lupyterlab server:
resources:
  cloud: kubernetes
  cpus: 16

workdir: .

setup: |
  pip install jupyterlab
run: |
  jupyter lab --no-browser --port 8888
  1. When I am trying to do ssh port-forwarding, I get this:
$ ssh -L 8888:localhost:8888 alexkim-jupyter -v
OpenSSH_9.9p1, LibreSSL 3.3.6
debug1: Reading configuration data /Users/alexkim/.ssh/config
debug1: Reading configuration data /Users/alexkim/.sky/generated/ssh/alexkim-jupyter
debug1: /Users/alexkim/.sky/generated/ssh/alexkim-jupyter line 2: Applying options for alexkim-jupyter
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Executing proxy command: exec ssh -tt -i ~/.sky/generated/ssh-keys/alexkim-jupyter.key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -W 2600:1f16:cf3:2520:f32c::c:22 sky@127.0.0.1 -o ProxyCommand='/Users/alexkim/.pyenv/versions/3.10.13/bin/python3.10 /Users/alexkim/.pyenv/versions/3.10.13/lib/python3.10/site-packages/sky/templates/websocket_proxy.py http://127.0.0.1:46580/ alexkim-jupyter'
debug1: identity file /Users/alexkim/.sky/generated/ssh-keys/alexkim-jupyter.key type -1
debug1: identity file /Users/alexkim/.sky/generated/ssh-keys/alexkim-jupyter.key-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.9
Bad stdio forwarding specification '2600:1f16:cf3:2520:f32c::c:22'
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535
$ sky --version
skypilot, version 1.0.0.dev20250424
$ sky --commit
skypilot, commit 3d993746d92a6d4cc8802156663bce1a4bf15b92

a while back this PR fixed it #4497
but the issue is back again

@cg505
Copy link
Collaborator

cg505 commented Apr 24, 2025

What is the content of /Users/alexkim/.sky/generated/ssh/alexkim-jupyter?

@alex000kim
Copy link
Contributor Author

$ cat /Users/alexkim/.sky/generated/ssh/alexkim-jupyter
# Added by sky (use `sky stop/down alexkim-jupyter` to remove)
Host alexkim-jupyter
  HostName 2600:1f16:cf3:2522:853b::a
  User sky
  IdentityFile /Users/alexkim/.sky/generated/ssh-keys/alexkim-jupyter.key
  IdentitiesOnly yes
  StrictHostKeyChecking no
  UserKnownHostsFile=/dev/null
  GlobalKnownHostsFile=/dev/null
  Port 22
  ProxyCommand ssh -tt -i ~/.sky/generated/ssh-keys/alexkim-jupyter.key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -W %h:%p sky@127.0.0.1 -o ProxyCommand='/Users/alexkim/miniforge3/bin/python3.12 /Users/alexkim/miniforge3/lib/python3.12/site-packages/sky/templates/websocket_proxy.py http://127.0.0.1:46580 alexkim-jupyter'

@romilbhardwaj
Copy link
Collaborator

Thanks @alex000kim, @kyuds is looking into this.

@kyuds
Copy link
Collaborator

kyuds commented Apr 28, 2025

@alex000kim @romilbhardwaj sorry for the late replies! Was a bit back-logged but will be able to resolve this soon!
done

@kyuds kyuds mentioned this issue Apr 28, 2025
5 tasks
@alex000kim
Copy link
Contributor Author

I just tested that it works. Thanks so much for a quick turn around!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants