Update rstudio-gpu image

[anirban-mukherjee]

Describe the bug
In gce_vm, option 'template = "rstudio-gpu"' leads to the launch of an instance using the deprecated rocker/ml-gpu image. While this still image works, all libraries and software on the instance are outdated and the instance has been officially deprecated. Please see: https://hub.docker.com/r/rocker/ml-gpu.

The current image is here: https://hub.docker.com/r/rocker/ml. Would it be possible to request for googleComputeEngineR to be updated to use the new image. As far as I can tell, rocker/ml should come with all GPU libraries and should be a drop-in replacement.

To Reproduce
Launch an instance using gce_vm and 'template = "rstudio-gpu"'.

Expected behavior
Instance with recent R and TF libraries.

[MarkEdmondson1234]

Yes I will look at updating these, and see if there is a way to always use the latest version.

[anirban-mukherjee]

Great, thanks so much! AFAIK if googleComputeEngineR uses the names listed at https://github.com/rocker-org/rocker-versioned2, it should always be the latest. I think the deprecation of ml-gpu was one off and occurred at the same time as the move to a new build system and R 4.0.0.

[MarkEdmondson1234]

Can you try it now with the github version?

[anirban-mukherjee]

Thanks for the quick response! The instance seems to launch fine. I see the instance in gce_list_instances and in Google cloud console. I am able to connect to the instance using SSH from Google cloud console. gcer_docker_image seems correct, install-nvidia-driver is True. Both googleComputeEngineR and the console give me the same external ip. The firewall rules seem fine.

But when I go to the external ip using a browser, I don't see a login screen. I do see the RStudio icon on the url bar, but I cannot connect to Rstudio server (Safari says "failed to open page" and I get a blank screen -- no username/login boxes).

Any ideas on what might be going wrong? On my end, I setup GCE_AUTH_FILE, GCE_DEFAULT_PROJECT_ID, and GCE_DEFAULT_ZONE, before calling gce_vm. Below is the command I used. This command works fine for template = "rstudio".

vm <- gce_vm(template = "rstudio-gpu", name = "rstudio-server", disk_size_gb = 1000, predefined_type="n1-highmem-16", username = "bleh", password = "bleh", acceleratorCount = 1, acceleratorType = "nvidia-tesla-p100")

Many thanks and sorry for the bother!

[MarkEdmondson1234]

I'll have to review it, perhaps changing the image means some configuration has also changed.

[anirban-mukherjee]

sudo apt install (for anything) gives me:

E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?

pgrep -a apt gives me

apt -y upgrade

I am not sure what's running apt -y upgrade in the startup script (I didn't run it) but either the command is failing and leaving apt locked or the command is still running. apt list --upgradable gives me:

Listing... Done
docker-ce/stretch 5:19.03.15~3-0~debian-stretch amd64 [upgradable from: 18.06.1~ce~3-0~debian]
google-cloud-packages-archive-keyring/google-cloud-packages-archive-keyring-stretch 1.2-391078977 all [upgradable from: 1.2-1]
google-cloud-sdk/cloud-sdk-stretch 353.0.0-0 all [upgradable from: 216.0.0-0]
google-compute-engine/google-compute-engine-stretch-stable 1:20210629.00-g1 all [upgradable from: 2.8.4-1]
google-compute-engine-oslogin/google-compute-engine-stretch-stable 1:20210728.00-g1+deb9 amd64 [upgradable from: 1.3.1-1+deb9]
kubectl/cloud-sdk-stretch,kubernetes-xenial 1.22.0-00 amd64 [upgradable from: 1.11.3-00]
nvidia-container-runtime/stretch 3.5.0-1 amd64 [upgradable from: 2.0.0+docker18.06.1-1]
nvidia-docker2/stretch 2.6.0-1 all [upgradable from: 2.0.3+docker18.06.1-1]
python-google-compute-engine/google-compute-engine-stretch-stable 1:20191210.00-g1 all [upgradable from: 2.8.4-1]
python3-google-compute-engine/google-compute-engine-stretch-stable 1:20191210.00-g1 all [upgradable from: 2.8.4-1]

That seems like a lot of updates but its been an hour and I would have expected it to complete by now. I will leave the instance running and let you know if I am able to access it (in which case, its a matter of having to wait for apt upgrade to finish) or if it's still inaccessible over http.

[MarkEdmondson1234]

It could very well take longer than an hour if the VM is a small one with limited CPU. The startup scripts may well run updates and I guess they need to use more up to date installs.

[anirban-mukherjee]

I don't think that's it. I waited for well over an hour and still no dice. In that period, CPU utilisation, network traffic, disk IO -- all were basically 0. There is no signs that the instance was doing anything. Also, this is a 8-core (16 vcpu) Skylake server with 60 GB of RAM so it should not take that long.

[MarkEdmondson1234]

Thanks for the info - I will keep an eye on it.

[anirban-mukherjee]

I poked around more. I don't think the issue is with the rocker ml image. Looking at the description on the rocker webpage, it looks like the image is kept updated. I think the dockerhub description is likely out of date (it says the image has CUDA 10.1 for example), which led me to believe that the rocker image is outdated but it doesn't make sense for those elements to be outdated. I have deleted the inaccurate parts of my comments above.

I looked at the googleComputeEngineR code (gpus.r) and inside gce_vm_gpu I see:

  if(is.null(dots$image_family)){
    dots$image_family <- "tf-latest-cu92"
  }

Do you think that might be the problem? Running these three commands below shows the creation date for tf-latest-cu92 is in 2018. tf2-2-6-cu110 is the latest image (list available at https://cloud.google.com/deep-learning-vm/docs/images). The latest image can be gotten by using tf2-ent-latest-gpu.

gce_get_image_family("deeplearning-platform-release", "tf-latest-cu92")
gce_get_image_family("deeplearning-platform-release", "tf2-2-6-cu110")
gce_get_image_family("deeplearning-platform-release", "tf2-ent-latest-gpu")

I tried passing image_family to gce_vm_gpu by doing vm <- gce_vm_gpu(template = "rstudio-gpu", name = "rstudio-server-gpu", disk_size_gb = 1000, predefined_type="n1-standard-16", username = "###", password = "###", image_family="tf2-ent-latest-gpu", image_project ="deeplearning-platform-release", acceleratorType = "nvidia-tesla-p100") but I got:

2021-08-19 00:26:47> Launching VM with GPU support. If using docker_cmd() functions make sure to include nvidia=TRUE parameter
2021-08-19 00:26:48> Creating template VM
2021-08-19 00:26:48> Launching VM with GPU support. If using docker_cmd() functions make sure to include nvidia=TRUE parameter
ℹ 2021-08-19 00:26:53 > Request Status Code:  400
Error: API returned: Invalid value for field 'resource.metadata': '{  "item": [{    "key": "startup-script",    "value": "#!/bin/bash\necho \"Docker RStudio GPU launch...'. Metadata has duplicate keys: install-nvidia-driver

Hence, I have not been able to test if changing the image_family changes things. Fingers crossed that this is the issue though.