juptyerlab-git serverextension does not work if you install/activate it after you start the server

[lordravo]

Hi,

I am launching a jupyterlab with git, making use of a dataproc initialization action;
For some reason I am not able to clone any repository. I am not sure if that is due to a workspace issue, or an unsuccessfull installation;

My cluster-config:

{
	"region": "us-east1",
	"zone": "us-east1-b",
	"bucket": "..",
	"master_type": "n1-standard-4",
	"worker_type": "n1-standard-4",
	"master_disk_size": 100,
	"worker_disk_size": 50,
	"num_workers": 2,
	"image_version": "1.4",
	"scopes": ["https://www.googleapis.com/auth/cloud-platform"],
	"optional_components": ["ANACONDA", "JUPYTER"],
	"initialization_actions": [
		"gs://tapps-bi-dwh.appspot.com/bi-dataproc/init-actions/init-script-jupyterhub-lab-git.sh"
	]
}

My initialization script:

ROLE=$(/usr/share/google/get_metadata_value attributes/dataproc-role)
if [[ "${ROLE}" == 'Master' ]]; then
    mkdir -p jupyter
    cd /jupyter
    conda upgrade -y -c conda-forge jupyterlab
    nohup jupyter notebook --port 8080 --notebook-dir '/jupyter' &
    nohup jupyter lab --port 8081 --notebook-dir '/jupyter' &>> jupyterlab.log &
    conda install -y -c conda-forge nodejs
    jupyter labextension install @jupyterlab/git
    pip install -e git+https://github.com/jupyterlab/jupyterlab-git.git#egg=jupyterlab_git
    jupyter serverextension enable --py jupyterlab_git --sys-prefix
    jupyter lab build
fi

The error:

Any ideas on how to approach this issue?

Thanks

[fcollonval]

Thank @lordravo to report the issue. Could you give us the server reply for the clone request?

[lordravo]

jupyterlab.log gives me only a single line

[D 19:31:02.723 LabApp] Using contents: services/contents
[W 19:31:02.724 LabApp] 404 GET /metrics?1576092663430 (177.92.86.174) 1.31ms referer=http://35.231.140.18:8081/lab?
[D 19:31:07.860 LabApp] Using contents: services/contents
[W 19:31:07.861 LabApp] 404 GET /metrics?1576092668584 (177.92.86.174) 1.55ms referer=http://35.231.140.18:8081/lab?
[D 19:31:09.648 LabApp] Using contents: services/contents
>>>[W 19:31:09.649 LabApp] 404 POST /git/clone?1576092670373 (177.92.86.174) 1.70msreferer=http://35.231.140.18:8081/lab?<<<
[D 19:31:09.924 LabApp] 200 GET /api/contents/?content=1&1576092670524 (177.92.86.174) 125.53ms
[D 19:31:11.580 LabApp] 200 GET /api/sessions?1576092672302 (177.92.86.174) 0.92ms

Is there any other log I could look into?
Thanks @fcollonval

[telamonian]

jupyter labextension install @jupyterlab/git
pip install -e git+https://github.com/jupyterlab/jupyterlab-git.git#egg=jupyterlab_git
jupyter serverextension enable --py jupyterlab_git --sys-prefix
jupyter lab build

@lordravo You're installing the latest release version of the labextension, but the HEAD version of the server extension. These versions have to match. Unless you're actually trying to help develop jupyterlab-git itself, just do the regular release install. Replace the above 4 lines with just:

pip install -U jupyterlab-git
jupyter lab build

and everything should work.

[lordravo]

Hi @telamonian

Fixed the init_script, but the behavior is still the same:

[W 22:06:50.551 LabApp] 404 POST /git/clone?1576102011245 (177.92.86.174) 1.83ms referer=http://35.190.139.86:8081/lab?

btw, git init is the same also

[W 22:12:24.404 LabApp] 404 POST /git/init?1576102345099 (177.92.86.174) 1.83ms referer=http://35.190.139.86:8081/lab?

Any idea why?
Is there any other log? Or test I can make?

Just for the sake of it, I'm attaching the pip list
jupyterlab_pip_list.txt

[telamonian]

It looks like now the serverextension is not running. Serverextension activation should happen automatically in the latest version of jupyterlab-git, but certain setups will still require explicit enabling of the serverextension. So also do:

jupyter serverextension enable --py jupyterlab_git --sys-prefix

[telamonian]

Any idea why?
Is there any other log? Or test I can make?

If you want to get down into the the weeds, open your browser's devtools and go to the "network" pane. Search for any /git/clone requests and click on one of them. That should let you see the details of the request and the associated response, if any

[lordravo]

Hi @telamonian

I'm still getting 404 after enabling the serverextension;
Looking into the browser dev tools, not much to give any hint. Basically a page with the following

404 : Not Found
You are requesting a page that does not exist!

My current init script is now:

#!/bin/bash

ROLE=$(/usr/share/google/get_metadata_value attributes/dataproc-role)
if [[ "${ROLE}" == 'Master' ]]; then
    mkdir -p jupyter
    cd /jupyter

    echo "$(date) Upgrading JupyterLab"
    conda upgrade -y -c conda-forge jupyterlab

    echo "$(date) Preparing Jupyter notebook"
    nohup jupyter notebook --port 8080 --notebook-dir '/jupyter' &
    echo "$(date) Preparing JupyterLab"
    nohup jupyter lab --port 8081 --notebook-dir '/jupyter' &>> jupyterlab.log &

    echo "$(date) Preparing Nodejs"
    conda install -y -c conda-forge nodejs

    echo "$(date) Provisioning JupyterLab Git"
    pip install -U jupyterlab-git
    jupyter serverextension enable --py jupyterlab_git --sys-prefix
    jupyter lab build
fi

Not sure if it helps, but I'm getting 404 for other endpoints like '/git/server_root?1576158664365':

Extensions enabled:

I'm starting to believe it may be a dataproc issue. What do you guys think?

[lordravo]

May I ask, what's the contents_manager exaclty?

[telamonian]

@lordravo I re-reviewed the info you posted and figured out how to reproduce the error that you're seeing. I 100% guarantee that what is going on is that the jupyterlab-git serverextension is not enabled.

Aside from jupyterlab-git not being installed correctly (and it does appear to be installed correctly, given your output from jupyter serverextension list), the only way that you'll get this 404 in your response:

is if the jupyterlab-git serverextension is not correctly enabled.

Going back over your init script, I think I see the exact problem. You start up the jupyterlab server:

    echo "$(date) Preparing Jupyter notebook"
    nohup jupyter notebook --port 8080 --notebook-dir '/jupyter' &
    echo "$(date) Preparing JupyterLab"
    nohup jupyter lab --port 8081 --notebook-dir '/jupyter' &>> jupyterlab.log &

before you actually install the serverextension. serverextensions have to be installed and activated before the server runs. So move the above block to the end of your init script, and then everything should work

[lordravo]

Hi @telamonian

That's interesting. Changing the order at least fixed the server_root endpoint:

Git is still malfunctioning, but now I'm having a traceback at a different endpoint (git/show_top_level):

{"message": "Unhandled error", "reason": null, "traceback": "Traceback (most recent call last):\n  File \"/opt/conda/anaconda/lib/python3.6/site-packages/tornado/web.py\", line 1541, in _execute\n    result = method(*self.path_args, **self.path_kwargs)\n  File \"/opt/conda/anaconda/lib/python3.6/site-packages/jupyterlab_git/handlers.py\", line 89, in post\n    result = self.git.show_top_level(current_path)\n  File \"/opt/conda/anaconda/lib/python3.6/site-packages/jupyterlab_git/git.py\", line 493, in show_top_level\n    cwd=os.path.join(self.root_dir, current_path),\n  File \"/opt/conda/anaconda/lib/python3.6/subprocess.py\", line 729, in __init__\n    restore_signals, start_new_session)\n  File \"/opt/conda/anaconda/lib/python3.6/subprocess.py\", line 1364, in _execute_child\n    raise child_exception_type(errno_num, err_msg, err_filename)\nFileNotFoundError: [Errno 2] No such file or directory: '/Untitled Folder': '/Untitled Folder'\n"}

It seems git is failing on reaching jupyter directories. As you can see 'Untitled Folder' is the current dir:

Which is interesting, since the notebooks don't exists physically on the host machine. Jupyter seems to virtualize them somehow. Maybe they are encrypted. (They are not on '/jupyter' as I would like them to be)

Git clone ends up the same:

{"message": "Unhandled error", "reason": null, "traceback": "Traceback (most recent call last):\n  File \"/opt/conda/anaconda/lib/python3.6/site-packages/tornado/web.py\", line 1541, in _execute\n    result = method(*self.path_args, **self.path_kwargs)\n  File \"/opt/conda/anaconda/lib/python3.6/site-packages/jupyterlab_git/handlers.py\", line 35, in post\n    response = self.git.clone(data['current_path'], data['clone_url'], data.get('auth', None))\n  File \"/opt/conda/anaconda/lib/python3.6/site-packages/jupyterlab_git/git.py\", line 188, in clone\n    cwd=os.path.join(self.root_dir, current_path),\n  File \"/opt/conda/anaconda/lib/python3.6/subprocess.py\", line 729, in __init__\n    restore_signals, start_new_session)\n  File \"/opt/conda/anaconda/lib/python3.6/subprocess.py\", line 1364, in _execute_child\n    raise child_exception_type(errno_num, err_msg, err_filename)\nFileNotFoundError: [Errno 2] No such file or directory: '/Untitled Folder': '/Untitled Folder'\n"}

Any ideas?

[Update]
After a few tests, I found out that cloning into root, ended up with status 200, and the following response: {"code": 0}

Nothing apears on the working directory though, making me believe the clone didn't succeeded. But if I try to clone it again (the same repo a 2 time but not a diferent one), jupyter freezes, returning only if I refresh the page.

Looking into the logs, I can only find that the second clone, which makes jupyter to freeze, actually resulted on 200, success.

That's really confusing.