Integration with JupyterHub (TLJH) Kernel always died (auto-restarting)

[HoudayerPierre]

Hello, I have a JupyterHub (TLJH) with JupyterLab installed, all behind a reverse HTTP/WebSocket proxy (python kernel works).
The robotkernel was installed and I can see it in the luncher tab.
If I open a notebook with a robot kernel the kernel seems to not start. If press the start button nothing is running and no output is shown.
After some time the kernel status is marked as "autorestarting" and a popup is shown.

Kernel Restarting
The kernel for RobotFramework/Untitled.ipynb appears to have died. It will restart automatically.

For installation I use a jupyter terminal (my user is admin) just using sudo before :

sudo jupyter labextension install jupyterlab_robotmode
sudo jupyter labextension install @jupyter-widgets/jupyterlab-manager

[datakurre]

@bollwyvl Any ideas what could cause this? Or how to get more traceback for this issue? Doesn’t Binder use somewhat similar setup, and it works fine?

[bollwyvl]

Does ipython work? There should be more logs than what is in the browser... These are likely on a systemd unit (e.g journalctl -u) and are for the specific single user notebook server, not the hub unit.

[HoudayerPierre]

Yes ipython works in this setup.

from a jupyter terminal with :
sudo journalctl -u jupyterhub -f

sept. 20 18:24:54 jupyterlab python3[1099]: [I 2020-09-20 18:24:54.400 JupyterHub proxy:319] Checking routes
sept. 20 18:24:55 jupyterlab python3[1099]: [I 2020-09-20 18:24:55.017 JupyterHub log:174] 200 GET /jupyterlab/hub/api/users (cull-idle@127.0.0.1) 76.31ms
sept. 20 18:25:55 jupyterlab python3[1099]: [I 2020-09-20 18:25:55.011 JupyterHub log:174] 200 GET /jupyterlab/hub/api/users (cull-idle@127.0.0.1) 70.32ms
sept. 20 18:26:12 jupyterlab python3[1099]: [I 2020-09-20 18:26:12.897 JupyterHub log:174] 200 POST /jupyterlab/hub/api/users/pitdu53/activity (pitdu53@127.0.0.1) 199.82ms
sept. 20 18:26:47 jupyterlab python3[1099]: [I 2020-09-20 18:26:47.207 JupyterHub log:174] 200 GET /jupyterlab/hub/api/authorizations/token/[secret] (pitdu53@127.0.0.1) 176.90ms
sept. 20 18:26:55 jupyterlab python3[1099]: [I 2020-09-20 18:26:55.018 JupyterHub log:174] 200 GET /jupyterlab/hub/api/users (cull-idle@127.0.0.1) 76.99ms
sept. 20 18:27:55 jupyterlab python3[1099]: [I 2020-09-20 18:27:55.025 JupyterHub log:174] 200 GET /jupyterlab/hub/api/users (cull-idle@127.0.0.1) 83.65ms

journalctl -u jupyter-pitdu53

sept. 20 18:22:05 jupyterlab bash[3445]: [I 2020-09-20 18:22:05.762 SingleUserNotebookApp restarter:110] KernelRestarter: restarting kernel (1/5), new random ports
sept. 20 18:22:06 jupyterlab bash[3445]: Traceback (most recent call last):
sept. 20 18:22:06 jupyterlab bash[3445]:   File "/opt/tljh/user/lib/python3.7/runpy.py", line 193, in _run_module_as_main
sept. 20 18:22:06 jupyterlab bash[3445]:     "__main__", mod_spec)
sept. 20 18:22:06 jupyterlab bash[3445]:   File "/opt/tljh/user/lib/python3.7/runpy.py", line 85, in _run_code
sept. 20 18:22:06 jupyterlab bash[3445]:     exec(code, run_globals)
sept. 20 18:22:06 jupyterlab bash[3445]:   File "/home/jupyter-pitdu53/.local/lib/python3.7/site-packages/robotkernel/kernel.py", line 11, in <module>
sept. 20 18:22:06 jupyterlab bash[3445]:     from robotkernel.executors import execute_python
sept. 20 18:22:06 jupyterlab bash[3445]:   File "/home/jupyter-pitdu53/.local/lib/python3.7/site-packages/robotkernel/executors.py", line 17, in <module>
sept. 20 18:22:06 jupyterlab bash[3445]:     from robotkernel.utils import data_uri
sept. 20 18:22:06 jupyterlab bash[3445]:   File "/home/jupyter-pitdu53/.local/lib/python3.7/site-packages/robotkernel/utils.py", line 7, in <module>
sept. 20 18:22:06 jupyterlab bash[3445]:     from lunr.builder import Builder
sept. 20 18:22:06 jupyterlab bash[3445]:   File "/home/jupyter-pitdu53/.local/lib/python3.7/site-packages/lunr/__init__.py", line 5, in <module>
sept. 20 18:22:06 jupyterlab bash[3445]:     from lunr.__main__ import lunr
sept. 20 18:22:06 jupyterlab bash[3445]:   File "/home/jupyter-pitdu53/.local/lib/python3.7/site-packages/lunr/__main__.py", line 4, in <module>
sept. 20 18:22:06 jupyterlab bash[3445]:     from past.builtins import basestring
sept. 20 18:22:06 jupyterlab bash[3445]: ModuleNotFoundError: No module named 'past'

[datakurre]

I have never seen that before, but if it is really “lunr” Python package breaking things, its current master seem to have dropped Python 2 support and at least that error cannot happen anymore.

Maybe

pip install git+https://github.com/yeraydiazdiaz/lunr.py@master

[HoudayerPierre]

From a jupyter terminal :

(base) jupyter-pitdu53@jupyterlab:~$ pip install git+https://github.com/yeraydiazdiaz/lunr.py@master
Defaulting to user installation because normal site-packages is not writeable
Collecting git+https://github.com/yeraydiazdiaz/lunr.py@master
  Cloning https://github.com/yeraydiazdiaz/lunr.py (to revision master) to /tmp/pip-req-build-hnxboncv
  Running command git clone -q https://github.com/yeraydiazdiaz/lunr.py /tmp/pip-req-build-hnxboncv
Requirement already satisfied (use --upgrade to upgrade): lunr==0.5.8 from git+https://github.com/yeraydiazdiaz/lunr.py@master in ./.local/lib/python3.7/site-packages
Building wheels for collected packages: lunr
  Building wheel for lunr (setup.py) ... done
  Created wheel for lunr: filename=lunr-0.5.8-py2.py3-none-any.whl size=34695 sha256=b938d76b3ee266fe5dfe46b65c4fe6e835852ae7c686a19349a963242fb63a25
  Stored in directory: /tmp/pip-ephem-wheel-cache-jsph6cs_/wheels/88/01/7b/233afaae653e5e4c68d4893d37b9a4966f31cf16aaf51a0e6d
Successfully built lunr

Then reboot.
if i use --upgrade : pip install git+https://github.com/yeraydiazdiaz/lunr.py@master --upgrade :
The kernel is now on idle.

sept. 21 07:44:24 jupyterlab bash[1189]: [I 2020-09-21 07:44:24.527 SingleUserNotebookApp kernelmanager:172] Kernel started: eeb208a9-68eb-406b-af4e-b80b09f4d171
sept. 21 07:44:24 jupyterlab bash[1189]: [I 2020-09-21 07:44:24.531 SingleUserNotebookApp log:174] 201 POST /jupyterlab/user/pitdu53/api/sessions?1600674264513 (pitdu53@172.16.0.17) 15.93ms
sept. 21 07:44:24 jupyterlab bash[1189]: [I 2020-09-21 07:44:24.546 SingleUserNotebookApp log:174] 200 GET /jupyterlab/user/pitdu53/api/sessions?1600674264538 (pitdu53@172.16.0.17) 4.55ms
sept. 21 07:44:24 jupyterlab bash[1189]: [I 2020-09-21 07:44:24.565 SingleUserNotebookApp log:174] 200 GET /jupyterlab/user/pitdu53/api/kernelspecs?1600674264555 (pitdu53@172.16.0.17) 6.70ms
sept. 21 07:44:24 jupyterlab bash[1189]: [I 2020-09-21 07:44:24.582 SingleUserNotebookApp log:174] 200 GET /jupyterlab/user/pitdu53/api/kernels?1600674264559 (pitdu53@172.16.0.17) 5.82ms
sept. 21 07:44:25 jupyterlab bash[1189]: [I 2020-09-21 07:44:25.300 SingleUserNotebookApp log:174] 101 GET /jupyterlab/user/pitdu53/api/kernels/eeb208a9-68eb-406b-af4e-b80b09f4d171/channels?session_id=e3d0b339-7023-484e-9fdd-7d25ad3ed4d0 (pitdu53@172.16.0.17) 720.74ms
sept. 21 07:44:27 jupyterlab bash[1189]: [I 2020-09-21 07:44:27.212 SingleUserNotebookApp log:174] 204 PUT /jupyterlab/user/pitdu53/lab/api/workspaces/lab?1600674267207 (pitdu53@172.16.0.17) 1.96ms
sept. 21 07:44:28 jupyterlab bash[1189]: [I 2020-09-21 07:44:28.609 SingleUserNotebookApp log:174] 200 GET /jupyterlab/user/pitdu53/metrics?1600674268601 (pitdu53@172.16.0.17) 5.25ms
sept. 21 07:44:28 jupyterlab bash[1189]: [I 2020-09-21 07:44:28.925 SingleUserNotebookApp log:174] 200 GET /jupyterlab/user/pitdu53/api/contents/RobotFramework?content=1&1600674268921 (pitdu53@172.16.0.17) 1.66ms
sept. 21 07:44:29 jupyterlab bash[1189]: [I 2020-09-21 07:44:29.250 SingleUserNotebookApp log:174] 200 GET /jupyterlab/user/pitdu53/api/terminals?1600674269246 (pitdu53@172.16.0.17) 1.00ms
sept. 21 07:44:33 jupyterlab bash[1189]: [I 2020-09-21 07:44:33.625 SingleUserNotebookApp log:174] 200 GET /jupyterlab/user/pitdu53/metrics?1600674273617 (pitdu53@172.16.0.17) 5.60ms
sept. 21 07:44:34 jupyterlab bash[1189]: [I 2020-09-21 07:44:34.559 SingleUserNotebookApp log:174] 200 GET /jupyterlab/user/pitdu53/api/sessions?1600674274556 (pitdu53@172.16.0.17) 1.24ms
sept. 21 07:44:34 jupyterlab bash[1189]: [I 2020-09-21 07:44:34.595 SingleUserNotebookApp log:174] 200 GET /jupyterlab/user/pitdu53/api/kernels?1600674274591 (pitdu53@172.16.0.17) 1.14ms
sept. 21 07:44:38 jupyterlab bash[1189]: [I 2020-09-21 07:44:38.607 SingleUserNotebookApp log:174] 200 GET /jupyterlab/user/pitdu53/api/contents/RobotFramework?content=1&1600674278602 (pitdu53@172.16.0.17) 1.55ms
sept. 21 07:44:38 jupyterlab bash[1189]: [I 2020-09-21 07:44:38.643 SingleUserNotebookApp log:174] 200 GET /jupyterlab/user/pitdu53/metrics?1600674278634 (pitdu53@172.16.0.17) 4.86ms
sept. 21 07:44:39 jupyterlab bash[1189]: [I 2020-09-21 07:44:39.110 SingleUserNotebookApp log:174] 204 PUT /jupyterlab/user/pitdu53/lab/api/workspaces/lab?1600674279104 (pitdu53@172.16.0.17) 3.03ms
sept. 21 07:44:39 jupyterlab bash[1189]: [I 2020-09-21 07:44:39.263 SingleUserNotebookApp log:174] 200 GET /jupyterlab/user/pitdu53/api/terminals?1600674279259 (pitdu53@172.16.0.17) 1.13ms
sept. 21 07:44:40 jupyterlab bash[1189]: [I 2020-09-21 07:44:40.760 SingleUserNotebookApp log:174] 204 PUT /jupyterlab/user/pitdu53/lab/api/workspaces/lab?1600674280755 (pitdu53@172.16.0.17) 2.15ms
sept. 21 07:44:43 jupyterlab bash[1189]: [I 2020-09-21 07:44:43.665 SingleUserNotebookApp log:174] 200 GET /jupyterlab/user/pitdu53/metrics?1600674283658 (pitdu53@172.16.0.17) 4.87ms
sept. 21 07:44:44 jupyterlab bash[1189]: [I 2020-09-21 07:44:44.592 SingleUserNotebookApp log:174] 200 GET /jupyterlab/user/pitdu53/api/sessions?1600674284588 (pitdu53@172.16.0.17) 1.97ms
sept. 21 07:44:44 jupyterlab bash[1189]: [I 2020-09-21 07:44:44.609 SingleUserNotebookApp log:174] 200 GET /jupyterlab/user/pitdu53/api/kernels?1600674284605 (pitdu53@172.16.0.17) 1.26ms
sept. 21 07:44:48 jupyterlab bash[1189]: [I 2020-09-21 07:44:48.625 SingleUserNotebookApp log:174] 200 GET /jupyterlab/user/pitdu53/api/contents/RobotFramework?content=1&1600674288621 (pitdu53@172.16.0.17) 1.66ms
sept. 21 07:44:48 jupyterlab bash[1189]: [I 2020-09-21 07:44:48.681 SingleUserNotebookApp log:174] 200 GET /jupyterlab/user/pitdu53/metrics?1600674288674 (pitdu53@172.16.0.17) 4.79ms
sept. 21 07:44:48 jupyterlab bash[1189]: [I 2020-09-21 07:44:48.767 SingleUserNotebookApp log:174] 200 GET /jupyterlab/user/pitdu53/api/contents/RobotFramework/Untitled.ipynb?content=0&1600674288761 (pitdu53@172.16.0.17) 1.36ms
sept. 21 07:44:48 jupyterlab bash[1189]: [I 2020-09-21 07:44:48.780 SingleUserNotebookApp handlers:164] Saving file at /RobotFramework/Untitled.ipynb
sept. 21 07:44:48 jupyterlab bash[1189]: [I 2020-09-21 07:44:48.903 SingleUserNotebookApp log:174] 200 PUT /jupyterlab/user/pitdu53/api/contents/RobotFramework/Untitled.ipynb?1600674288776 (pitdu53@172.16.0.17) 123.97ms
sept. 21 07:44:48 jupyterlab bash[1189]: [I 2020-09-21 07:44:48.919 SingleUserNotebookApp log:174] 200 GET /jupyterlab/user/pitdu53/api/contents/RobotFramework/Untitled.ipynb/checkpoints?1600674288914 (pitdu53@172.16.0.17) 2.37ms
sept. 21 07:44:48 jupyterlab bash[1189]: [I 2020-09-21 07:44:48.931 SingleUserNotebookApp log:174] 200 GET /jupyterlab/user/pitdu53/api/contents/RobotFramework?content=1&1600674288915 (pitdu53@172.16.0.17) 1.65ms

[HoudayerPierre]

For the module past, I search python past module here https://pypi.org/project/past/

from jupyter terminal :

(base) jupyter-pitdu53@jupyterlab:~$ pip install future
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: future in /usr/bin/anaconda3/lib/python3.7/site-packages (0.18.2)

[bollwyvl]

Hm.... you have a lot of different python environments that are interacting with each other (I see at least 3, plus potentially a user site packages).... i guess if you can `conda list` and `pip freeze` on each of the offending environments, that might help shed some light on it... if at all possible, try and do everything with pip OR conda, the mix of the two can be very unpredictable... and hot installing from master can make _everything_ go haywire...

[HoudayerPierre]

With a jupyter terminal with pip freeze I can see :

future==0.18.2
lunr==0.5.8
robotframework==3.2.2
robotkernel==1.4.0

Did you want specific items or the full output ?

With conda list there is no robot* item in the list.
(Also I remove and reinstall lunr)

[HoudayerPierre]

with a jupyter terminal :

(base) jupyter-pitdu53@jupyterlab:~$ python
Python 3.7.6 (default, Jan  8 2020, 19:59:22) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import lunr
>>> import past
>>> from past.builtins import basestring
>>> exit()

I'm lost but this is two differents environnement from the robot kernel and my jupyter terminal ?

[datakurre]

@HoudayerPierre And journalctl -u jupyter-pitdu53 still shows robotkernel crashing with a Python stacktrace?

And yes, I am still puzzled too, because I still have never seen that error before.

Have you already tried creating a new clean Conda Python 3 environment and installing just robotkernel with pip? I used to install robotkernel with miniconda and pip and it used to just work.

[HoudayerPierre]

Yes same stacktrace as before with the error in past module.
What did you recommand to do because in the jupyter terminal I tried :

sudo jupyter labextension uninstall jupyterlab_robotmode
sudo jupyter labextension uninstall @jupyter-widgets/jupyterlab-manage
pip uninstall robotkernel

and followed the readme with (just added sudo) :

pip install robotkernel
sudo jupyter labextension install jupyterlab_robotmode
sudo jupyter labextension install @jupyter-widgets/jupyterlab-manage

[datakurre]

But it should not be possible to get the same exception with pip install git+https://github.com/yeraydiazdiaz/lunr.py@master --upgrade, because master no longer imports past... 👀

That said, the most probably issue, as Nick noticed, comes from the Python setup in the JupyterHub node in question. In this issue we have seen that the environment has at least

1. a custom Python installation at /opt/tljh/user/lib/python3.7 (maybe has /opt/tljh/user/lib/python3.7/site-packages/)
2. conda Python at /usr/bin/anaconda3/lib/python3.7/site-packages
3. userspace packages at /home/jupyter-pitdu53/.local/lib/python3.7/site-packages

With a clean setup, I believe, we should see only one of these paths in logs / stacktraces.

I would expect making a new clean Python virtualenv / Conda environment for robotkernel and pip installing robotkernel there to work, but I don't know how it should be configured for JupyterHub to launch robotkernel from that environment 🤔

[HoudayerPierre]

Yes if i use --upgrade : pip install git+https://github.com/yeraydiazdiaz/lunr.py@master --upgrade :
The kernel is now on idle.