Re-executing a cell hides the figure when it is given a `num=...` argument

[tovrstra]

Describe the issue

Explicit figure numbers seem to cause a small glitch in ipympl. When the following cell is executed once, it shows the plot. When re-executing the cell, it disappears again.

x = np.linspace(-np.pi/2, np.pi/2)
plt.figure(num=1)
plt.plot(x, np.sin(x))

I've attached a working example: repeat_plot_issue.zip

The inline and notebook backends do not have this issue, which leads me to believe this is a bug. There are two workarounds:

1. Close the figure in the beginning of the cell with e.g. plt.close(1)
2. Show the figure at the end of the cell with plt.show()

(I'm using Jupyter notebooks with a group of about 40 students, and they got confused by this difference in behavior between the backends. I understand this is not a major issue because there are simple workarounds. Still, for newcomers, it is an extra difficulty: "Help, my plot has disappeared!")

Versions

 3.9.7 | packaged by conda-forge | (default, Sep 29 2021, 19:20:46) 
[GCC 9.4.0]
ipympl version: 0.8.2
Selected Jupyter core packages...
IPython          : 7.30.1
ipykernel        : 6.6.0
ipywidgets       : 7.6.5
jupyter_client   : 6.1.12
jupyter_core     : 4.9.1
jupyter_server   : 1.13.1
jupyterlab       : 3.2.5
nbclient         : 0.5.9
nbconvert        : 6.3.0
nbformat         : 5.1.3
notebook         : 6.4.6
qtconsole        : not installed
traitlets        : 5.1.1
Known nbextensions:
  config dir: /home/toon/.jupyter/nbconfig
    notebook section
      splitcell/splitcell disabled
      nbextensions_configurator/config_menu/main  enabled 
      - Validating: problems found:
        - require?  X nbextensions_configurator/config_menu/main
      contrib_nbextensions_help_item/main  enabled 
      - Validating: OK
      spellchecker/main  enabled 
      - Validating: OK
      toc2/main  enabled 
      - Validating: OK
      k3d/extension  enabled 
      - Validating: OK
    tree section
      nbextensions_configurator/tree_tab/main  enabled 
      - Validating: problems found:
        - require?  X nbextensions_configurator/tree_tab/main
  config dir: /home/toon/.local/etc/jupyter/nbconfig
    notebook section
      nbdime/index  enabled 
      - Validating: problems found:
        - require?  X nbdime/index
      rise/main  enabled 
      - Validating: OK
  config dir: /home/toon/miniconda3/envs/py4sci/etc/jupyter/nbconfig
    notebook section
      jupyter-matplotlib/extension  enabled 
      - Validating: OK
      rise/main  enabled 
      - Validating: OK
      voila/extension  enabled 
      - Validating: OK
      jupyter-js-widgets/extension  enabled 
      - Validating: OK
      nbextensions_configurator/config_menu/main  enabled 
      - Validating: problems found:
        - require?  X nbextensions_configurator/config_menu/main
      contrib_nbextensions_help_item/main  enabled 
      - Validating: OK
    tree section
      nbextensions_configurator/tree_tab/main  enabled 
      - Validating: problems found:
        - require?  X nbextensions_configurator/tree_tab/main
  config dir: /etc/jupyter/nbconfig
    notebook section
      jupyter-js-widgets/extension  enabled 
      - Validating: OK
JupyterLab v3.2.5
/home/toon/miniconda3/envs/py4sci/share/jupyter/labextensions
        k3d v2.11.0 enabled OK (python, k3d)
        jupyter-matplotlib v0.10.2 enabled OK
        @jupyter-widgets/jupyterlab-manager v3.0.1 enabled OK (python, jupyterlab_widgets)
        @ijmbarr/jupyterlab_spellchecker v0.7.2 enabled OK (python, jupyterlab-spellchecker)
        @ryantam626/jupyterlab_code_formatter v1.4.10 enabled OK (python, jupyterlab-code-formatter)
        @voila-dashboards/jupyterlab-preview v2.1.0 enabled OK (python, voila)

[martinRenou]

Thanks for reporting this issue.

For some reason, the issue doesn't appear if the activation of the backend is done in the same cell:

%matplotlib widget

import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(-np.pi/2, np.pi/2)
plt.figure(num=4)
plt.plot(x, np.sin(x));

[ianhi]

This got kinda long the tldr is: I think this is actually a bug in nbagg and the best solution is probably to teach your students to use the explicit (nee OO) interface like so:

fig1, ax1 = plt.subplots()
ax1.plot(...)

The inline and notebook backends do not have this issue, which leads me to believe this is a bug. There are two workarounds:

Inline probably isn't the best comparison because it displays and then closes all figures at the end of every cell. I'm a little bit surprised by the behavior of nbagg though. As I look at this more though I think this may actually be a bug in nbagg?

My understanding of the behavior of plt.figure(num=1) is that it will generate and display(when in a notebook) a new figure if no figure with ID=1 exists, otherwise it will set the current figure (plt.gcf()) and not do any further display. It seems like the nbagg backend is regenerating the frontend every time plt.figure is called, while ipympl is, I think correctly, recognizing that the figure already exists and now doing any more display or creation logic. Then the outputs of the cell are overwritten as expected by the new execution.

For example if you run the following two cells:

cell 1

%matplotlib nbagg
import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(-np.pi/2, np.pi/2)
i=0

cell 2 (run this one repeatedly)

plt.figure(num=1)
plt.plot(x, np.sin(x*i))
i += 1

then you only ever have a plot with a single line in it.

but I think the correct behavior should be to have multiple lines show up. To get that you can do the equivalent from ipython script

import matplotlib.pyplot as plt
import numpy as np

plt.ion()
x = np.linspace(-np.pi/2, np.pi/2)

# each iteration of loop represents a cell re-execution
i = 0
def run_cell():
    global i
    plt.figure(num=1)
    plt.plot(x, np.sin(x*i))
    i += 1

For some reason, the issue doesn't appear if the activation of the backend is done in the same cell:

I think this is because setting the backend closes all figures so you are no longer refering to the same figure.

[ianhi]

This got kinda long the tldr is: I think this is actually a bug in nbagg and the best solution is probably to teach your students to use the explicit (nee OO) interface like so:

That's a bit short of an explanation - happy to talk more about this if I can be helpful!

[tovrstra]

@ianhi Thank you for all the insightful comments! I'm going to experiment with this, before asking more questions.

[tacaswell]

As @jklymak and @QuLogic pointed out in matplotlib/matplotlib#21957 the issue is with the life cyle management of the implicit figures.

In the case of nbagg we have logic that when the js side is removed from view we "close" the figure and drop it from Matplotlib's registry of open figures. From the behavior I think the order of things is output is cleared -> call back from js-to-kernel is run to close the figure -> cell is run -> matplotlib finds there is no figure 1 makes a new one -> new figures are shown at the end of cell execution.

In the IPython + GUI case because the window is never closed, each time through the loop we find that after the first pass there exists a figure 1, plt.figure(num=1) sets it to be the "current figure" and the plt functions operate on its current axes. In the last gif from @ianhi if he closed the window by clicking the 'x' between calling run_cell() (or added plt.close(1) you would get the same behavior as nbagg.

As @ianhi notes with inline it always closes the figure after rendering the cell so it always makes a new one.

In both the nbagg and ipympl cases if you create the figure in one cell, and then repeatedly do
plt.figure(num=1) in a subsequent cell, you will get the GUI like behavior of adding lines to the same plot.

Of these semantics, it is not clear which one is "right" and how this should interact with having multiple views of the same output.

Also see the discussion in #171

[tovrstra]

After trying a few things, I found nbagg's behavior more intuitive. I'll summarize the situation for the common use case of someone refining their plotting code and re-executing that incrementally improved code cell several times.

• ipympl behavior. When re-executing the cell with plotting code, previously created plots will stay in matplotlib's registry, also when the output containing the widget is cleared due to the re-execution. To avoid an increasing number of plots, extra lines of code are needed to remove/clear the old version of the plot and/or to make sure the new version is displayed. There are a few ways of achieving this:

  First (old matplotlib API):

  plt.close(1)
  plt.figure(num=1)
  plt.plot([0, 1], [2, 3])

  Second (old matplotlib API):

  plt.figure(num=1, clear=True)
  plt.plot([0, 1], [2, 3])
  plt.show()

  First (new matplotlib API):

  plt.close(1)
  fig, ax = plt.subplots(num=1)
  plt.plot([0, 1], [2, 3])

  Second (new matplotlib API): <-- does not work, figure not shown after re-execution

  fig, ax = plt.subplots(num=1, clear=True)
  ax.plot([0, 1], [2, 3])
  fig.show()

• nbagg behavior. When re-executing the cell, there is no risk of creating an increasing number of plots, which are not used.
  They get removed when the output containing the widget is cleared.
  E.g. The following can be re-executed without creating an increasing number of figures:

  old matplotlib API:

  plt.plot([0, 1], [2, 3])

  new matplotlib API:

  fig, ax = plt.subplots()
  ax.plot([0, 1], [2, 3])

Comparing all these, a modification of ipympl's behavior to match that of nbagg, seems ideal, but I understand that that is a lot of work.

For the time being, calling plt.close before making any plot seems to be the safest workaround. This always works with any backend or matplotlib API: it avoids an increasing number of unused figures and makes sure the figure is always shown.

[tacaswell]

completely off-topic: the explicit (aka Object Oriented) API is "new" as of ~15 years ago ;)

---

Try:

fig, ax = plt.subplots(num=1, clear=True)
ax.plot([0, 1], [2, 3])
plt..show()

instead.

[tovrstra]

Thanks for your help! That works indeed, but I'm still puzzled by how ipympl decides what to show when plt.show() is called. The documentation of plt.show mentions that this function shows all open figures. Ipympl seems to work differently (for good reasons). I tried different ways of using it, but did see any pattern yet. Is there some simple rule to understand what is shown by plt.show?

In comparison, the behavior of calling plt.close upfront is more predictable. You always get to see the figure, irrespective of what happened elsewhere in the notebook. It is also a bit clumsy, though, because previously shown widgets for the same figure will stop working.

P.S. Yes, "very old" versus "old" would have been more fitting. :]