marimo's text format is hard for humans/non-marimo tools to understand

[Ubehebe]

One of the main advantages of marimo compared to other notebook formats is that marimo notebooks are syntactically valid Python files. This means that tools that analyze Python files (linters, formatters, type-checkers, IDEs) can generally do something useful with marimo notebooks without any setup.

As I've used marimo more, I've discovered some exceptions. marimo notebooks are syntactically valid Python, but they aren't idiomatic Python. This means that some tools can't analyze marimo notebooks in a useful way.

Here's an example. When you use an import in a notebook:

import pandas as pd

df = pd.DataFrame(...)

marimo serializes that to disk as something like:

@app.cell
def __():
  import pandas as pd
  return pd

@app.cell
def __(pd):
  df = pd.DataFrame(...)

This is basically a serialized DAG: the nodes (cells) are represented by top-level functions decorated with @app.cell, and the edges (dependencies) are represented by function params/return values.

The serialization is elegant, but tools other than marimo can't understand the indirection -- for example, they can't understand that the DataFrame constructor comes from the pandas import. This means that:

1. tools like ruff can't remove unused imports from marimo notebooks. If you delete the DataFrame instantiation above and run the file through ruff, the import pandas as pd statement remains.
2. IDEs can't navigate from the DataFrame instantiation to its definition in pandas.

I can see a few approaches we might take to improve this situation, but before proposing anything specific, I wanted to start a discussion. Maintainers, have you thought about this? How important do you think it is to improve?

My own view is that it's medium importance. For (1), unused imports can significantly slow down notebook execution. And for (2), being able to use IDE features to edit marimo notebooks would make large codebases significantly more maintainable (refactoring, etc.).

Thanks for your time!

[akshayka]

@Ubehebe , sorry for the late response -- just saw your post.

I can see a few approaches we might take to improve this situation, but before proposing anything specific, I wanted to start a discussion. Maintainers, have you thought about this? How important do you think it is to improve?

I've thought about this insofar as the indirection bothers me, too. But I haven't spent time trying to design something better. In particular, number 2 resonates with me -- it would be great to make editing in an IDE/text editor easier.

Definitely open to hearing your suggestions. Thanks so much for the thoughtful message!

[MeRe9]

Design something better: https://hamilton.dagworks.io

[dmadisetti]

The trade off marimo has to make is automatically serializing from a notebook/cells to python, while keeping it readable. With pure python approach, there is definitely a bit more flexibility. Reading through the docs, I don't think Hamilton is inherently more readable, and seems to even have a few design patterns similar to marimo

That being said, it looks like Hamilton has some good ideas for data pipelines that could inspire marimo. Do you have any experience with Hamilton drivers? What has been your impression?

[Ubehebe]

I think the main question I have is: why does marimo have to serialize the dataflow graph into the source file? Why can't it be an in-memory data structure on the server?

The first problem is that marimo needs some way to partition a Python source file into cells. The @app.cell decoration and the synthetic functions are as good a way as any. (There are other possible approaches, but the synthetic functions don't confuse non-marimo tools, so I'm not that concerned about them.)

The edges of the dataflow graph (the parameters of the synthetic functions) are what confuse non-marimo tools. Why do they need to be in the source file? Couldn't marimo run the dataflow analysis once on startup and keep the graph in-memory? This might slow down the initial time to interactive, but I doubt it would be significant when import statements regularly take 1+ second.

If we're able to make marimo notebooks more idiomatic Python so that other tools can work on them seamlessly, I think that's a good tradeoff.

[akshayka]

Why do they need to be in the source file? Couldn't marimo run the dataflow analysis once on startup and keep the graph in-memory?

@Ubehebe, you're totally right, they don't need to be in the source file. In fact marimo doesn't even read the args and returns of the app.cell decorated functions, and instead just redoes the dataflow analysis as you suggest.

The main reason references are included as cell/function args is so that the code is more legible to human eyes. For example, when designing the file format, I believed:

@app.cell
def __():
  x = 0
  return x

@app.cell
def __(x):
  y = x + 1
  return y

was more legible than

@app.cell
def __():
  x = 0

@app.cell
def __():
  y = x + 1

Because in the former, at least x is bound to something (the function argument). Plus, it makes using the cell.run() API a bit easier, since you can read off the references from the signature.

But, I might have been mistaken! I am open to not serializing the edges in the file format if it would make the Python more idiomatic. How does this improve the IDE experience? Do you have a suggestion based on removing the edges that would make the file format more idiomatic?

[Ubehebe]

I am open to not serializing the edges in the file format if it would make the Python more idiomatic. How does this improve the IDE experience?

I realized I've been thinking only about imports. marimo currently serializes all names that appear in a Python file into the params of synthetic functions. Top-level imports are a kind of name. If marimo didn't serialize top-level imports into the function params, IDEs could navigate from the use of an import to its definition. Simple example:

# user writes
import pandas as pd

df = pd.DataFrame(...)

# current marimo serialization
import marimo

app = marimo.App()

@app.cell
def __():
  import pandas as pd
  return pd

@app.cell
def __(pd):
  df = pd.DataFrame(...) # bad: `pd` param "shadows" top-level import
  return df

# if marimo didn't serialize top-level imports
import marimo
import pandas as pd

app = marimo.App()

@app.cell
def __():
  df = pd.DataFrame(...) # good: IDE knows `pd` is pandas

This might be worth doing to get ruff unused imports working, plus limited IDE navigation. But this doesn't help tools with names that are not top-level imports (like local variables). If marimo doesn't serialize any names, then non-marimo tools can't do much with synthetic functions like this, as you point out.

@app.cell
def __():
  y = x + 2 # marimo knows where this x comes from, other tools do not

I think the fundamental problem is that marimo's choice of marker for cells (functions) introduces a lexical scope. This hides relationships that would otherwise be legible to both humans and non-marimo tools.

Did you consider using a cell marker that doesn't introduce a lexical scope?

# marimo:cell
x = 1

# marimo:cell
y = x + 2

Putting behavior in comments is grungy, but this has the advantage that the relationship between the names is immediately apparent.

[akshayka]

Sorry for the delayed response @Ubehebe

Did you consider using a cell marker that doesn't introduce a lexical scope?

Yea we did consider this. The reason we didn't go down this route is that we wanted the notebook files to be importable as Python modules, for reusability -- to be able to support reusing cells, functions, or classes from one notebook in another.

Writing the notebook code as a flat script would either mean that the notebook would be executed on import, or the code wouldn't be reusable because it would be nested under an if __name__ == "__main__" guard.

But I do agree that the flat version you suggest provides a much better editing experience. I wonder if we can somehow get the best of both worlds.

[debnath-d]

Writing the notebook code as a flat script would either mean that the notebook would be executed on import

Why would that be? Allow the user to make that choice, just like regular .py files. In a flat script format, a user can always choose to manually guard it with if __name__ == "__main__" or not as they desire.

You don't have to automatically guard the whole notebook with if __name__ == "__main__" by default. Leave it up to the user.

The only issue with jupyter notebooks is the separate ipynb format. Just solving that alone is what most people would want. I don't think most people would appreciate having marimo specific syntax in their regular .py file. The whole selling point should be being able to run the same file, both as a notebook and as a script, depending on whether you're experimenting or finally running it when you're done experimenting.

A .py file with marimo specific syntax can't be used like a regular python file, even if it technically is, which defeats the purpose and rules it out as a viable option for me.

[Ubehebe]

I renamed this topic to reflect the most important issue (and to focus less on specific solutions). marimo's text format is good compared to other notebook formats, but I think it can be even better.

[chenller]

According to my understanding, marimo is a functional programming idea. Marimo describes the relationship between cells as the relationship between functions. Functional programming is necessary in the process of writing code, but using only functional programming can lack flexibility. The main function of the notebook is to enable the execution of Python code in blocks. All variables are global variables.
My point is, the basic execution logic is similar to a notebook, but some cells have 'reactive' properties. This means there are two types of cells. Variables in notebook-like cells should not have the same global variable names as those in cells with 'reactive' properties.
Scripts can be stored as:

# current marimo serialization
import marimo

app = marimo.App()

@app.cell
def __fun1():
  import pandas as pd
  return pd

@app.cell
def __fun2(pd):
  df = pd.DataFrame(...) # bad: `pd` param "shadows" top-level import
  return df
@app.cell_notebook
def __fun_notebook():
  notebook code 1
  notebook code 2
  ...
  pd=__fun1()
  df=__fun2(pd)
  notebook code n
  notebook code n+1
  ...

[akshayka]

Thanks for putting together thoughtful response.

I've thought about this and I've landed where you have:

deeper code editor integration (think "VS code" or vim plugin) could be set up to use marimo's LSP directly and overcome the issues described. marimo's current imports are actually good, because they are lazy.
I think this is the best solution.

Installing VS code/vim plugins wouldn't be too much friction, and would allow for a good editing experience without compromising on lazy imports and simplicity of the file format.

[dmadisetti]

@Ubehebe FYI we released top level definitions under a "setup" block, and have also started propagating user declared types in the signatures- hopefully making it a bit easier for computers and humans to read.

[gaardhus]

Propagation of types in the signatures would be so nice since it would greatly improve the experience of editing marimo notebooks in another editor :)

[dmadisetti]

This should work.

arg : dict[str, int] = {"key": 10}
some_arg1 = 0

Produces

@app.cell
def _():
    arg : dict[str, int] = {"key": 10}
    some_arg1 = 0
    return arg, some_arg1


@app.cell
def _(arg: dict[str, int], some_arg1):
    arg
    some_arg1
    return

It is much more tricky to do for implicit typing- but maybe in the future we can hook into ty or something.