`run_uvloop` not loaded | NIX

[elliott-farrall]

I (among others) would be very interested in seeing this included in nixpkgs. I had a go at writing a derivation myself but am having trouble with run_uvloop.

I managed to get the installer to start but it crashes when trying to load run_uvloop from ~/.config/wljs-notebook/wl_packages/KirillBelov_CSockets/LibraryResources/Linux-x86-64-v<version>/usockets.so. It tries version 7, 6 (failing to load on both) and then 5 (failing since it does not exist).

I have linked in libuv by wrapping the executable like so:

{
  lib,
  stdenv,
  fetchzip,
  makeWrapper,
  glib,
  ffmpeg,
  nss,
  nspr,
  dbus,
  atk,
  cups,
  libdrm,
  gtk3,
  pango,
  cairo,
  xorg,
  libgbm,
  expat,
  libxkbcommon,
  alsa-lib,
  libuv,
  libGL
}:

stdenv.mkDerivation rec {
  pname = "wolfram-js-frontend";
  version = "2.7.0";

  src = fetchzip {
    url = "https://github.com/JerryI/wolfram-js-frontend/releases/download/v${version}/wljs-notebook-${version}-x64-gnulinux.zip";
    hash = "sha256-dvJ9P4+Ym7seAS0gn+BINlsL4+OsWqOSWTyYSDoHAxs=";
    stripRoot = false;
  };

  nativeBuildInputs = [ makeWrapper ];

  installPhase = ''
    runHook preInstall

    makeWrapper $src/wljs-notebook $out/bin/wolfram-js-frontend \
      --prefix LD_LIBRARY_PATH : "${lib.makeLibraryPath [ glib ffmpeg nss nspr dbus atk cups libdrm gtk3 pango cairo xorg.libX11 xorg.libXcomposite xorg.libXdamage xorg.libXext xorg.libXfixes xorg.libXrandr libgbm expat xorg.libxcb libxkbcommon alsa-lib libuv libGL ]}"

    runHook postInstall
  '';

  meta = {
    description = "Freeware notebook environment for Wolfram Language written in Javascript";
    homepage = "https://github.com/JerryI/wolfram-js-frontend";
    license = lib.licenses.gpl3Only;
    # maintainers = with lib.maintainers; [ ];
    mainProgram = "wolfram-js-frontend";
    platforms = lib.platforms.all;
  };
}

I understand that not many people use nixpkgs but any pointers on what might be causing this would be greatly appreciated.

[JerryI]

Hi @elliott-farrall
Thank you for your efforts!

The way how we build sockets library is automated in Github Actions and it does this command on x86 Ubuntu

generated by CreateLibrary function

/usr/bin/gcc -shared -o "/root/test/CSocketListener/LibraryResources/Linux-x86-64-v7/Working-wljs-343749-607313088-1/usockets.so" \
-m64  -fPIC -O2  -I"/usr/local/Wolfram/WolframEngine/14.2/SystemFiles/IncludeFiles/C" \
-I"/usr/local/Wolfram/WolframEngine/14.2/SystemFiles/Links/MathLink/DeveloperKit/Linux-x86-64/CompilerAdditions" \
-I"/usr/include" "/root/test/CSocketListener/Source/usockets.c"  \
-L"/usr/local/Wolfram/WolframEngine/14.2/SystemFiles/Links/MathLink/DeveloperKit/Linux-x86-64/CompilerAdditions" \
-L"/usr/local/Wolfram/WolframEngine/14.2/SystemFiles/Libraries/Linux-x86-64"  \
"/usr/lib/x86_64-linux-gnu/libuv.so"  2>&1

I am not in expert in GCC, but I guess, it should statically link uv. If not, may be it checks LD_LIBRARY_PATH...

Another possibility, it looks like Wolfram Engine itself uses dynamic libs located in its installation directory. At least $LibraryPath gives a big list. (sorry if my understanding of linking and your wrapper is too off :0)

[elliott-farrall]

I'm not very familiar with loading libraries in wolfram so this has been a bit of a learning experience. I did some experimenting and realised that my install of wolfram just could not find libuv, which is needed for usockets.so to load properly.

I checked $LibraryPath and found that it includes a path in ~/.WolframEngine so I added libuv there with no luck. It turns out that the kernel also has to load the library using LibraryLoad["libuv"] so I added this to my Kernels init.m. In future, I'll probably adjust my derivation to make sure that wolfram-js-frontend has a patched version of wolfram-engine in its path with the library paths and init command setup.

After all this, I can now get version 2.5.8 to run using either the standard or offline AppImage. I've also managed to get the derivation for 2.7.0 to run successfully a couple of times but sometimes the app window doesn't open despite the fact that there are no errors. I am not sure what is causing this but it may be to do with my current janky implementation of getting the bundle files.

Currently, my derivation does not create the required bundles files so I fetch them seperately

bundle = fetchzip {
  url = "https://github.com/JerryI/wolfram-js-frontend/releases/download/v${version}/wljs-notebook-${version}-arm64-gnulinux.zip";
  hash = "sha256-LqmHFCxQzwWlfuLcxZo5iXXAob4aZl4uWh7P+pffpHU=";
  stripRoot = false;
};

and then copy over the files during install

cd ${bundle}/resources/app/bundle
find . -type f -exec install -Dm 755 "{}" "$out/dist/resources/app/bundle/{}" \;

These files then appear in ~/.config/wljs-notebook but with restricted permissions (probably a nix specific thing).

I should probably create the bundle files during build but have no experience with electron so not sure how to do this. At the moment I just build using

npm exec electron-builder -- \
  --dir \
  -c.electronDist=${electron.dist} \
  -c.electronVersion=${electron.version}

[JerryI]

Amazing work, @elliott-farrall !
I did not expect the rabbit-hole would be that deep.

Regarding LibraryLoad["libuv"] on my mac machine it returns $Failed (could be only Linux feature), nevertheless we can just include it directly to CSockets library in the next release.

It is still puzzling for me, why we can't force CreateLibrary to link statically libuv even though we specify explicitly the path to a library file (which should force static linking on gcc if i am not mistaken)

I've also managed to get the derivation for 2.7.0 to run successfully a couple of times but sometimes the app window doesn't open despite the fact that there are no errors.

If you could dump the logs (via launcher or from the console), I would love to help!

These files then appear in ~/.config/wljs-notebook but with restricted permissions (probably a nix specific thing).

This may cause some issues with modules (~/.config/wljs-notebook/wljs_packages) if an update is available...

Also a question, can I try your NIX image (or package) you made on my LinuxX86 machine? If it requires for you that many steps, we can try to include it to our building pipeline for the releases.

[elliott-farrall]

`Regarding LibraryLoad["libuv"] on my mac machine it returns $Failed (could be only Linux feature), nevertheless we can just include it directly to CSockets library in the next release.

LibraryLoad["some_lib"] checks paths in $LibraryPath for some_lib.so so if you haven't installed libuv.so into a path in $LibraryPath then it will fail. LibraryLoad should work on all platforms but the libraries will be .dylib files on MacOS instead of .so files.

If you could dump the logs (via launcher or from the console), I would love to help!

What is the best way to get the logs? I don't think much of interest gets outputted to the terminal and I can't seem to copy from the window with the Wolfram logs. In case it is of any use, here is the terminal output for wljs-notebook ~.

Mica-Electron only works on Windows!
Mica-Electron: Disabled
{
  new_file: [ 'Cmd+N', 'Ctrl+N' ],
  new_file_folder: [ 'Cmd+Shift+N', 'Ctrl+Shift+N' ],
  new_quick_file: [ 'Cmd+Shift+N', 'Ctrl+Shift+N' ],
  overlay: [ 'Shift+Alt+Space', 'Shift+Alt+Space' ],
  open_file: [ 'Cmd+O', 'Ctrl+O' ],
  save: [ 'Cmd+S', 'Ctrl+S' ],
  find: [ 'Cmm+F', 'Ctrl+F' ],
  toggle_cell: [ 'Cmd+2', 'Alt+2' ],
  delete_cell: [ "Cmd+'", 'Ctrl+d' ],
  abort: [ 'Cmd+.', 'Alt+.' ],
  evaluate_init: [ 'Cmd+i', 'Alt+i' ],
  clear_outputs: [ 'Cmd+u', 'Alt+u' ],
  evaluate_all: [ 'Cmd+a', 'Alt+a' ],
  close: [ 'Cmd+q', 'Alt+q' ]
}
Electron >> starting powersafe blocker
<|"Autostart" -> True|>

{ Autostart: true }
<|"Autostart" -> True|>

{ Autostart: true }
APPIMAGE env is not defined, current application is not an AppImage
checkForUpdatesAndNotify called, downloadPromise is null
[1988814:0331/151720.629588:ERROR:browser_main_loop.cc(278)] Gtk: gtk_widget_get_scale_factor: assertion 'GTK_IS_WIDGET (widget)' failed
Cli is not installed
skipped because of a user
TRY
Started! app

Hee uvTask: 2

Open first window
OPEN a FILE WIN/Linux


ReplaceAll::reps: 
   {Join[CoffeeLiqueur`Notebook`AppExtensions`SidebarIcons, 
      {wln -> <svg class="w-4 h-5 text-gray-800" viewBox="2 0 20 24"
          stroke="current<<559>> 4V20M12 11H16M12 8H16"></path></svg>, <<1>>}]
      } is neither a list of replacement rules nor a valid dispatch table, and
     so cannot be used for replacing.

General::stop: Further output of ReplaceAll::reps
     will be suppressed during this calculation.

Keys::invrl: The argument CoffeeLiqueur`Notebook`AppExtensions`SidebarIcons
     is not a valid Association or a list of rules.

General::stop: Further output of Keys::invrl
     will be suppressed during this calculation.

Join::heads: Heads Keys and List at positions 1 and 2
     are expected to be the same.


General::stop: Further output of Join::heads
     will be suppressed during this calculation.


OptionValue::optnf: 
   Option name AppEvent not found in defaults for Component$47677.

Default router

>> InputNamePacket[In[3]:= ]

>> InputNamePacket[In[4]:= ]

New connection for 5
hash >> allocate 140387582771440 with offset 0
hash >> 792
hash >> ok!
uv start readingbroadcast NewClient state!
New connection for 6
hash >> allocate 140387582771696 with offset 0
hash >> 950
hash >> ok!
uv start readingbroadcast NewClient state!
New connection for 7
hash >> allocate 140387582771952 with offset 0
hash >> 1108
hash >> ok!
uv start readingbroadcast NewClient state!
New connection for 8
hash >> allocate 140387582772208 with offset 0
hash >> 1266
hash >> ok!
uv start readingbroadcast NewClient state!
New connection for 9
hash >> allocate 140387582772464 with offset 0
hash >> 1425
hash >> ok!
uv start readingbroadcast NewClient state!
Checking\
 
>   /home/elliott/.config/wljs-notebook/wljs_packages/Styles/Custom.css...
Checking /home/elliott/.config/wljs-notebook/Assets/Styles/Custom.css...

Checking\
 
>   /home/elliott/.config/wljs-notebook/wljs_packages/Styles/Optimized.css...
Checking /home/elliott/.config/wljs-notebook/Assets/Styles/Optimized.css...

Checking\
 
>   /home/elliott/.config/wljs-notebook/wljs_packages/wljs-cells/src/module.j\
 
>   s...

Checking\
 
>   /home/elliott/.config/wljs-notebook/wljs_packages/wljs-sharedlib-mk/dist/\
 
>   kernel.js...

Checking\
 
>   /home/elliott/.config/wljs-notebook/wljs_packages/wljs-html-support/src/s\
 
>   tyle.css...

Checking\
 
>   /home/elliott/.config/wljs-notebook/wljs_packages/wljs-wlx-support/src/st\
 
>   yles.css...

Checking\
 
>   /home/elliott/.config/wljs-notebook/wljs_packages/wljs-sharedlib-three/di\
 
>   st/kernel.js...

Checking\
 
>   /home/elliott/.config/wljs-notebook/wljs_packages/wljs-magic-support/src/\
 
>   kernel.js...

Checking\
 
>   /home/elliott/.config/wljs-notebook/wljs_packages/wljs-html-support/src/k\
 
>   ernel.js...

Checking\
 
>   /home/elliott/.config/wljs-notebook/wljs_packages/wljs-mermaid-support/di\
 
>   st/kernel.js...

Checking\
 
>   /home/elliott/.config/wljs-notebook/wljs_packages/wljs-plotly/dist/kernel\
 
>   .js...

Checking\
 
>   /home/elliott/.config/wljs-notebook/wljs_packages/wljs-wlx-support/src/ke\
 
>   rnel.js...

Checking\
 
>   /home/elliott/.config/wljs-notebook/wljs_packages/wljs-excalidraw/dist/ke\
 
>   rnel.js...

Checking\
 
>   /home/elliott/.config/wljs-notebook/wljs_packages/wljs-print-redirect-fix\
 
>   /kernel.js...

Checking\
 
>   /home/elliott/.config/wljs-notebook/wljs_packages/wljs-snippets/dist/kern\
 
>   el.js...

Checking\
 
>   /home/elliott/.config/wljs-notebook/wljs_packages/wljs-wxf-accelerator/ov\
 
>   erride.js...

>> InputNamePacket[In[5]:= ]

>> InputNamePacket[In[6]:= ]

>> InputNamePacket[In[7]:= ]

>> InputNamePacket[In[8]:= ]

>> InputNamePacket[In[9]:= ]

>> InputNamePacket[In[10]:= ]

>> InputNamePacket[In[11]:= ]

>> InputNamePacket[In[12]:= ]


Hee uvTask: 1

>> InputNamePacket[In[13]:= ]

>> InputNamePacket[In[14]:= ]

>> InputNamePacket[In[15]:= ]

>> OutputNamePacket[Out[15]= ]

>> ReturnTextPacket[{Interpretation, InterpretationBox}]

>> InputNamePacket[In[16]:= ]

>> ReturnPacket[Null]

>> LocalKernel >> local kernel link connected!

>> SocketObject[4ebba5fa-e305-4bb4-b62a-09d96cfbce4f]

>> Created! Spinner

Init Kernel!!!

>> Loading into Kernel... wljs-plotly/src/Kernel.wl
>> CoffeeLiqueur`Notebook`LocalKernel`Private`LocalKernelObject[
 
>    CoffeeLiqueur`Notebook`LocalKernel`Private`LocalKernelObject`$23]
>> LTPTransport[SocketObject[4ebba5fa-e305-4bb4-b62a-09d96cfbce4f]]

>> Loading into Kernel... wljs-neuralnet/Kernel.wl
>> CoffeeLiqueur`Notebook`LocalKernel`Private`LocalKernelObject[
 
>    CoffeeLiqueur`Notebook`LocalKernel`Private`LocalKernelObject`$23]
>> LTPTransport[SocketObject[4ebba5fa-e305-4bb4-b62a-09d96cfbce4f]]

>> Loading into Kernel... wljs-print-redirect-fix/Kernel.wl
>> CoffeeLiqueur`Notebook`LocalKernel`Private`LocalKernelObject[
 
>    CoffeeLiqueur`Notebook`LocalKernel`Private`LocalKernelObject`$23]
>> LTPTransport[SocketObject[4ebba5fa-e305-4bb4-b62a-09d96cfbce4f]]

>> Loading into Kernel... wljs-video/src/Kernel.wl
>> CoffeeLiqueur`Notebook`LocalKernel`Private`LocalKernelObject[
 
>    CoffeeLiqueur`Notebook`LocalKernel`Private`LocalKernelObject`$23]
>> LTPTransport[SocketObject[4ebba5fa-e305-4bb4-b62a-09d96cfbce4f]]

>> Loading into Kernel... wljs-wxf-accelerator/Override.wl
>> CoffeeLiqueur`Notebook`LocalKernel`Private`LocalKernelObject[
 
>    CoffeeLiqueur`Notebook`LocalKernel`Private`LocalKernelObject`$23]
>> LTPTransport[SocketObject[4ebba5fa-e305-4bb4-b62a-09d96cfbce4f]]

>> Starting WS link

>> LocalKernel Init >> Normal

nothing to do...
Kernel init...

>> LocalKernel Init >> Normal

nothing to do...

>> LocalKernel Init >> Normal

>> LocalKernel Init >> Normal

Ok!

>> ReturnPacket[Null]

>> ReturnPacket[Null]

>> ReturnPacket[Null]

>> ReturnPacket[Null]

>> ReturnPacket[Null]

>> ReturnPacket[Null]

>> ReturnPacket[Null]

>> ReturnPacket[Null]

>> ReturnPacket[Null]

Open http://127.0.0.1:20560 in your browser

Open http://127.0.0.1:20560 in your browser

Open http://127.0.0.1:20560 in your browser

Open http://127.0.0.1:20560 in your browser

Also a question, can I try your NIX image (or package) you made on my LinuxX86 machine? If it requires for you that many steps, we can try to include it to our building pipeline for the releases.

Currently my derivation looks like

{ lib
, buildNpmPackage
, fetchFromGitHub
, fetchzip
, makeWrapper
, electron
, python312
, libGL
, polkit
, makeDesktopItem
, copyDesktopItems
}:

buildNpmPackage rec {
  pname = "wolfram-js-frontend";
  version = "2.7.0";

  src = fetchFromGitHub {
    owner = "JerryI";
    repo = "wolfram-js-frontend";
    rev = "v${version}";
    hash = "sha256-bwVhyLfWDh3Pyli8LF0oHn2BOxl2w7G9olg6YGqZNlg=";
  };

  npmDepsHash = "sha256-spEzasCBhBUOUW1g8rIxy0xmRdVdexmq5LS8K/JS/Og=";
  npmFlags = [ "--force" ]; # ignore platform incompatibility for some deps

  nativeBuildInputs = [
    (python312.withPackages (ps: with ps; [ pip setuptools ]))
    makeWrapper
    copyDesktopItems
  ];

  env.ELECTRON_SKIP_BINARY_DOWNLOAD = true;

  buildPhase = ''
    runHook preBuild

    npm exec electron-builder -- \
      --dir \
      -c.electronDist=${electron.dist} \
      -c.electronVersion=${electron.version}

    runHook postBuild
  '';

  installPhase =
    let
      bundle = fetchzip {
        url = "https://github.com/JerryI/wolfram-js-frontend/releases/download/v${version}/wljs-notebook-${version}-arm64-gnulinux.zip";
        hash = "sha256-LqmHFCxQzwWlfuLcxZo5iXXAob4aZl4uWh7P+pffpHU=";
        stripRoot = false;
      };
    in
    ''
      runHook preInstall

      mkdir -p $out
      cp -r dist/*-unpacked $out/dist

      cd ${bundle}/resources/app/bundle
      find . -type f -exec install -Dm 755 "{}" "$out/dist/resources/app/bundle/{}" \;

      runHook postInstall
    '';

  postInstall = ''
    for sz in 512; do
      mkdir -p $out/share/icons/hicolor/''${sz}x''${sz}/apps
      cp $src/Electron/build/file/''${sz}x''${sz}.png $out/share/icons/hicolor/''${sz}x''${sz}/apps/wljs-notebook.png
    done
  '';

  postFixup = ''
    makeWrapper $out/dist/wljs-notebook $out/bin/wljs-notebook \
      --add-flags "--no-sandbox" \
      --add-flags "\''${NIXOS_OZONE_WL:+\''${WAYLAND_DISPLAY:+--ozone-platform-hint=auto --enable-features=WaylandWindowDecorations --enable-wayland-ime=true}}" \
      --prefix LD_LIBRARY_PATH : "${lib.makeLibraryPath [ libGL ]}"

    substituteInPlace $out/dist/resources/app/node_modules/sudo-prompt/index.js \
      --replace "/usr/bin/pkexec" "${polkit}/bin/pkexec"
  '';

  desktopItems = [
    (makeDesktopItem {
      name = "wljs-notebook";
      icon = "wljs-notebook";
      exec = "wljs-notebook %f";
      desktopName = "Wolfram";
      categories = [ "Development" ];
    })
  ];

  meta = {
    description = "Freeware notebook environment for Wolfram Language written in Javascript";
    homepage = "https://github.com/JerryI/wolfram-js-frontend";
    license = lib.licenses.gpl3Only;
    # maintainers = with lib.maintainers; [ ];
    mainProgram = "wljs-notebook";
    platforms = lib.platforms.all;
  };
}

You would need to install nix (vm and container images are available) to build and run the package. Once built the packages does run fairly consistently now. However, the UI is mostly non-functional.

• Opening files shows nothing.
• Pressing 'New Notebook' does nothing
• Clicking folders in sidebar does nothing
• Files in side bar have bugged icons
• Basically all UI buttons do nothing
• No kernels

If I uninstall and then run the AppImage instead then everything works fine but the AppImage is not up to date and I think it would be nicer to run things natively.

I think one of the biggest challenges here is that nix uses a read-only file system for packages. This probably won't play nicely with the builtin update system. I believe it's also what causes restricted permissions for some of the files in ~/.config as they get copied from the packages directory which is read-only.

PS: Sorry for long post. Wouldn't let me attach logs separately.

[telestew]

Hello, I know this issue has been targeted at the NIX package but I am encountering a similar issue in the Linux development environment on a Chromebook. For some reason I can't copy-paste the launcher text output, but I've included a (sequence of) screen cap(s) of it.

I'll add that originally wolframscript itself was failing to activate because it was choking on some other item (shown below). I've included KPM_AFFINITY=disabled in /etc/environment which fixed that. Clearly wljs-notebook can start the Wolfram kernel so I'm not sure why it reports license error after failing to load run_uvloop (unless it's related to OPM in some way; not familiar territory for me).

[JerryI]

Hello, I know this issue has been targeted at the NIX package but I am encountering a similar issue in the Linux development environment on a Chromebook. For some reason I can't copy-paste the launcher text output, but I've included a (sequence of) screen cap(s) of it.

I'll add that originally wolframscript itself was failing to activate because it was choking on some other item (shown below). I've included KPM_AFFINITY=disabled in /etc/environment which fixed that. Clearly wljs-notebook can start the Wolfram kernel so I'm not sure why it reports license error after failing to load run_uvloop (unless it's related to OPM in some way; not familiar territory for me).

Hello @telestew !
This license error is some non-sense single error message, that is generated on any critical exception by WL Engine and has nothing to do with licensing :)

Regrading uv_loop, could you, please try to evaluate in wolfram session

LibraryLoad["libuv"]

If it returns something other than $Failed, one can try to include it to init.m as @elliott-farrall did. And in the next release we just put this line to one of our core packages.

[JerryI]

@elliott-farrall thank you for your answer.

I think one of the biggest challenges here is that nix uses a read-only file system for packages. This probably won't play nicely with the builtin update system. I believe it's also what causes restricted permissions for some of the files in ~/.config as they get copied from the packages directory which is read-only.

Is there a conventional read/write directory for apps in NIX env? It is not a big issue to change it to basically any installation path in electron wrapper.

[telestew]

No dice:

In[2]:= LibraryLoad["libuv"]                                                                                                                      

LibraryFunction::notfound: Symbol libuv not found.

Out[2]= $Failed      

[JerryI]

No dice:

In[2]:= LibraryLoad["libuv"]                                                                                                                      

LibraryFunction::notfound: Symbol libuv not found.

Out[2]= $Failed      

Ok. I found a working linux machine in the office. We will try to set both on it:

• simulate absence of libuv
• nix env

[JerryI]

No dice:

In[2]:= LibraryLoad["libuv"]                                                                                                                      

LibraryFunction::notfound: Symbol libuv not found.

Out[2]= $Failed      

Could you, please, download this file https://gist.github.com/JerryI/8428be0b8bf3b2b631d14c1422fce377

And then run it via command line

wolframscript -f test.wls

The console log will help us a lot