How to develop shared arrays with workers in Pluto

[CMoebus]

Hi, I am working through Tom Kwong's book Hands-On Design Patterns and Best Practices with Julia. I try to implement Tom's ideas within Pluto notebooks. Up to chapter this went quite well. But now, in ch.06 I don't get ahead. It is the "shared array pattern" with the conept of parallel workers which generates lots of error messages generated by Pluto. GhatGPT and CoPilot couldn't help either. So I ask the community whether it is altogether possible to use Pluto and parallel workers at the same time.

Tom Kwong's code is here: https://github.com/PacktPublishing/Hands-on-Design-Patterns-and-Best-Practices-with-Julia/tree/master/Chapter06%2FSharedArrayPattern

Maybe somebody knows how to develop the pattern in the Pluto eco system. All the best Claus

[fonsp]

Can you share the error message and describe what did not work? Try to show:

• What you did
• What you want to see
• What you actually see

[CMoebus]

I try to implement a simplified version of Kwong's code. First I check my input, a 3D array called myArray. Then I generate a SharedArray, called mySharedArray. According to Kwong this array should be filled by workers with the content of myArray. Then some statistics should be computed for mySharedArray. I simplified the code to get the workers running. So I delayed some computations (input to mySharedArray and statistics). Now, mySharedArray should only be filled by random numbers exploiting the help of all workers.

The state of my implementation is here: https://uol.de/f/2/dept/informatik/ag/lks/download/Probabilistic_Programming/JULIA/Pluto.jl/DP_BP/DP_BP_Ch06_03_PerformancePatterns_III_PtII.html?v=1757505241

As you can see I get into trouble with workers. They need information which I can't provide. Probabably you know the secrets of workers and Pluto cells. All the best Claus

[CMoebus]

The generation code for the data matrix is here:
https://uol.de/f/2/dept/informatik/ag/lks/download/Probabilistic_Programming/JULIA/Pluto.jl/DP_BP/DP_BP_Ch06_03_PerformancePatterns_III_PtI.html?v=1757090690

[fonsp]

Hi @CMoebus !

Thanks for sharing your notebook! Now I have a better picture.

Looking at your code, I feel like Pluto is not the right environment for your project. Jupyter, VS Code notebooks (with the Julia extension), or (one day) BonitoBook would be a better fit for this. The reasons are:

• Distributed can be hard to use in Pluto. Most things will work, but you sometimes run into "workspaces" break serialization #1030 . In that case, you need careful understanding of Distributed and Serialization, and motivation to find a different solution for Pluto. You could also switch to Malt.jl if you want to do distributed programming, or work with multithreading instead.
• Imperative programming and print. Pluto works best with declarative/functional programming, rich output and Logging.

If your goal is to become more familiar with Julia then I would recommend taking a look at these other tools! Especially the VS Code Julia extension can be a really good tool to learn, it's also very useful when writing your own Julia packages in the future.

If you are doing something else, like developing a course in Julia, let me know!

[CMoebus]

Hi Fons,
Thank you for taking care of my problem combining several workers with Pluto. I try to explore your hints. I already used VSC with Julia extension. But I prefer to stay with Pluto. It's a great tool for explorative programming.

You asked whether I have some kind of Julia tutorial ? Yes, I implemented about 50% of SICP. I interrupted that in favor of Kwong's DPBP because I wanted to learn more Julia hacks. After this detour I plan to complete the Julia interpretation of SICP. I think that Julia is a great successor of Scheme.
All the best, Claus

[fonsp]

Alright! Hope it goes well und viel Spaß! Let me know it goes