Thoughts about OSX support & "common" dependencies

[oderwat]

I wonder what the status is on OSX support?

As an OSX user, I found that the eliasdb console won't work because of the missing implementation of parts of getch in the termutil package of the common module. So I quickly cloned that repository, used a replace statement in the go.mod file, and implemented a very minimal and unfinished Darwin version to play around a bit (not really worth a PR so far).

Next, in my journey, I found that the datamining-example copies over the eliasdb executable into the docker image. I also changed the build script to use a cross compiled version (this could be a small PR).

All in all, I could run all examples on my Mac (strafing in asteroids is cool), and I am thinking about the possibilities for using EliasDB for some projects.

But I read somewhere that ElisaDB was created from scratch using "no dependencies" but the gorilla/websocket. This is true if one does not count for the pretty large dependency on all of those packages below https://devt.de/krotik/common.

While those packages are really cool, they form a substantial part of EliasDB, and this is "just a private gogs server" where all of this is located. This will render the whole EliasDB useless for anybody using it if that repository vanishes. Wouldn't it be better to include the used parts as internal packages or add the whole "common" repo to GitHub?

[mladkau]

Hey there, thank you for raising this (and the PR) 🙏 - To your points:

• I don't have OSX available to me so I developed most of the support kind of "blind" 😅 . There should be proper getch() support for Windows and Linux and for other operating systems a fallback should kick in: https://devt.de/krotik/common/src/master/termutil/term.go#L181 - based on an error which is returned here: https://devt.de/krotik/common/src/master/termutil/getch/util.go#L19 - I wonder why this is not working 😕.
  If we could have a working Darwin implementation that would be AWESOME 🏆 ! Please note though that the code in the common repository is in the public domain - any code contribution would have the same requirements 🙏.

• The reason why the binary is copied are the binary distributions. The idea is that the examples can be included and will work without having a Go development environment. I saw the PR you raised - I am a bit confused, why would it not work under Debian? I just tried it on a Linux Mint (based on Ubuntu/Debian) using a binary distribution and it does work for me. Could you elaborate a bit on why you suggest to build the binary for the example?

• I can see your point about the common packages. I can certainly upload the common code to Github. However, I would be a bit reluctant to change the go.mod locations - I still see Github more as a distribution platform rather than a code repository (still have some bad memories of M$ not being very friendly with open source).

[oderwat]

Hey :) We may create some more OSX stuff in the future. To be honest: I love what EliasDB has to offer. It would be very cool to have this working for larger projects too.

• The original getchr() fallback does work and lets you type. But it panics when a password is requested (because it can't change echo). This may be easy to fix for you? As is, this makes the EliasDB console useless on OSX. So I added a (mostly wrong but working) implementation of getch() to play around with EliasDB. I have to confess that implementing a complete getch() on OSX may be over my head. Currently, I fool around with the golang.org/x/term raw terminal.
• I saw that common is public domain and appreciate this. For sure, we will "steal" stuff here and there :)
• I don't see how binary distributions are related. Suppose you would have an OSX binary distribution. The eliasdb executable would be compiled for OSX and will not work in the alpine docker image anyway. I have to confess that I did not look inside what you did for windows. But shouldn't this be the same problem?
• About the Alpine problem: I created a Parallels VM with Debian on my Mac and moved into that. When I compile eliasdb there and copy it into the docker file, I get "file not found" when executing it. I searched the internet and found that it is related to the missing glibc on alpine. ldd shows this. There may be solutions for a better way to cross-compile, but I solved it by changing the docker base for the eliasdb images to Debian.
• We have a similar problem but even more complicated. As a company, we have now adopted Jetbrains Space and used this to host all repositories. But Space does not include public repositories. We made it possible to go get modules anyway (rewriting servers in git makes this possible). But we (and I) also want to publish stuff. So we set up GitHub syncs. Which then leaves the question: What to import? The GitHub or the Space repo. For us, this is easy: Everything public has to use the GitHub version. But you can also import the public versions on gogs. Still, I think you should consider changing it to use also GitHub for this. This could be easy, though, changing the mod file slightly and add: replace devt.de/krotik/common => github.com/krotik/common v0.
• M$ concerns me too, but then, we all use GitHub, and until it "does not work" anymore, it is pretty nice to use.

[oderwat]

Additional information:

I researched how to actually cross-compile for the alpine Linux musl lib and ended up with this:

#!/bin/sh
cd "$(dirname "$0")"
export ROOT_PATH=`pwd`

# Build the collector component

CC=$(which musl-gcc) go build --ldflags '-w -linkmode external -extldflags "-static"' ../../../../cli/eliasdb.go
docker build --no-cache --build-arg cluster_id=1 --tag data-mining/eliasdb1 .
docker build --build-arg cluster_id=2 --tag data-mining/eliasdb2 .
docker build --build-arg cluster_id=3 --tag data-mining/eliasdb3 .

# Run an interactive shell on the build image with:
# docker run -it data-mining/eliasdb sh

To make this work on Debian, one needs to install musl-tools (apt-get install musl-tools) which installs musl-gcc.

Nearly the same can be done on Mac by using:

brew install FiloSottile/musl-cross/musl-cross

and creating a softlink

ln -s $(brew --PREFIX)/opt/musl-cross/bin/x86_64-linux-musl-gcc $(brew --PREFIX)/bin/musl-gcc (should work on Intel and Apple Silicon homebrew installations)

Or just using the x86_64-linux-musl-gcc instead of musl-gcc in the build.sh. One could extend the build.sh to check for those.

With this knowledge, I actually wonder what happens when not using musl-gcc on OSX, as the result of the "normal" cross-compile will work on alpine, while the Debian-based build executable was not. It seems that Go compiles on OSX statically against something "else" than glibc.

I also wonder why your Docker works on Linux. Which Linux distribution did you use for testing?

P.S.: The Linux binary with the musl lib should be statically linked and work on every Linux, glibc or musl or whatever.

Resources used:

• https://www.reddit.com/r/golang/comments/4cxrcv/whats_the_proper_way_to_build_golang_apps_for/
• https://github.com/FiloSottile/homebrew-musl-cross
• https://wiki.alpinelinux.org/wiki/Running_glibc_programs

[krotik]

I am using Linux Mint (20.1) - I think it works on Linux for me because the default Makefile sets: export CGO_ENABLED=0 🤔

[oderwat]

Oh yes. I see. I followed:

You can build EliasDB's executable with:
go build cli/eliasdb.go

I didn't even saw the Makefile till now. And I can confirm that it works in Debian when EliasDB was build through the Makefile. Well. Learned something... as usual :)

And (back on the Mac) of course, you can't use the Makefile on the Mac. I think I could extend the PR at the weekend and add stuff so it builds/examples on Mac and Linux. What do you think?

[oderwat]

Well.. https://devt.de/ is down right now while I am talking to a colleague about using eliasdb for something in production.

[krotik]

I though about your point about using github.com and I changed my mind. Since I already mirror eliasdb and ecal on github I could also go all the way and also mirror common. I've linked up all the code on github and there should be now no more code dependencies to devt.de on github (some of the documentation links stay however, e.g. build results).

[krotik]

I am not sure I follow with "can't use the Makefile on the Mac".
Re: building the examples - I would still like to aim for not having to build examples so the binary distributions can include examples which do not require a go build environment. I wonder if the data-mining example could use the image on DockerHub https://hub.docker.com/r/krotik/eliasdb which is a scratch image ...

[oderwat]

I think we may have just different ideas about how stuff "should" work. I expected running make to build a working executable for my system. As is, it builds a Linux executable. But then this works for building the examples ;-)

About the examples themselves: I think it depends if they are "examples of how to build something" or "showcases". For me, an example should let me create everything from scratch. Even the front end. Using the DockerHub image may help to demo them. For letting people use them as the basis for experiments, it may be better to make the build process and documentation "foolproof" when using different systems.

I think I would change the Makefile so that it detects the system and also creates the right executable for the demos. Probably even build/run everything from there and ditch the individual scripts (or call them from the Makefile).

But this is a lot of work and hard to do if you don't have all the target systems for testing. As I said, we may do some work in that direction, but we are still evaluating if we want to use EliasDB in production.