nix-copy-closure, nix copy doesn't work

[irisjae]

I've been trying to use nix-copy-closure to copy the build products, and very strangely an error is complaining that the ssh file doesn't exist. After some debugging, I've determined that most options don't matter, and using a minimal nix-portable nix-copy-closure command triggers the error:
./nix-portable-aarch64 nix-copy-closure --to a@b

error: unable to execute 'ssh': No such file or directory

(Yes, ssh exists on my machine).

I'm running nix-portable-012, and this seems to occur whether I use NP_RUNTIME nix, bwrap, or proot.

This is puzzling, as judging from the error message, the external program call comes from src/libstore/ssh.cc (or more unlikelily, from src/nix/run.cc) of the NixOS/nix repository, and they do use execvpe() which should get ssh from my PATH. I tried adding fake binaries in NP_LOCATION and corresponding PATH entries, and that didn't work either.

[irisjae]

Okay, I must have done something wrong with my fake ssh binary testing. It appears the problem was indeed ssh not being available in the bwrap context. After I put PATH="$(dirname $(nix-portable-aarch64 nix shell nixpkgs#openssh -c which ssh)):$PATH", it managed to find ssh.

However, it still runs into problems with Couldn't execute bash -c "echo started", because nix-copy-closure runs ssh ... -oLocalCommand=echo started, which tries to execute $SHELL whose value is no longer valid in the bwrap context. So, I tacked on SHELL=$(nix-portable-aarch64 nix shell nixpkgs#bash -c which bash) as well and the thing worked.

This means nix-copy-closure and nix copy basically won't work in nix-portable by default, unless we add ssh and bash (or some other shell) to the nix-portable environment, somewhat similarly to what we do for git. Let me see if I can add this.

For some more context, the reason that I was making use of nix-copy-closure was because I was attempting to use nixos-rebuild --target-host from a machine with a small primary disk (so, I couldn't just use nixos-rebuild --target-host since my Nix store is too small, and needed to temporarily use another disk using nix-portable; the target is a Pi -- I wouldn't want to build on that).

If anyone else tries to also use nix-portable with nixos-rebuild, note that the nixos-rebuild script does some things that make using nix-portable somewhat harder by default:

1. It adds its own Nix versions to PATH (can be disabled with _NIXOS_REBUILD_REEXEC=1; might as well add the --fast flag)
2. It forces SHELL to be some system shell (so defeats setting a custom $SHELL to fix nix-copy-closure)
3. It creates a SSH socket at TMPDIR (assumed to be shared by nix-copy-closure and the nixos-rebuild script, which is not true if nix-copy-closure comes from nix-portable).

As a workaround, I made a copy of the nixos-rebuild script and commented out the line forcing $SHELL and removed the ssh option passing the socket to nix-copy-closure ( -o ControlMaster=auto -o ControlPath=$tmpDir/ssh-%n -o ControlPersist=60) (it does make ssh prompt for passwords a few more times, but it works) (we can make it work by adding some more path magic).

[irisjae]

With something like #150, we would be able to run nixos-rebuild unmodified using nix-portable! (By adding nix binaries provided by nix-portable to the PATH, and also overriding the ssh socket path by adding mktemp binaries in PATH)