invenio-rdm.yaml problems from the setup generated by `invenio_rdm_machine.bash vanilla`

[rsdoiel]

The vanilla invenio-rdm machine's cloud init is still not quire right. I'm breaking on the following step when logged in as ubuntu user in /Sites/vanilla

ubuntu@invenio-rdm:/Sites/vanilla$ invenio-cli containers start --lock --build --setup
Starting InvenioRDM instance...
Locking python dependencies...
Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/cli/options.py", line 58, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/click/decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/cli/command.py", line 344, in lock
    do_lock(
  File "/usr/local/lib/python3.9/dist-packages/pipenv/routines/lock.py", line 34, in do_lock
    lockfile = project.lockfile(categories=lockfile_categories)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/project.py", line 775, in lockfile
    plette.Pipfile.load(pf), categories=categories
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/plette/pipfiles.py", line 65, in load
    return cls(data)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/plette/models/base.py", line 8, in __init__
    self.validate(data)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/plette/pipfiles.py", line 44, in validate
    klass.validate(data[key])
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/plette/models/base.py", line 118, in validate
    cls.item_class.validate(d)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/plette/models/packages.py", line 35, in validate
    PackageSpecfiers.validate(data)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/plette/models/base.py", line 49, in validate
    raise DataValidationError(f"Invalid type for field {k}: {type(data[k])}")
pipenv.vendor.plette.models.base.DataValidationError: Invalid type for field editable: <class 'pipenv.vendor.tomlkit.items.String'>
Failed to start containerized instance.

I think this is failing due to a Node/npm "package.json" file not existing but the error message from pipenv seems to point to a TOML configuration issue. If the problem is TOML related I am not sure which file it is in.

[rsdoiel]

Updated the Python install to set the default python to 3.9 to ensure that calling invenio-rdm container start --lock --build --setup wasn't actually getting confused. Now the error message is slightly difference (gets further into the process).

Starting InvenioRDM instance...
Locking python dependencies...
Creating a virtualenv for this project...
Pipfile: /Sites/vanilla/Pipfile
Using /usr/bin/python3 (3.9.19) to create virtualenv...
⠼ Creating virtual environment...created virtual environment CPython3.9.19.final.0-64 in 309ms
  creator CPython3Posix(dest=/home/ubuntu/.local/share/virtualenvs/vanilla-FOfWfWzA, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/ubuntu/.local/share/virtualenv)
    added seed packages: pip==24.0, setuptools==69.5.1, wheel==0.43.0
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment!
Virtualenv location: /home/ubuntu/.local/share/virtualenvs/vanilla-FOfWfWzA
Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/cli/options.py", line 58, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/click/decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/cli/command.py", line 344, in lock
    do_lock(
  File "/usr/local/lib/python3.9/dist-packages/pipenv/routines/lock.py", line 34, in do_lock
    lockfile = project.lockfile(categories=lockfile_categories)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/project.py", line 775, in lockfile
    plette.Pipfile.load(pf), categories=categories
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/plette/pipfiles.py", line 65, in load
    return cls(data)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/plette/models/base.py", line 8, in __init__
    self.validate(data)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/plette/pipfiles.py", line 44, in validate
    klass.validate(data[key])
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/plette/models/base.py", line 118, in validate
    cls.item_class.validate(d)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/plette/models/packages.py", line 35, in validate
    PackageSpecfiers.validate(data)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/plette/models/base.py", line 49, in validate
    raise DataValidationError(f"Invalid type for field {k}: {type(data[k])}")
pipenv.vendor.plette.models.base.DataValidationError: Invalid type for field editable: <class 'pipenv.vendor.tomlkit.items.String'>
Failed to start containerized instance.

failed to complete: invenio-cli containers start --lock --build --setup
ubuntu@invenio-rdm:~$ 

[tmorrell]

This isn't a system setup problem, it's this bug in the Pipfile inveniosoftware/cookiecutter-invenio-rdm#283 and the new pipenv version. I emailed about it last week.

You can either manually make the change in the Pipfile or delete your local setup and re-run invenio-cli init -c master.

I also would use the local install commands invenio-cli install and invenio-cli services setup instead of the container commands, since the fully containerized version is hard to work with.

[rsdoiel]

NOTE: to self, per Tom, on line 11 editable="True", should be editable=true in Pipefile. I can use sed to fix that.

Now I need to understand the build processing as /bin/sh -c pipenv install --deploy --system is failing.

Building images...
[+] Building 1.1s (14/41)                                                                                                                                      docker:default
 => [worker internal] load build definition from Dockerfile                                                                                                              0.0s
 => => transferring dockerfile: 1.09kB                                                                                                                                   0.0s
 => [worker internal] load .dockerignore                                                                                                                                 0.0s
 => => transferring context: 86B                                                                                                                                         0.0s
 => [web-api internal] load metadata for registry.cern.ch/inveniosoftware/almalinux:1                                                                                    0.6s
 => [web-api internal] load build definition from Dockerfile                                                                                                             0.0s
 => => transferring dockerfile: 1.09kB                                                                                                                                   0.0s
 => [web-api internal] load .dockerignore                                                                                                                                0.0s
 => => transferring context: 86B                                                                                                                                         0.0s
 => [web-ui internal] load build definition from Dockerfile                                                                                                              0.0s
 => => transferring dockerfile: 1.09kB                                                                                                                                   0.0s
 => [web-ui internal] load .dockerignore                                                                                                                                 0.0s
 => => transferring context: 86B                                                                                                                                         0.0s
 => [worker  1/11] FROM registry.cern.ch/inveniosoftware/almalinux:1@sha256:66cfd9d08441708169397c0243c83097c3585638932c24c66ce93bbd053dc0d1                             0.0s
 => [web-api internal] load build context                                                                                                                                0.2s
 => => transferring context: 19.92MB                                                                                                                                     0.2s
 => [web-ui internal] load build context                                                                                                                                 0.2s
 => => transferring context: 19.92MB                                                                                                                                     0.2s
 => [worker internal] load build context                                                                                                                                 0.2s
 => => transferring context: 19.92MB                                                                                                                                     0.2s
 => CACHED [web-api  2/11] COPY site ./site                                                                                                                              0.0s
 => [web-api  3/11] COPY Pipfile Pipfile.lock ./                                                                                                                         0.2s
 => ERROR [web-api  4/11] RUN pipenv install --deploy --system                                                                                                           0.1s
------                                                                                                                                                                        
 > [web-api  4/11] RUN pipenv install --deploy --system:
0.131 exec /bin/sh: exec format error
------
failed to solve: process "/bin/sh -c pipenv install --deploy --system" did not complete successfully: exit code: 1
Failed to start containerized instance.

failed to complete: invenio-cli containers start --lock --build --setup

[rsdoiel]

Another note to self. The "container" option is broken. It really means "all container" option. When you choose a "local" install it still uses containers for the parts (E.g. Postgres, Redis, etc).

Martin Fenner has a good setup for the full containerized option see https://starter.front-matter.io/

[rsdoiel]

OK, now I can install a local instance in the Multipass VM. The invenio-cli run command seems to work. The command invenio-cli services startinvenio-cli services stop` work.

Now I need to wrap up the install setting things up to work like our production system and with the related steps.

• generate the *.service files for systemctl to load then use

[rsdoiel]

Note to self: I have invenio "local" install working reliably. I need to finish adding additional files like those to configure systemd management, web server configuration, etc. These are partially implemented with today's code commit - 6f09cfd

[rsdoiel]

Still tracking down the some NginX vars that needed escaping in the Bash scripts. Commit hash - 68da3d8