Failed to Load Tokenizers

[warren-lorenz]

Hi All,

I'm trying to run this locally. Right after I run docker build -t video-summarizer-using-watson:v1.0 . I'm having an issue that's been blocking me. I have Python3 installed version 3.11.1, pip version 22.3.1, and yet I'm getting an error that my tokenizers won't build due to a rust compilation issue. I've also installed rustup as per the error handling, but that didn't solve the issue.

Any advice on how to move forward?

My console is showing me this error:

`#12 30.39 copying py_src/tokenizers/decoders/init.pyi -> build/lib.linux-aarch64-cpython-39/tokenizers/decoders
#12 30.39 copying py_src/tokenizers/normalizers/init.pyi -> build/lib.linux-aarch64-cpython-39/tokenizers/normalizers
#12 30.39 copying py_src/tokenizers/pre_tokenizers/init.pyi -> build/lib.linux-aarch64-cpython-39/tokenizers/pre_tokenizers
#12 30.39 copying py_src/tokenizers/processors/init.pyi -> build/lib.linux-aarch64-cpython-39/tokenizers/processors
#12 30.39 copying py_src/tokenizers/trainers/init.pyi -> build/lib.linux-aarch64-cpython-39/tokenizers/trainers
#12 30.39 copying py_src/tokenizers/tools/visualizer-styles.css -> build/lib.linux-aarch64-cpython-39/tokenizers/tools
#12 30.39 running build_ext
#12 30.39 running build_rust
#12 30.39 error: can't find Rust compiler
#12 30.39
#12 30.39 If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.
#12 30.39
#12 30.39 To update pip, run:
#12 30.39
#12 30.39 pip install --upgrade pip
#12 30.39
#12 30.39 and then retry package installation.
#12 30.39
#12 30.39 If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.
#12 30.39 [end of output]
#12 30.39
#12 30.39 note: This error originates from a subprocess, and is likely not a problem with pip.
#12 30.39 ERROR: Failed building wheel for tokenizers
#12 30.39 Building wheel for sacremoses (setup.py): started
#12 30.62 Building wheel for sacremoses (setup.py): finished with status 'done'
#12 30.62 Created wheel for sacremoses: filename=sacremoses-0.0.53-py3-none-any.whl size=895254 sha256=b40b5cb73859638852d015475cb937ab832946ee01d74ec34966b764b2728baf
#12 30.62 Stored in directory: /root/.cache/pip/wheels/42/79/78/5ad3b042cb2d97c294535162cdbaf9b167e3b186eae55ab72d
#12 30.62 Successfully built moviepy ibm-watson gensim keybert ibm-cloud-sdk-core sentence-transformers sacremoses
#12 30.62 Failed to build tokenizers
#12 30.62 ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

executor failed running [/bin/sh -c python3 -m pip install -r requirements.txt]: exit code: 1`