llama 0.16.0 on 8xPi 4 4GB -> Critical error: Cannot open model file

[somera]

Hi,

I have 8x Raspberry Pi 4 Model B Rev 1.1 4GB with:

Raspbian GNU/Linux 11 (bullseye) aarch64
Linux xxx 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr  3 17:24:16 BST 2023 aarch64 GNU/Linux

I followed this https://github.com/b4rtaz/distributed-llama/blob/main/docs/HOW_TO_RUN_RASPBERRYPI.md instructions.

I could compile the sources (* (HEAD detached at v0.16.0)).

$ make dllama
g++ -std=c++11 -Werror -Wformat -Werror=format-security  -march=native -mtune=native -O3 -c src/nn/nn-quants.cpp -o nn-quants.o
g++ -std=c++11 -Werror -Wformat -Werror=format-security  -march=native -mtune=native -O3 -c src/nn/nn-core.cpp -o nn-core.o
g++ -std=c++11 -Werror -Wformat -Werror=format-security  -march=native -mtune=native -O3 -c src/nn/nn-executor.cpp -o nn-executor.o
g++ -std=c++11 -Werror -Wformat -Werror=format-security  -march=native -mtune=native -O3 -c src/nn/nn-network.cpp -o nn-network.o
g++ -std=c++11 -Werror -Wformat -Werror=format-security  -march=native -mtune=native -O3 -c src/nn/llamafile/sgemm.cpp -o llamafile-sgemm.o
g++ -std=c++11 -Werror -Wformat -Werror=format-security  -march=native -mtune=native -O3 -c src/nn/nn-cpu-ops.cpp -o nn-cpu-ops.o
g++ -std=c++11 -Werror -Wformat -Werror=format-security  -march=native -mtune=native -O3 -c src/nn/nn-cpu.cpp -o nn-cpu.o
g++ -std=c++11 -Werror -Wformat -Werror=format-security  -march=native -mtune=native -O3 -c src/tokenizer.cpp -o tokenizer.o
g++ -std=c++11 -Werror -Wformat -Werror=format-security  -march=native -mtune=native -O3 -c src/llm.cpp -o llm.o
g++ -std=c++11 -Werror -Wformat -Werror=format-security  -march=native -mtune=native -O3 -c src/app.cpp -o app.o
g++ -std=c++11 -Werror -Wformat -Werror=format-security  -march=native -mtune=native -O3 src/dllama.cpp nn-quants.o nn-core.o nn-executor.o nn-network.o llamafile-sgemm.o nn-cpu-ops.o nn-cpu.o tokenizer.o llm.o app.o -o dllama -lpthread

...

$ make dllama-api
g++ -std=c++11 -Werror -Wformat -Werror=format-security  -march=native -mtune=native -O3 src/dllama-api.cpp nn-quants.o nn-core.o nn-executor.o nn-network.o llamafile-sgemm.o nn-cpu-ops.o nn-cpu.o tokenizer.o llm.o app.o -o dllama-api -lpthread
In file included from /usr/include/c++/10/vector:72,
                 from src/dllama-api.cpp:9:
/usr/include/c++/10/bits/vector.tcc: In member function ‘void std::vector<_Tp, _Alloc>::_M_realloc_insert(std::vector<_Tp, _Alloc>::iterator, _Args&& ...) [with _Args = {const Model&}; _Tp = Model; _Alloc = std::allocator<Model>]’:
/usr/include/c++/10/bits/vector.tcc:426:7: note: parameter passing for argument of type ‘std::vector<Model>::iterator’ changed in GCC 7.1
  426 |       vector<_Tp, _Alloc>::
      |       ^~~~~~~~~~~~~~~~~~~
In file included from /usr/include/c++/10/map:60,
                 from src/json.hpp:164,
                 from src/dllama-api.cpp:24:
/usr/include/c++/10/bits/stl_tree.h: In function ‘std::_Rb_tree<_Key, _Val, _KeyOfValue, _Compare, _Alloc>::iterator std::_Rb_tree<_Key, _Val, _KeyOfValue, _Compare, _Alloc>::_M_emplace_hint_unique(std::_Rb_tree<_Key, _Val, _KeyOfValue, _Compare, _Alloc>::const_iterator, _Args&& ...) [with _Args = {const std::piecewise_construct_t&, std::tuple<const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>, std::tuple<>}; _Key = std::__cxx11::basic_string<char>; _Val = std::pair<const std::__cxx11::basic_string<char>, nlohmann::json_abi_v3_11_3::basic_json<> >; _KeyOfValue = std::_Select1st<std::pair<const std::__cxx11::basic_string<char>, nlohmann::json_abi_v3_11_3::basic_json<> > >; _Compare = std::less<std::__cxx11::basic_string<char> >; _Alloc = std::allocator<std::pair<const std::__cxx11::basic_string<char>, nlohmann::json_abi_v3_11_3::basic_json<> > >]’:
/usr/include/c++/10/bits/stl_tree.h:2458:7: note: parameter passing for argument of type ‘std::_Rb_tree<std::__cxx11::basic_string<char>, std::pair<const std::__cxx11::basic_string<char>, nlohmann::json_abi_v3_11_3::basic_json<> >, std::_Select1st<std::pair<const std::__cxx11::basic_string<char>, nlohmann::json_abi_v3_11_3::basic_json<> > >, std::less<std::__cxx11::basic_string<char> >, std::allocator<std::pair<const std::__cxx11::basic_string<char>, nlohmann::json_abi_v3_11_3::basic_json<> > > >::const_iterator’ changed in GCC 7.1
 2458 |       _Rb_tree<_Key, _Val, _KeyOfValue, _Compare, _Alloc>::
      |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/c++/10/map:61,
                 from src/json.hpp:164,
                 from src/dllama-api.cpp:24:
/usr/include/c++/10/bits/stl_map.h: In member function ‘bool nlohmann::json_abi_v3_11_3::detail::json_sax_dom_parser<BasicJsonType>::key(nlohmann::json_abi_v3_11_3::detail::json_sax_dom_parser<BasicJsonType>::string_t&) [with BasicJsonType = nlohmann::json_abi_v3_11_3::basic_json<>]’:
/usr/include/c++/10/bits/stl_map.h:501:37: note: parameter passing for argument of type ‘std::_Rb_tree<std::__cxx11::basic_string<char>, std::pair<const std::__cxx11::basic_string<char>, nlohmann::json_abi_v3_11_3::basic_json<> >, std::_Select1st<std::pair<const std::__cxx11::basic_string<char>, nlohmann::json_abi_v3_11_3::basic_json<> > >, std::less<std::__cxx11::basic_string<char> >, std::allocator<std::pair<const std::__cxx11::basic_string<char>, nlohmann::json_abi_v3_11_3::basic_json<> > > >::const_iterator’ changed in GCC 7.1
  501 |    __i = _M_t._M_emplace_hint_unique(__i, std::piecewise_construct,
      |          ~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  502 |          std::tuple<const key_type&>(__k),
      |          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  503 |          std::tuple<>());
      |          ~~~~~~~~~~~~~~~
/usr/include/c++/10/bits/stl_map.h: In member function ‘bool nlohmann::json_abi_v3_11_3::detail::json_sax_dom_callback_parser<BasicJsonType>::key(nlohmann::json_abi_v3_11_3::detail::json_sax_dom_callback_parser<BasicJsonType>::string_t&) [with BasicJsonType = nlohmann::json_abi_v3_11_3::basic_json<>]’:
/usr/include/c++/10/bits/stl_map.h:501:37: note: parameter passing for argument of type ‘std::_Rb_tree<std::__cxx11::basic_string<char>, std::pair<const std::__cxx11::basic_string<char>, nlohmann::json_abi_v3_11_3::basic_json<> >, std::_Select1st<std::pair<const std::__cxx11::basic_string<char>, nlohmann::json_abi_v3_11_3::basic_json<> > >, std::less<std::__cxx11::basic_string<char> >, std::allocator<std::pair<const std::__cxx11::basic_string<char>, nlohmann::json_abi_v3_11_3::basic_json<> > > >::const_iterator’ changed in GCC 7.1
  501 |    __i = _M_t._M_emplace_hint_unique(__i, std::piecewise_construct,
      |          ~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  502 |          std::tuple<const key_type&>(__k),
      |          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  503 |          std::tuple<>());
      |          ~~~~~~~~~~~~~~~
In file included from /usr/include/c++/10/vector:67,
                 from src/dllama-api.cpp:9:
/usr/include/c++/10/bits/stl_vector.h: In function ‘void handleModelsRequest(HttpRequest&, const char*)’:
/usr/include/c++/10/bits/stl_vector.h:1198:21: note: parameter passing for argument of type ‘__gnu_cxx::__normal_iterator<Model*, std::vector<Model> >’ changed in GCC 7.1
 1198 |    _M_realloc_insert(end(), __x);
      |    ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~

And I downloaded two models:

$ ls -alFR models/
models/:
insgesamt 16K
drwxr-xr-x  4 pi pi 4,0K 15. Sep 21:48 ./
drwxr-xr-x 11 pi pi 4,0K 15. Sep 22:06 ../
drwxr-xr-x  2 pi pi 4,0K 15. Sep 21:53 llama3_2_3b_instruct_q40/
drwxr-xr-x  2 pi pi 4,0K 15. Sep 21:44 qwen3_1.7b_q40/

models/llama3_2_3b_instruct_q40:
insgesamt 3,2G
drwxr-xr-x 2 pi pi 4,0K 15. Sep 21:53 ./
drwxr-xr-x 4 pi pi 4,0K 15. Sep 21:48 ../
-rw-r--r-- 1 pi pi 3,2G 15. Sep 21:53 dllama_model_llama3_2_3b_instruct_q40.m
-rw-r--r-- 1 pi pi 1,8M 15. Sep 21:53 dllama_tokenizer_llama3_2_3b_instruct_q40.t

models/qwen3_1.7b_q40:
insgesamt 2,1G
drwxr-xr-x 2 pi pi 4,0K 15. Sep 21:44 ./
drwxr-xr-x 4 pi pi 4,0K 15. Sep 21:48 ../
-rw-r--r-- 1 pi pi 2,1G 15. Sep 21:44 dllama_model_qwen3_1.7b_q40.m
-rw-r--r-- 1 pi pi 2,1M 15. Sep 21:44 dllama_tokenizer_qwen3_1.7b_q40.t

Than I started 7 workers:

$ sudo nice -n -20 ./dllama worker --port 9999 --nthreads 4
Listening on 0.0.0.0:9999...

But than on root mode ...

$ sudo nice -n -20 ./dllama inference \
  --prompt "Hello world" \
  --steps 32 \
  --model ./models/llama3_2_3b_instruct_q40/dllama_model_llama3_2_3b_instruct_q40.m \
  --tokenizer ./models/llama3_2_3b_instruct_q40/dllama_tokenizer_llama3_2_3b_instruct_q40.t \
  --buffer-float-type q80 \
  --nthreads 4 \
  --max-seq-len 4096 \
  --workers 192.168.178.139:9999 192.168.178.141:9999 192.168.178.142:9999 192.168.178.143:9999 192.168.178.144:9999 192.168.178.145:9999 192.168.178.146:9999
🚨 Critical error: Cannot open model file

or

$ sudo nice -n -20 ./dllama inference \
  --prompt "Hello world" \
  --steps 32 \
  --model models/qwen3_1.7b_q40/dllama_model_qwen3_1.7b_q40.m \
  --tokenizer models/qwen3_1.7b_q40/dllama_tokenizer_qwen3_1.7b_q40.t \
  --buffer-float-type q80 \
  --nthreads 4 \
  --max-seq-len 4096 \
  --workers 192.168.178.139:9999 192.168.178.141:9999 192.168.178.142:9999 192.168.178.143:9999 192.168.178.144:9999 192.168.178.145:9999 192.168.178.146:9999
🚨 Critical error: Cannot open model file

And the model file is here:

$ ls -al models/qwen3_1.7b_q40/dllama_model_qwen3_1.7b_q40.m models/qwen3_1.7b_q40/dllama_tokenizer_qwen3_1.7b_q40.t
-rw-r--r-- 1 pi pi 2,1G 15. Sep 21:44 models/qwen3_1.7b_q40/dllama_model_qwen3_1.7b_q40.m
-rw-r--r-- 1 pi pi 2,1M 15. Sep 21:44 models/qwen3_1.7b_q40/dllama_tokenizer_qwen3_1.7b_q40.t

[b4rtaz]

Hey @somera,

I think you could try to use absolute paths:

realpath models/qwen3_1.7b_q40/dllama_tokenizer_qwen3_1.7b_q40.t => <absolute_path>

sudo nice -n -20 ./dllama inference ... --model <absolute_path> --tokenizer <absolute_path>

Make sure the model/tokenizer files are downloaded to the root node, workers don’t need them.

[somera]

I can use it in the Open WebUI.

But only for one call. Cause my Raspis run under 100% after this ...

All of them.

[somera]

I waited ~1 minute and the CPU usage goes doen. Than I could send the 2nd prompt.

[b4rtaz]

I think the reason might be that Open WebUI, after receiving a response from dllama-api, makes another request to dllama-api to generate a list of possible follow-up questions. So it’s essentially making an additional request to the LLM. I believe there may be an option to disable this.

[somera]

Exactly. Yes, there is an option. But I think this is a global option. I need to check this.

But now I have a working solution. Later I will create some bash scripts for better usability and try other clients to comunicate with. The speed is slow, but I can play with some SLMs.

[somera]

No i dobrej nocy! Spadam na dzisiaj. ;)