Alpaca 7B faults on both macOS arm64 and Linux ppc64le

[classilla]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• [ X] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• [ X] I carefully followed the README.md.
• [ X] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• [ X] I reviewed the Discussions, and have a new bug or useful enhancement to share.

From tip as of this posting, trying to use the available Alpaca 7B model weights causes a fault (out of memory?) on both my macOS and Linux systems. However, LLaMA 7B works fine. M1 MacBook Air with Ventura and 16GB of RAM, and Raptor Talos II 64-thread POWER9 with 64GB RAM. Using the recommended command line,

% ./main -m ./models/alpaca/7B/ggml-alpaca-7b-q4.bin --color -f ./prompts/alpaca.txt -ins
main: seed = 1679458525
llama_model_load: loading model from './models/alpaca/7B/ggml-alpaca-7b-q4.bin' - please wait ...
llama_model_load: n_vocab = 32000
llama_model_load: n_ctx   = 512
llama_model_load: n_embd  = 4096
llama_model_load: n_mult  = 256
llama_model_load: n_head  = 32
llama_model_load: n_layer = 32
llama_model_load: n_rot   = 128
llama_model_load: f16     = 2
llama_model_load: n_ff    = 11008
llama_model_load: n_parts = 1
llama_model_load: ggml ctx size = 4529.34 MB
llama_model_load: memory_size =   512.00 MB, n_mem = 16384
llama_model_load: loading model part 1/1 from './models/alpaca/7B/ggml-alpaca-7b-q4.bin'
llama_model_load: ..........................libc++abi: terminating with uncaught exception of type std::bad_alloc: std::bad_alloc
Abort
macbookair2:/Users/spectre/src/llama.cpp/% uname -a
Darwin macbookair2.local 22.3.0 Darwin Kernel Version 22.3.0: Mon Jan 30 20:39:35 PST 2023; root:xnu-8792.81.3~2/RELEASE_ARM64_T8103 arm64

[classilla]

The error message is slightly different on the POWER9 Linux system but faults at the same point. Backtrace:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
main: seed = 1679457708
llama_model_load: loading model from './models/ggml-alpaca-7b-q4.bin' - please wait ...
llama_model_load: n_vocab = 32000
llama_model_load: n_ctx   = 512
llama_model_load: n_embd  = 4096
llama_model_load: n_mult  = 256
llama_model_load: n_head  = 32
llama_model_load: n_layer = 32
llama_model_load: n_rot   = 128
llama_model_load: f16     = 2
llama_model_load: n_ff    = 11008
llama_model_load: n_parts = 1
llama_model_load: ggml ctx size = 4529.34 MB
llama_model_load: memory_size =   512.00 MB, n_mem = 16384
llama_model_load: loading model part 1/1 from './models/ggml-alpaca-7b-q4.bin'
llama_model_load: ..........................terminate called after throwing an instance of 'std::length_error'
  what():  basic_string::_M_create

Program received signal SIGABRT, Aborted.
0x00007ffff784a18c in __pthread_kill_implementation () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.36-9.fc37.ppc64le libgcc-12.2.1-4.fc37.ppc64le libstdc++-12.2.1-4.fc37.ppc64le
(gdb) bt
#0  0x00007ffff784a18c in __pthread_kill_implementation ()
   from /lib64/libc.so.6
#1  0x00007ffff77e7bcc in raise () from /lib64/libc.so.6
#2  0x00007ffff77c095c in abort () from /lib64/libc.so.6
#3  0x00007ffff7cc9f68 in __gnu_cxx::__verbose_terminate_handler() ()
   from /lib64/libstdc++.so.6
#4  0x00007ffff7cc50d4 in __cxxabiv1::__terminate(void (*)()) ()
   from /lib64/libstdc++.so.6
#5  0x00007ffff7cc51a0 in std::terminate() () from /lib64/libstdc++.so.6
#6  0x00007ffff7cc5664 in __cxa_throw () from /lib64/libstdc++.so.6
#7  0x00007ffff7c797ec in std::__throw_length_error(char const*) ()
   from /lib64/libstdc++.so.6
#8  0x00007ffff7dbab8c in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_create(unsigned long&, unsigned long) ()
   from /lib64/libstdc++.so.6
#9  0x00007ffff7dbacc4 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct(unsigned long, char) ()
   from /lib64/libstdc++.so.6
#10 0x0000000010019e24 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string (__a=..., __c=0 '\000', 
    __n=<optimized out>, this=0x7fffffffcbc8)
    at /usr/include/c++/12/bits/basic_string.h:659
#11 llama_model_load (fname="./models/ggml-alpaca-7b-q4.bin", model=..., 
    vocab=..., n_ctx=<optimized out>, n_parts=1, memory_type=<optimized out>)
    at main.cpp:428
#12 0x0000000010011078 in main (argc=<optimized out>, argv=0x7fffffffea68)
    at main.cpp:976
(gdb)

[anzz1]

Are your model files correct?

sha256sum of ggml-alpaca-7b-q4.bin should be 8d5562ec1d8a7cfdcf8985a9ddf353339d942c7cf52855a92c9ff59f03b541bc

e: this was probably wrong

(some of?) the hashes currently found in the SHA256SUMS file are wrong #374

[classilla]

The hash indeed does not match. I have now downloaded an older set of weights for Alpaca 7B that matches the hash in the alpaca.cpp project (9c1bb4808f40aa0059d5343d3aac05fb75d368c240b664878d53d16bf27ade2b) and those work with alpaca.cpp, so I converted it according to the instructions in #324. That does work now, but the hash doesn't match. I don't know if that's expected.

% openssl dgst -sha256 models/alpaca/7B/ggml-alpaca-7b-q4.bin
SHA256(models/alpaca/7B/ggml-alpaca-7b-q4.bin)= 6187814e53dfe9eab3a020f57035d093638b4033b6da40c3d400e60e51d1648c

This can probably be closed if this is really just a special case of #374.