llama.ts: failed to load model. Error: create_tensor: tensor 'blk.0.ffn_gate.weight' not found
#21
by
AlexanderWillamowski
- opened
Hi community,
constantly I am facing the following issue. I downloaded version dolphin-2.5-mixtral-8x7b.Q4_K_M.gguf on my Linux Debian 12 machine, 32 RAM both with GPT4all (v2.5.4) and LM Studio (AppImage LM+Studio-0.2.8-beta-v1) and on both applications, the model throws the following error wenn I want to load the model:
Error loading model: create_tensor: tensor 'blk.0.ffn_gate.weight' not found
llama_load_model_from_file: failed to load model
llama_init_from_gpt_params: error: failed to load model '/media/berion/daten/gpt4all/models/TheBloke/dolphin-2.5-mixtral-8x7b-GGUF/dolphin-2.5-mixtral-8x7b.Q4_K_M.gguf'
load: error: failed to load model '/media/berion/daten/gpt4all/models/TheBloke/dolphin-2.5-mixtral-8x7b-GGUF/dolphin-2.5-mixtral-8x7b.Q4_K_M.gguf'
Debug #4/media/berion/daten/gpt4all/models/TheBloke/dolphin-2.5-mixtral-8x7b-GGUF/dolphin-2.5-mixtral-8x7b.Q4_K_M.gguf
llama.ts: failed to load model. Error: create_tensor: tensor 'blk.0.ffn_gate.weight' not found
[modelprocess.ts] Error in loadModel: Error: create_tensor: tensor 'blk.0.ffn_gate.weight' not found
on message 2 [remotellm.ts] LLM process error: Error: create_tensor: tensor 'blk.0.ffn_gate.weight' not found
[modelprocesscontainer.ts] error handler: Error: Error: create_tensor: tensor 'blk.0.ffn_gate.weight' not found
[appserver] Error from LLM: Error: Error: create_tensor: tensor 'blk.0.ffn_gate.weight' not found
[appserver] No inferencingReplyPort to send error to
Error: create_tensor: tensor 'blk.0.ffn_gate.weight' not found
at t.Llama.<anonymous> (/tmp/.mount_LM+Stu4ZNZrJ/resources/app/.webpack/main/utility.js:2:261927)
at Generator.next (<anonymous>)
at r (/tmp/.mount_LM+Stu4ZNZrJ/resources/app/.webpack/main/utility.js:2:260462)
[modelprocesscontainer.ts] #forkUtilityProcessAndLoadModel Error loading model: Error: create_tensor: tensor 'blk.0.ffn_gate.weight' not found
[modelprocesscontainer.ts] error handler: Error: Model failed (exit code: 42). Please try loading it again.
[appserver] Error from LLM: Error: Model failed (exit code: 42). Please try loading it again.
[appserver] No inferencingReplyPort to send error to
[remotellm.ts] INSIDE LOAD PROMISE EXIT LLM process exited with code 42.
I tried it also with a newer version like dolphin-2.6-mixtral-8x7b.Q3_K_M.gguf ending up the same and
the minimal model dolphin-2.5-mixtral-8x7b.Q2_K.gguf
I've also checked the SHA256 checksum. Everything should be fine with the downloaded GGUF file.
$ sha256sum dolphin-2.5-mixtral-8x7b.Q4_K_M.gguf
ee4474c3f6f9c20d7ba738ab63c7c19d3e1471749a72e8624e5b9db0c827b525 dolphin-2.5-mixtral-8x7b.Q4_K_M.gguf
Any suggestions how to fix this issue?
My memory configurations:
$ cat /proc/meminfo
MemTotal: 32643500 kB
MemFree: 10057616 kB
MemAvailable: 26913816 kB
Buffers: 6538016 kB
Cached: 11745012 kB
SwapCached: 0 kB
Active: 1800968 kB
Inactive: 18772184 kB
Active(anon): 8208 kB
Inactive(anon): 3581172 kB
Active(file): 1792760 kB
Inactive(file): 15191012 kB
Unevictable: 1229592 kB
Mlocked: 5964 kB
SwapTotal: 7812092 kB
SwapFree: 7812092 kB
Zswap: 0 kB
Zswapped: 0 kB
Dirty: 5600 kB
Writeback: 0 kB
AnonPages: 3469052 kB
Mapped: 768908 kB
Shmem: 1294052 kB
KReclaimable: 341452 kB
Slab: 511628 kB
SReclaimable: 341452 kB
SUnreclaim: 170176 kB
KernelStack: 25984 kB
PageTables: 50308 kB
SecPageTables: 0 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 24133840 kB
Committed_AS: 12432884 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 72596 kB
VmallocChunk: 0 kB
Percpu: 7520 kB
HardwareCorrupted: 0 kB
AnonHugePages: 811008 kB
ShmemHugePages: 1105920 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 190068 kB
DirectMap2M: 3780608 kB
DirectMap1G: 30408704 kB