llama.cpp mmap Fails on a Network Drive
llama.cpp crashes or errors when loading a GGUF model from an NFS or SMB network share. Disable mmap or copy the model to local storage to fix it.
#llamacpp
Articles tagged with #llamacpp
llama.cpp Quality Drops After Switching to More Aggressive Quant
Responses degrade noticeably after moving from Q5_K_M to Q4_0 or lower in llama.cpp. Identify quality-sensitive layers and choose the right quantization tier.
Chat-Template Mismatch Produces Garbage Output
Local LLM returns scrambled, repetitive, or role-confused output because the chat template doesn't match the model. Identify and apply the correct template.
Misconfigured RoPE Scaling Garbles Long-Context Output
Local model output becomes incoherent or repetitive beyond a certain context length due to wrong RoPE scaling settings. Diagnose and fix dynamic NTK or linear scaling config.
Tokenizer Drift Causes Token-Count Mismatch
Token counts from your application's tokenizer disagree with the local inference server, causing context overflow or incorrect billing. Align tokenizer versions to fix the drift.