Local Embedding Server Crashes Under Batched Requests
Local embedding server (Ollama, llama-server, or sentence-transformers) crashes or OOMs when processing large batches. Fix batch size, sequence length, and memory allocation.
#ollama
Articles tagged with #ollama
Multi-GPU Not Used — Model Runs Only on GPU 0
A local LLM uses only one GPU even though multiple are present. Fix tensor-parallel splits, NCCL setup, and Ollama multi-GPU configuration to distribute the workload.
Local Model Output Truncated Mid-Token
Local LLM stops generating mid-sentence or mid-word without an EOS token. Diagnose max_tokens limits, stop sequences, and streaming buffer issues.
Local Model Very Slow on First-Token After Cold Start
Local LLM takes 30-120 seconds to produce the first token after loading. Diagnose model loading, KV cache allocation, and GPU warmup to reduce cold-start latency.
Local Model Ignores the Tool-Calling Format
Local LLM outputs tool names in plain text instead of structured JSON, or ignores the tools list entirely. Fix tool-call templates, grammar constraints, and model selection.
Local RAG Index Rebuild Is Unbearably Slow
Rebuilding a local vector index from thousands of documents takes hours instead of minutes. Tune batch size, parallelism, and chunking to speed up RAG indexing.
MLX Conversion From HuggingFace Fails
mlx_lm.convert fails when converting a HuggingFace model to MLX format on Apple Silicon. Fix architecture support, dtype mismatches, and memory limits during conversion.
Ollama Doesn't Detect the GPU, Falls Back to CPU
Ollama ignores your NVIDIA or AMD GPU and runs inference on CPU only. Diagnose driver, CUDA, and ROCm mismatches and force GPU offloading.
Ollama Model Download Stalls at Some Percentage
Ollama pull freezes mid-download at a specific percentage. Diagnose network, disk, and registry issues and resume cleanly.
Ollama Pull Succeeds but the Model Isn't Listed
Ollama pull completes without error but the model doesn't appear in ollama list. Fix manifest path, OLLAMA_MODELS conflicts, and corrupted registry state.
Modelfile SYSTEM Prompt Is Ignored
The SYSTEM directive in an Ollama Modelfile has no effect on the model's behavior. Diagnose template structure, system role injection, and chat API vs. generate API differences.
Ollama Startup Fails With port already in use
Ollama refuses to start because port 11434 is already bound. Find the conflicting process, free the port, or run Ollama on an alternate port.