You set up a local Llama 3.1 70B server configured for 128k context, send a 60,000-token document for summarization, and the first 10,000 tokens produce coherent output. Then the response starts degenerating — repeated phrases, random topic switches, wrong pronoun references, and eventually nonsensical token streams. The model is “functioning” in the sense that it generates tokens, but it has lost its positional grounding for anything past the base training context length. This is the signature of misconfigured or entirely absent RoPE (Rotary Position Embedding) scaling, which is required for any context length beyond what the model was originally trained on.
Common causes
Ordered by hit rate, highest first.
1. No RoPE scaling configured at all for an extended-context request
Models like Llama 2 were trained on 4096-token context. Asking a vanilla Llama 2 model to process 32k tokens without any RoPE extension will produce garbage past position 4096 because the positional encodings are extrapolating far outside their training distribution.
How to spot it: Check the model’s config.json for rope_scaling. If it’s absent or null, and you’re using a context longer than the model’s max_position_embeddings, garbling is expected.
2. Wrong RoPE scaling type — linear vs. dynamic NTK
Llama 2 fine-tunes use either “linear” scaling (simple frequency scaling, works up to 3-4x the training context) or “dynamic NTK” (frequency adaptation at inference time, works up to 10-16x). Applying linear scaling for a 32x extension will garble output past the linear range. Conversely, applying dynamic NTK with the wrong alpha factor produces misaligned positional embeddings.
How to spot it: Check whether the model card specifies “NTK-aware” or “linear” scaling. In config.json, look for "type": "linear" vs. "type": "dynamic" in the rope_scaling field.
3. rope_theta set to the wrong value for an extended-context model
Models extended to 128k context (like Llama 3.1) use a higher rope_theta value (500000 for Llama 3.1 vs. 10000 for Llama 2) to allow stable embeddings at high sequence positions. If a GGUF file was converted without preserving the rope_theta from the model’s config.json, or if --rope-freq-base is manually set to the wrong value, long-context coherence breaks down.
How to spot it: Run python3 -c "import gguf; r=gguf.GGUFReader('model.gguf'); print(r.fields.get('llama.rope.freq_base'))". Compare against the model’s HuggingFace config.json rope_theta field.
4. —rope-scale flag overriding the correct embedded value
Some llama.cpp invocations specify --rope-scale (a multiplier for the base frequency). If this is set incorrectly (e.g., --rope-scale 8 when the model was trained with scale 1 and a higher theta), it amplifies the frequency mismatch and causes positional confusion at moderate context lengths (even 8k-16k).
How to spot it: Check your llama-server or llama-cli startup command for --rope-scale. If present, remove it and rely on the embedded GGUF rope settings instead.
5. vLLM not applying rope_scaling from the HuggingFace config
vLLM reads rope_scaling from the model’s config.json automatically. But if you specify --override-neuron-config or a custom --hf-config-path that doesn’t include rope_scaling, vLLM will use default (unscaled) RoPE. The symptom appears only on requests longer than the base training context.
How to spot it: Log the vLLM startup output for rope_scaling information, or run python -c "from transformers import AutoConfig; c=AutoConfig.from_pretrained('model'); print(c.rope_scaling)".
6. YaRN scaling not implemented in the runtime version
YaRN (Yet Another RoPE ExtensioN) is used by Mistral 7B 128k and some other models. Not all llama.cpp versions support YaRN; older builds treat it as linear scaling, which garbles output at extended lengths. YaRN support was added in llama.cpp in mid-2024.
How to spot it: Check whether your model uses "type": "yarn" in rope_scaling, then verify your llama.cpp build date. Run ./llama-server --version and compare against the commit date when YaRN was added.
Shortest path to fix
Step 1: Check the model’s expected RoPE configuration
# From HuggingFace (requires transformers package)
from transformers import AutoConfig
config = AutoConfig.from_pretrained(
"meta-llama/Llama-3.1-8B-Instruct",
trust_remote_code=True
)
print(f"max_position_embeddings: {config.max_position_embeddings}")
print(f"rope_theta: {config.rope_theta}")
print(f"rope_scaling: {config.rope_scaling}")For Llama 3.1: rope_theta=500000, rope_scaling={"factor": 8.0, "high_freq_factor": 4.0, "low_freq_factor": 1.0, "original_max_position_embeddings": 8192, "rope_type": "llama3"}.
Step 2: Verify the GGUF has correct rope_theta embedded
python3 << 'EOF'
import gguf
reader = gguf.GGUFReader("models/llama-3.1-8b-instruct-Q4_K_M.gguf")
# Check rope frequency base (theta)
rope_freq = reader.fields.get("llama.rope.freq_base")
if rope_freq:
print(f"rope_theta in GGUF: {rope_freq.parts[-1][0]}")
else:
print("rope_theta not found in GGUF — will use default (10000)")
# Check context length
ctx_len = reader.fields.get("llama.context_length")
if ctx_len:
print(f"context_length in GGUF: {ctx_len.parts[-1][0]}")
EOFIf rope_theta is 10000 but should be 500000, the GGUF needs to be reconverted with the correct settings.
Step 3: Re-convert the GGUF with correct rope settings
# Reconvert from HuggingFace weights with correct rope_theta preserved
python convert_hf_to_gguf.py \
/path/to/Meta-Llama-3.1-8B-Instruct \
--outtype f16 \
--outfile llama-3.1-8b-instruct-f16.gguf \
--verbose
# Verify the theta was preserved
python3 -c "
import gguf
r = gguf.GGUFReader('llama-3.1-8b-instruct-f16.gguf')
print(r.fields.get('llama.rope.freq_base'))
"
# Should show 500000 for Llama 3.1Step 4: Override rope_theta at runtime if GGUF embedding is wrong
# llama-server: override rope theta at runtime
./llama-server \
-m models/llama-3.1-8b-instruct-Q4_K_M.gguf \
--rope-freq-base 500000 \
--rope-scaling llama3 \
--ctx-size 131072 \
--n-gpu-layers 99
# For YaRN scaling (Mistral 128k models)
./llama-server \
-m models/Mistral-7B-Instruct-v0.3-128k-Q4_K_M.gguf \
--rope-scaling yarn \
--rope-scale 4 \
--ctx-size 131072Step 5: For vLLM, verify rope_scaling is read from config
# Start vLLM with explicit rope_scaling display
python3 << 'EOF'
from vllm import LLM
llm = LLM(
model="meta-llama/Meta-Llama-3.1-8B-Instruct",
max_model_len=32768,
gpu_memory_utilization=0.90,
)
# vLLM will log the rope_scaling config during initialization
EOFCheck the startup logs for: INFO: Using rope_scaling with type=llama3 factor=8.0.
Step 6: Test long-context coherence with a reference task
# Generate a long, structured prompt and test coherence at different positions
python3 << 'EOF'
import requests
# Create a test with known content at different positions
numbers = " ".join([f"Position {i}: the keyword is MARKER{i}." for i in range(1, 201)])
prompt = f"Here is a numbered sequence:\n{numbers}\n\nWhat is the keyword at Position 150?"
resp = requests.post("http://localhost:8080/v1/chat/completions", json={
"messages": [{"role": "user", "content": prompt}],
"max_tokens": 50,
"temperature": 0,
})
print(resp.json()["choices"][0]["message"]["content"])
# Should say: "MARKER150"
EOFPrevention
- Before deploying any model for long-context use, verify
rope_thetain the GGUF matches the HuggingFace config. - Always re-convert GGUF files from HuggingFace source weights when updating to a newer model version — don’t use third-party converted GGUFs without checking their RoPE configuration.
- Document the
--rope-freq-baseand--ctx-sizevalues in your launch script comments alongside the source model name. - Test long-context coherence with the positional marker test above before putting a model into production.
- For models using
rope_type=llama3orrope_type=yarn, verify your llama.cpp build date is recent enough to support that type. - Avoid using
--rope-scaleunless you’ve specifically verified it against the model’s training configuration. - When using vLLM, always load from the HuggingFace Hub ID (not a local path missing
config.json) so rope_scaling is read automatically.
FAQ
Q: What context length is safe without any RoPE scaling configuration?
A: Use at most the model’s max_position_embeddings value from config.json. For Llama 2, that’s 4096. For Llama 3 (not 3.1), it’s 8192. Going even 10% beyond this without RoPE scaling will start producing degraded output on long-range dependencies.
Q: Does quantization affect RoPE scaling? A: Indirectly. The RoPE frequencies are applied in fp32 during inference regardless of weight quantization. However, extremely aggressive quantization (Q2_K, Q3_K_S) can amplify existing position encoding errors because the attention weights have less precision to compensate for mild positional mismatches.
Q: My model generates perfectly for 8k tokens then degrades at 9k — what’s the threshold?
A: The threshold is likely the model’s native max_position_embeddings (8192 for Llama 3 base). The degradation at exactly this boundary is the clearest indicator of missing RoPE scaling. Set --rope-freq-base 500000 (Llama 3.1 style) or enable dynamic NTK scaling and the threshold will move to the full extended length.
Q: Is YaRN better than dynamic NTK for long contexts? A: YaRN is generally considered superior for very long contexts (32k-128k) because it uses different scaling factors for low- and high-frequency components, preserving short-range attention patterns while extending long-range position coverage. Dynamic NTK is simpler and works well up to 16-32k contexts but may degrade at extreme lengths.