You generate a portrait and the eyes are slightly wrong: one is a few pixels higher than the other, the irises point in subtly different directions, or one pupil is noticeably bigger. At thumbnail size the face looks fine, but anyone who looks at the full-res image immediately notices it feels “off.” This happens because the eye region in a 1024x1024 portrait is only about 60x40 pixels of latent space, and the diffusion model simply doesn’t have enough resolution to lock in symmetric anatomy.
The fix is to give the face region its own dedicated denoising pass — through a face-detailer node, a face-restore model like CodeFormer or GFPGAN, or a manual inpaint at 2x resolution.
Common causes
Ordered by hit rate, highest first.
1. Face region is too small in latent space
In a standard SDXL 1024x1024 render of a half-body portrait, the actual eye area might be 50x30 pixels. The model is making anatomy decisions at that resolution — it has no chance of placing two pupils with sub-pixel precision.
How to spot it: zoom in on the face. If the face occupies less than 30% of the frame, you are in low-resolution-face territory and the eyes will drift.
2. Tall portrait or non-square aspect
When you render at 832x1216 (SDXL portrait) or 9:16, the model trained mostly on 1:1 data has to extrapolate. Faces in tall frames often skew, tilt slightly, or end up with mismatched eye height.
How to spot it: same prompt, same seed, switch to 1024x1024 — if the eyes line up there but not at 832x1216, aspect ratio is the cause.
3. No face-detailer / face-restore in the pipeline
Default ComfyUI / Automatic1111 / Forge workflows do not run a face pass. The model outputs whatever it outputs and you ship it. Without a dedicated face pass, eye misalignment is the norm, not the exception.
How to spot it: check your workflow JSON. If there is no FaceDetailer, ADetailer, CodeFormer, or GFPGAN node, you have no safety net.
4. Heavy style LoRA distorting anatomy
Strong style LoRAs (especially anime-photoreal hybrids or painterly LoRAs at weight 1.0+) often warp facial proportions. The style imprints on top of the base anatomy and shifts eye placement.
How to spot it: drop LoRA weight from 1.0 to 0.6 and regenerate. If the eyes correct themselves, the LoRA is overdriving.
5. Low step count or wrong sampler
Some samplers (Euler a, DPM++ 2M Karras) need 25-35 steps to fully resolve fine anatomy. If you are at 15 steps with a turbo sampler, the model hasn’t finished refining facial features.
How to spot it: bump steps from 20 to 35 with the same seed. If eyes improve, you were undersampling.
Shortest path to fix
Step 1: Add a face-detailer pass
In ComfyUI, drop in a FaceDetailer node from Impact Pack. In Automatic1111 / Forge, install the ADetailer extension. Recommended settings:
ADetailer model: face_yolov8n.pt
Inpaint denoise: 0.35-0.45
Mask blur: 4
Inpaint width/height: 768 or 1024 (NOT the base render size)
Sampler: same as base, 20 stepsThe detailer crops the face, upscales the crop, re-renders at 1024x1024 of just the face, then composites back. The eye region now gets full latent budget.
Step 2: Run CodeFormer or GFPGAN at moderate weight
If face-detailer isn’t available, fall back to a face-restore model:
CodeFormer weight: 0.5 (NOT 1.0 — 1.0 destroys skin texture)
GFPGAN weight: 0.4These are specifically trained on aligned face datasets, so they correct eye symmetry, pupil size, and iris direction. The trade-off is they slightly smooth the skin — keep weight below 0.6 to preserve texture.
Step 3: Manual inpaint when automated fails
For stubborn cases:
- Open the image in Automatic1111 / Forge inpaint
- Mask just the eye region with a mask blur of 8
- Set inpaint denoise to 0.45
- Set “Inpaint area” to “Only masked” with padding 32
- Re-prompt with
symmetric eyes, matching pupils, both eyes same size, looking at camera, sharp iris detail - Generate 4 variants and pick the cleanest
Step 4: Prevent at gen time with prompt hints
Add to the positive prompt:
symmetric face, both eyes aligned, matching pupils, perfect eye anatomy, sharp iris detail, eye contactAnd to the negative prompt:
asymmetric eyes, mismatched pupils, lazy eye, cross-eyed, different iris color, droopy eyelidThese hints alone won’t fix the resolution problem, but they help the model pick better defaults before the face-detailer cleans up.
Prevention
- Always render portraits with a face-detailer / ADetailer pass enabled by default
- For tall portraits, render at 1024x1024 first then outpaint vertically, instead of going straight to 832x1216
- Keep CodeFormer weight at 0.3-0.5 globally so you never destroy skin texture
- When using style LoRAs, cap weight at 0.7-0.8 to leave anatomy intact
- Save a workflow preset with face-detailer wired in and use it as your default starting point