By the time Search Console surfaces a broken link, Googlebot has been wasting crawl requests on it for weeks. The cost in 2026 is rarely a direct ranking penalty (Google treats hard 404s as a normal part of the web) but the second-order damage is real: dead internal links leak link equity, soft 404s burn crawl budget worse than clean 404s because Googlebot keeps re-fetching them, and dead external citations quietly erode reader trust. The fix is a 30-minute monthly loop: a deterministic checker finds every dead URL, then AI clusters the findings by root cause and ranks the fixes. This walks through the full loop with exact commands.
TL;DR
- Run one checker against your sitemap monthly.
lychee(Rust) is fastest and most accurate on external URLs;linkinator(Node) is the simplest npm setup. - Feed the raw findings to a chat model and ask it to cluster by root cause (internal 404, redirect chain, typo, dead external, 410 gone) and assign HIGH/MED/LOW.
- Fix internal 404s first (link equity + crawl), shorten redirect chains, replace or archive dead external links. Never silently delete an external citation.
- Re-run the checker. The diff between runs is your fix evidence. Schedule it via GitHub Actions so nobody has to remember.
What this covers
A monthly maintenance workflow for content sites with 100+ articles. Output: a triaged list of broken internal and external links, grouped by root cause, with a fix proposal per cluster. The checker is the source of truth; AI does the grouping and prioritization. It does not invent URLs and it does not crawl — those are jobs for the deterministic tool.
Who this is for
Content site owners with 100+ articles. SEO leads at small companies. Indie developers running affiliate or blog sites, where a dead external link is both an SEO drag and a credibility problem.
When to reach for it
Monthly maintenance is the baseline. Run an extra sweep before any major SEO push (you do not want fixable 404s eating crawl requests right when Google notices new activity), after a domain migration, after a slug rename, after a content cleanup, and after importing content from another platform.
Pick a checker
As of June 2026 these are the three mature options. Pick one and stick with it so your run-to-run diffs stay comparable.
| Tool | Engine | Default concurrency | Caching | Best for |
|---|---|---|---|---|
lychee | Rust, async | --max-concurrency 128 (10 per host) | .lycheecache, --max-cache-age 1d | Speed + external-URL accuracy |
linkinator | Node | 100 connections | none built in | Simple npm-only stacks, CSS URLs |
broken-link-checker | Node (older) | configurable | none | Legacy setups already using it |
lychee defaults worth knowing: it retries failed requests 3 times (--max-retries), waits 1s between retries, accepts status codes 100..=103,200..=299 by default (override with --accept), and throttles to one request per host every 50ms. Those defaults are why it rarely produces false 404s from transient blips. linkinator is shallow by default — it only scans the top-level page unless you pass --recurse.
Before you start
- Install your chosen checker.
- Have your sitemap URL ready (
/sitemap.xml). The checker uses it as the crawl entry point. - If you check links inside
github.com, set aGITHUB_TOKENenv var or pass--github-token. Unauthenticated GitHub requests get rate-limited fast; with a token the limit is 1,000 requests per hour per repository. - Decide your action threshold up front: external 404s with under 10 backlinks may not be worth fixing; internal 404s are always worth fixing.
Step by step
- Run the checker against production. Caching makes re-runs cheap, so turn it on from day one:
# lychee (fastest): cache results, follow your sitemap
lychee --cache --max-concurrency 64 --include-fragments \
https://yoursite.com/sitemap.xml > broken.txt
# linkinator: recurse the whole site, emit JSON
npx linkinator https://yoursite.com --recurse \
--format JSON > broken.jsonCross-check what the checker found against a Codex sitemap review so you do not miss URLs the checker never crawled (orphan pages with no inbound links never enter a crawl that starts from the sitemap).
-
Feed the output to a chat model as context. For 100+ findings, paste a CSV; for fewer, paste the raw output. Any current model handles this well (GPT-5.5, Claude Sonnet 4.6, or Gemini 3.1 Pro) — it is a clustering task, not a reasoning-heavy one.
-
Prompt the model:
Here are N broken-link findings from a link checker.
Cluster by root cause: internal 404, internal redirect
chain (>2 hops), typo in URL (extra slash, wrong case),
dead external (host unreachable), external 410 (gone).
For each cluster, propose fix priority (HIGH/MED/LOW)
based on: how many source articles link to it, whether it
is internal or external, whether it returns 410 vs 404.
Output a table, one row per finding, with a fix action.- Internal 404s (always HIGH). Grep your source content for the broken URL and fix it to the correct slug. Most are typos or post-rename leftovers. These are the highest-leverage fixes because they restore internal link equity and stop crawl waste.
- Internal redirect chains. Shorten to a single hop in the source content. Chains over 2 hops add latency and dilute link equity at each hop.
- Dead external links. Replace with an equivalent source if one exists, or archive it: swap the URL for a Wayback Machine snapshot, or mark it
[archived]with a note. Never silently delete a citation — other pages may link to yours citing that source. - Re-run the checker. Use the same flags so the diff is clean. The set of findings that disappeared is your proof of work.
Worked example: one rename, thirty broken links
A typical first run on a 1,200-page bilingual content site looks like this:
| Cluster | Count | Priority | Root cause |
|---|---|---|---|
| Internal 404 | 10 | HIGH | One slug rename left 8 inbound links pointing at the old path |
| Redirect chain >2 hops | 15 | MED | Trailing-slash redirect feeding into an old-domain redirect |
| Dead external | 18 | LOW–MED | Vendor blog posts that 404’d after a site redesign |
| URL typo | 7 | HIGH | Hand-typed href with a doubled slash or wrong case |
Fix internal first, then shorten the redirect chains, then triage the externals. Re-run, and the only survivors are usually 2–3 external hosts returning transient 5xx errors that flap. Schedule a re-check next month and move on. Total time: about 90 minutes the first time, 30 minutes once you have the cluster prompt saved. Treat the sweep as one row in a larger stack-tailored technical SEO checklist, alongside schema, hreflang, and render-mode checks.
Verify before you trust the AI
The AI is doing pattern-matching, not ground truth. Spot-check it:
- Are the clusters meaningful, or did the model lump unrelated issues together? Manually check 5 findings per cluster.
- Did the checker miss URLs that exist in your content? Compare against
grep -roE 'href="[^"]+"' src/to catch links the crawler never reached. - Were any “dead” externals hit at a bad moment? Re-test after 24 hours — transient 5xx errors flap, and a single retry window is not enough to declare a host dead. (
lychee --cachehelps here: cached good results are not re-queried.) - Does the proposed “fix” need human judgment? “Replace with similar source” is a task, not a fix — the model cannot find the replacement for you.
Automate the cadence
Drift returns within about 60 days on an active site, so make the run automatic.
- Schedule the checker with GitHub Actions on a monthly cron. The official
lycheeverse/lychee-actionruns lychee, writes a report, and can open or update an issue with the findings — so the report comes to you instead of waiting for someone to remember to run a command. - Save the AI clustering prompt as a saved prompt or Custom GPT. The prompt is stable; only the input changes month to month.
- Track findings over time. A recurring category (always “rename leftovers”) is a signal that your slug-rename process needs a fix step built in, not just a faster fixing cadence.
Common mistakes
- Waiting for Search Console to complain. By then you have lost weeks of crawl efficiency.
- Silently deleting external links. Readers and citing pages expect them. Archive with Wayback Machine instead.
- No monthly cadence. Drift returns within 60 days on an active site.
- Removing an external 404 without finding a replacement. Half the value of a citation is the link itself.
- Trusting the AI’s “fix” literally without confirming the target page exists.
- Ignoring redirect chains. They do not show as broken, but they cost crawl budget and time-to-first-byte and dilute link equity at each hop.
- Running without
--cache. Every re-run re-hammers every external host, which is slow and gets you rate-limited.
FAQ
- Which checker is most accurate? As of June 2026,
lycheeis the most accurate on external URLs and the fastest, thanks to its retry and per-host throttling defaults.linkinatoris the simplest if you are already in a Node project. Both work; consistency between runs matters more than the choice. - What about JavaScript-rendered links? Most checkers see only static HTML. For SPA-heavy sites, use a checker with a headless-browser mode or a Playwright-based tool, otherwise links injected by client-side JS are invisible to the crawl.
- How fast should I fix? Internal 404s within 7 days (link equity and crawl). External within 30 days. Redirect chains within 60 days.
- Will AI replace the link checker? No. The checker is deterministic and is the source of truth; AI sits one layer above it (clustering, prioritization, fix proposals). A model that “checks links” by reasoning will hallucinate status codes. Always use a real checker for the URLs.
- How do I check links inside MDX or markdown source, not the deployed site? Run a separate pre-build script that resolves internal links against your content collection — the live checker only sees the deployed HTML. lychee can also scan local Markdown and HTML files directly if you point it at a glob instead of a sitemap.
- Do 404s actually hurt my rankings? Not directly in 2026 — Google treats them as normal. The cost is indirect: wasted crawl budget, lost internal link equity, and soft 404s (a “not found” page that returns 200 OK) that Googlebot keeps re-crawling. Fix them for crawl efficiency and link equity, not fear of a penalty.
Related
- SEO audit prompts
- Run site content audit
- AI Check Your Hreflang Setup (Bilingual / Multi-Locale)
- AI Astro content audit tutorial
- Codex sitemap review tutorial
Tags: #Tutorial #SEO #AI coding #Broken links