Crawl errors are one of the most damaging — and most overlooked — SEO problems. When search engine bots can't access a page, it never gets indexed, and a page that isn't indexed can't rank, no matter how good the content is. This guide shows you how to find crawl errors, fix the most common types, and stop them from coming back.
What are crawl errors?
A crawl error happens when a search engine tries to reach a URL on your site and fails. Google groups these into two broad buckets: site-level errors that affect your whole site (DNS problems, server outages, a misconfigured robots.txt) and URL-level errors that affect individual pages (404s, redirects, blocked resources). Site-level errors are the most urgent — they can make your entire site uncrawlable.
Where to find crawl errors
Your primary tool is free: Google Search Console → Indexing → Pages. It lists every reason URLs aren't indexed, with example URLs for each. Also check the Crawl stats report (Settings → Crawl stats) for spikes in server errors. For a deeper scan, a crawler like Screaming Frog surfaces broken links and redirect chains across the whole site.
The most common crawl errors and how to fix them
404 Not Found
The page doesn't exist. Fine if intentional — but fix it when the URL still has value or internal links point to it. Fix: 301-redirect the old URL to the closest relevant page, or restore the content. For permanently removed pages, return a clean 404 or 410 so Google drops them.
Soft 404
The page returns a "200 OK" status but shows little or no real content (or an empty "not found" message). Google flags these because the status lies about what's on the page. Fix: either add real content and make the page worth indexing, or return a proper 404/410 status.
5xx server errors
The server failed to respond. Often caused by overload, timeouts, or hosting issues. Fix: check server logs and resources, ensure your host can handle crawl load, and resolve timeouts. Persistent 5xx errors can quickly shrink your indexed pages.
Blocked by robots.txt
A Disallow rule is stopping the crawler. Fix: open your robots.txt and confirm you're not accidentally blocking important pages, folders, CSS, or JavaScript. Remove or narrow the offending rule.
"Excluded by noindex" mistakes
A page carries a noindex tag it shouldn't. Common after migrations or when a staging setting leaks to production. Fix: remove the noindex from pages you want indexed, then request reindexing.
Redirect errors and chains
Redirect loops or long chains (A → B → C → D) waste crawl budget and can break. Fix: point redirects directly to the final destination in a single hop, and avoid loops.
After fixing: help Google recrawl
Once you've fixed an error, use URL Inspection → Request indexing in Search Console to prompt a recrawl. For pages and backlinks you want discovered faster, a link clicker helps get fresh URLs and links crawled sooner.
How to prevent crawl errors
- Audit indexing in Search Console at least monthly.
- Run a crawl after every migration, redesign, or CMS change.
- Keep internal links pointing to live, canonical URLs.
- Monitor server health and uptime.
- Maintain a clean sitemap containing only indexable, canonical URLs.
The bottom line
Crawl errors quietly cost you rankings by keeping pages out of the index. Start in Search Console's Pages report, fix issues by type — 404s, soft 404s, server errors, robots.txt blocks, stray noindex tags, and redirect chains — then request recrawls and monitor monthly. Clean crawlability is the foundation everything else in SEO is built on.
Frequently asked questions
What causes crawl errors? Missing pages (404s), server failures (5xx), robots.txt blocks, stray noindex tags, redirect chains, and DNS issues are the usual causes.
Do crawl errors hurt SEO? Yes. If a page can't be crawled, it can't be indexed or ranked. Widespread server errors can even reduce how much of your site Google keeps in its index.
How do I find crawl errors on my site? Use Google Search Console's Indexing → Pages report for a full list by cause, and a crawler like Screaming Frog for a deeper scan of broken links and redirects.
What is the difference between a 404 and a soft 404? A 404 correctly reports that a page is missing. A soft 404 shows a "not found" or empty page but returns a 200 OK status, which misleads search engines — so Google flags it.
How long does it take Google to recrawl a fixed page? It varies from days to weeks. You can speed it up by requesting indexing in Search Console and ensuring the page is linked and in your sitemap.
