In a recent LinkedIn post, Google Analyst Gary Illyes raised awareness about two issues plaguing web crawlers: soft 404 and other “crypto” errors.
These seemingly innocuous mistakes can negatively affect SEO efforts.
Understanding Soft 404s
Soft 404 errors occur when a web server returns a standard “200 OK” HTTP status code for pages that don’t exist or contain error messages. This misleads web crawlers, causing them to waste resources on non-existent or unhelpful content.
Illyes likened the experience to visiting a coffee shop where every item is unavailable despite being listed on the menu. While this scenario might be frustrating for human customers, it poses a more serious problem for web crawlers.
As Illyes explains:
“Crawlers use the status codes to interpret whether a fetch was successful, even if the contents of the page is basically just an error message. They might happily go back to the same page again and again wasting your resources, and if there are many such pages, exponentially more resources.”
The Hidden Costs Of Soft Errors
The consequences of soft 404 errors extend beyond the inefficient use of crawler resources.
According to Illyes, these pages are unlikely to appear in search results because they are filtered out during indexing.
To combat this issue, Illyes advises serving the appropriate HTTP status code when the server or client encounters an error.
This allows crawlers to understand the situation and allocate their resources more effectively.
Illyes also cautioned against rate-limiting crawlers with messages like “TOO MANY REQUESTS SLOW DOWN,” as crawlers cannot interpret such text-based instructions.
Why SEJ Cares
Soft 404 errors can impact a website’s crawlability and indexing.
By addressing these issues, crawlers can focus on fetching and indexing pages with valuable content, potentially improving the site’s visibility in search results.
Eliminating soft 404 errors can also lead to more efficient use of server resources, as crawlers won’t waste bandwidth repeatedly visiting error pages.
How This Can Help You
To identify and resolve soft 404 errors on your website, consider the following steps:
- Regularly monitor your website’s crawl reports and logs to identify pages returning HTTP 200 status codes despite containing error messages.
- Implement proper error handling on your server to ensure that error pages are served with the appropriate HTTP status codes (e.g., 404 for not found, 410 for permanently removed).
- Use tools like Google Search Console to monitor your site’s coverage and identify any pages flagged as soft 404 errors.
Proactively addressing soft 404 errors can improve your website’s crawlability, indexing, and SEO.
Featured Image: Julia Tim/Shutterstock
