Site Downtime & Negative Search Engine Rankings
All too often I come across sites that are slow or even go down for a few minutes. Worse, I’ve had clients sites go down for days at a time due to hosting issues. Many times the site owner does not realize how big of an impact this has on their rankings both now and into the future. In this article we go back to the basics by looking at how a search engine indexes your site, and what impacts there will be if the crawler can not find your site.
Untold problems can occur that can bring a website down and sometimes these things are outside of the website owner’s control. However the impact to search engine ranking can be even greater than the short term loss of website visitors. Most likely, the site will return to the index at a lower overall position and have to earn it’s rankings back while in rare cases, a website can be removed from a search engine index never to return.
What a Search Engine Does : When a site is added to a search engine index, either through a submission or a link found from another site or directory, it becomes part of the engines ‘regular’ crawl cycle.
Search engines send out programs called crawlers or spiders to retrieve these pages and ensure they have fresh copies of the pages in the index at all times. The crawler can find the sites pages many different ways, either from the submission made, or from a link on another site and also links within the site itself.
How Crawlers Work : Crawlers are much more advanced than they used to be. The search engines know which sites are the highest traffic sites, so tend to send out the crawlers to these sites more often during the day. Smaller sites, or sites that don’t change that often tend to see the crawler on a less frequent basis.
For the average site it is not uncommon to see a search engine crawler request hundreds of pages per visit. One may even see crawlers appearing on the site as different systems, including different versions of web browsers, or using different hardware platforms.
What Happens When a Crawler Can’t Find a Page? : Well, the first thing to say is don’t worry. Generally, a search engine crawler will attempt to revisit a page many times. The engine understands that due to web server traffic peaks and network congestion a site may be slow to load. While most indexing tends to happen at off-peak hours (such as during the middle of the night) a crawler will still visit during the day for various reasons, including testing the web servers responsiveness. If they find the web server is not reliable, they could consider penalizing the site(s) hosted on it. Usually, though, most crawlers will tolerate some slowness in websites.
Even after repeated attempts where a search crawler still can not connect to a site or page it will generally save that URL and attempt to visit it later. As mentioned, they do have a tolerance for slow servers, so will not penalize a site on the first slow visit.
However, after repeated visits, if they continue to receive errors they will likely consider penalizing the site. Generally it takes a few repeat visits before the search engine considers the penalization. What that threshold is, however, is anyone’s guess but it is suggested that the crawl delay is tied to visitor traffic.
In other words, a site that receives less visitors from the search engine may have a longer ‘grace period’ before it is considered too slow to be included in the index, while a site which receives more referrals may be bound to a shorter time between crawls, therefore allowing for less tolerance due to slow load times.
What Will Be The Result? : Initially, the site will likely be removed from the regular index after repeated failed attempts at indexing. That does not mean that the site is removed permanently, only that it has been temporarily removed. You may find the site or pages in Google’s supplemental index, for example. This index seems to be a catchall of slow URLs, new URLs and those on their way out of the index for various reasons.
Even after the site or pages have been moved the crawler will still attempt to connect to the site a few more times, and then if it does the site will come out of the ‘hold’ status it was placed into by the engine.
If, however, the site remains unavailable, then it may remain removed from the index permanently.
How Does One Get a Site Re-Instated? : Once the site has been removed it could take from days to months to get it re-indexed by the search engine. This too seems to be dependent on the traffic volumes generated by the site. A high volume site would likely get reinstated sooner (provided it proved it’s reliability to the engine) while a lower traffic site may take longer.
The search engine crawler will begin visiting to ensure the stability of the site, perhaps more often than usual. It may follow the same patterns it used to use, visiting many times per day, sometimes at peak times but generally at low times – or it may change its schedule in an attempt to test the stability of the site.
Once it has determined that the site is indeed stable, the engine may then re-include the site in the index. Once this happens, however, it is at the search engine’s sole discretion as to when and how the re-inclusion happens.
For example, the engine may determine that the site can be re-included at its former position and rank, or it may deem the site to be as a new site, and it will have to earn its way back to its previous position over time.
If the site had been down for a short time, say a few hours or days (again depending on traffic volumes), it could be re-included back at its previous position. If it had been down for a longer period of time, such as a week or more, it may have to earn its way back to where it once was.
What Determines This Re-Inclusion Process? : Many factors can influence the re-inclusion including the site’s ability to perform, its historical performance and even other sites competing in the same market.
For example, if the site in question was in a non-competitive market space, the search engine may decide to include it sooner to help build its inventory of sites in that category.
If the site in question belonged to a highly competitive category, the search engine may deem that the site is not as important as it once was, and therefore it may have to earn its position back, or the search engine may determine that the category was lacking during the time of the removal and will therefore re-include the site to its previous rank and position.
In any case, the ultimate decision on timing and position of the re-inclusion belongs to the search engine.
Summary : While it may appear that we are treating the search engine crawler and the search engine itself as sentient beings that is not the case.
It is important to note, however, that these programs are very sophisticated and they can seem to act as if they do understand what they are doing.
In reality they are nothing more than extremely complex mathematical algorithms making the ‘decisions’ alluded to within this document.
What is important to note is that the search engine could react quite negatively to a site that has been down even for a short time. Even being down for a few hours during the late night or early morning can have a devastating impact to search engine rankings. In rare cases some sites never recover the rankings the achieved prior to the outage.
Therefore it is of utmost importance that a website remain available to all users all the time with minimal to no downtime experienced.
If you are finding that your site is having issues I’d recommend reviewing your hosting contract and perhaps moving to a more reliable host.
–
Columnist Rob Sullivan is an SEO Specialist and Internet Marketing Consultant at Text Link Brokers
