One of the many things I love about SEO is that you often get to work on a wide range of projects, across various types of clients. I’ve had the opportunity to work with startups, ecommerce sites, big brands, small businesses, local businesses, etc. There are times that my schedule has me jumping from one end of the spectrum to another, yet all focused on Search. I love that, but along with working on a wide range of projects, you get to see the good, the bad, and ugly of SEO. And some of those situations are tough to witness. For example, when companies reach out to me after seeing a major drop in organic search traffic, and in a very short period of time.
Unfortunately, I can usually tell within the first 30 seconds of the call how bad their situation is. Their frustration and confusion is easy to pick up as I start to learn more about the SEO problem at hand.
The Dreaded Redesign or Migration
There are many reasons that a company could see a drop in organic search traffic. It’s sometimes easy to understand why that happened, while other times it takes serious detective work. One of my first questions to companies that have seen a big drop is, “Did you redesign your website recently, or did you migrate your site to a new CMS?” If you’ve been doing SEO for a while, you inevitably have come across the dreaded redesign and its horrific impact on SEO. In a worst-case scenario, all of the URL’s on a website change, the site loses a significant amount of Search Equity, and organic search traffic plummets faster than a lead weight in a backyard pool. On that note, I’ve had four companies reach out to me over the past few weeks that are the victims of this happening…
I hate when I see websites get absolutely crushed based on a redesign or CMS migration. It’s totally unnecessary, since a thorough migration plan could avoid the disaster. The problem is that many companies either don’t know they have to migrate their current URL’s to the new ones (via 301 redirects), or they receive bad advice from the people working on the website. I’ve seen both of these situations, and the resulting conversations are not pretty (once the company understands what should have been done).
The Importance of Finding Your Lost URL’s
If a redesign or migration has occurred without the proper redirection plan in place, it’s important to hunt down your old URL’s. You know, the ones that have helped your website build up its Search power over the past x number of years. If you can find those URL’s and properly redirect them, you could salvage some of your hard-earned Search Equity. This is because some of the old URL’s have built up valuable inbound links and were contributing to your rankings and search traffic prior to the redesign or migration. If you let them go, your site can pay dearly from an SEO standpoint.
Four Ways to Identify Your Lost URL’s
For smaller sites, hunting down old URL’s might be a relatively simple process to go through. But for larger sites (with tens of thousands of pages or more), the process of hunting down your old URL’s can be tough, time-consuming, and frustrating. In addition, many companies don’t know where to start, where to look, and what to do with the information. Therefore, I’m going to list four ways to track down those lost URL’s (if you are in the unfortunate situation of having lost them due to a redesign or migration). Using the four methods below, you can hopefully identify, document, and then redirect them to the proper place on the new site.
1. Google Analytics Search and Content Reports
One of the first things I try to do when a client has gone through a bad redesign or migration is to gain access to their analytics package. Since many clients use Google Analytics, I’ll focus on GA in this post. That said, you can use any package to help find lost URL’s.
The first thing you would want to do is to access your organic search reporting for a time period BEFORE your redesign or migration took place. Beware, this can sting since you are going to see all of the keywords that used to lead to your site. Since the keywords alone won’t help you, let’s find the landing pages that used to receive the Search traffic. You can dimension the Organic Search report by “Landing Page” to view all of the pages receiving organic search traffic. In the new version of Google Analytics, you can simply click the dimension “Landing Page” in the short list at the top of the report. See the screenshot below.
Once you have all of the old URL’s that were receiving traffic, you can export this report to analyze those URL’s in Excel. It’s a great start and can help you get on the right path. Unfortunately, this report will only show you the pages that have received traffic, and won’t show you pages with inbound links that haven’t received traffic during that time period. Those are extremely valuable as well. Don’t worry, I’ll cover how to find those below.
2. Google Webmaster Tools Crawl Errors
Google Webmaster Tools is an incredible resource for webmasters. There’s a boatload of data and functionality in GWT, and I highly recommend setting up your sites today if you haven’t already. For our purposes, I’m going to focus on the Diagnostics Tab. Within the Diagnostics Tab, you will see a link for Crawl Errors. This report will contain information directly from Google about the errors it is encountering as it crawls your site.
If you or your client has gone through a bad redesign or migration, then you will want to pay particular attention to the 404 (Not Found) errors. That’s Googlebot essentially telling you that it tried to access a page on your site, but couldn’t find it (which can mean that URL might be from your old site). In addition, it will show you the pages that are linking to the URL’s that can’t be found. You can view this information via the “Linked From” column in the report (located at the far right of the screen). This will enable you to see pages on your site that can’t be found and that have links pointing to them (some internal and some external). Needless to say, you would want to make sure these pages resolve or 301 redirect to an equivalent page on your new site.
From within Google Webmaster Tools, you can export this report to organize, analyze, and then redirect the old URL’s. Also, since this report shows you the pages that have inbound links, you are starting to address the lost link equity you experienced during the redesign or migration. We’re not done with inbound links yet, but this is a good start. See the screenshot below of the Crawl Errors report in Google Webmaster Tools.
A Note About The “Links to Your Site” Report
In addition to the Crawl Errors report, Google Webmaster Tools also provides a Links to Your Site report. You can find this in the Your Site on the Web section of GWT. Although this report shows you the inbound links pointing to the pages on your site, it will not show you links to pages that throw errors (like 404’s). Therefore, it’s not necessarily going to help you with the process of tracking down lost URL’s. That’s one of the reasons I focused on the Crawl Errors report instead.
3. Open Site Explorer
I’ve mentioned the importance of regaining inbound links in the previous two methods for finding your old URL’s. Again, during redesigns or migrations, URL’s often change. If those URL’s had valuable, hard-earned inbound links, then your website will lose the Search Equity it gained from having those inbound links. And as any SEO will tell you, a link is a horrible thing to waste. This is why it’s extremely important to perform a backlink analysis on the site in question. This will help you identify the pages linking to your site, along with the pages on your site that are being linked to. Those pages will probably be your old URL’s.
I use a number of tools for analyzing backlinks, but I’m a big fan of Open Site Explorer (from the good people over at SEOmoz). Once you run your site through Open Site Explorer, you can export the report to analyze, document, and then redirect the old URL’s. You can use the dropdowns at the top of the report to select “Links From External Pages Only”, and then “To All Pages on the Root Domain” to view all inbound links to the site. If you are new to SEO, definitely spend some time with Open Site Explorer to test out all of the functionality and to review all of the metrics. It’s a great tool and a valuable resource for analyzing inbound links. See the screenshot below to view Open Site Explorer in action.
4. An Archive From Your Web Developer or Agency
In my opinion, this is the X Factor for both recovering all lost URL’s and for understanding the way the old site was optimized. Some companies might have their entire old site sitting on a development server. If that’s the case, you can literally walk through the old site and identify the entire URL structure. Then you can analyze the old site, create a redirection plan, and then implement those redirects based on the original site.
In addition, you can also check out how each page was optimized, how title tags were structured, which keywords were used for important pages, how the navigation was structured and worded, etc. This can be incredibly helpful as you try and rebuild lost Search Equity.
Important Note: If you can access an archive of the old site, just make sure the old site isn’t publicly available. If it is, you can run into duplicate content issues (if the old site somehow gets crawled and indexed). If the old site has to be activated on a public-facing web server, then force any requests to a login. Once logged in, let that user freely traverse the site.
Recover Those Lost URL’s
If a redesign or migration has crippled your site from an SEO standpoint, then don’t accept SEO failure. Form a plan for hunting down your lost URL’s in order to redirect them. Depending on your site and your specific situation, you might be able to gain much of that lost Search Equity back. Check Google Analytics, Google Webmaster Tools, and Open Site Explorer to identify those lost URL’s. Then ask your web developer or agency if they have an archive of your old site. If they do, you should have everything you need to track down your old URL’s, and the inbound links they built up.
And when you’re done, you’ll realize that there’s a solid lesson from the experience. SEO should be an integral part of your website redesign, or migration to a new CMS. The good news is that you’ll probably never make this mistake again. 🙂
