Search engine marketers have come to rely more heavily on work-arounds in what everyone’s calling the era of “Not Provided.” The beginning of this era was like waking up “Johnny Got His Gun” style and having no arms, legs, or resources to compete in the hunger games of Google SERPS.
Understandably so, Analytics keyword data was helping all of us to make actionable decisions for our companies and our clients. The keyword data was useful because we could see exactly how our clients found us and how differently users behaved when using these intent-rich words. Once this data was no longer available, a new quest began: to fill in the data-gap needed to guide search marketers with valuable insights.
Here’s a complete (non-sponsored) roundup of analytical tools and techniques to help you get closer to the information Google is no longer providing.
The Biggest Loss in SEM History: Not Provided
In case there is one remaining digital marketer who hasn’t noticed the increasing Not Provided proportions in Google’s organic keyword data, let’s just sum it up in a few lines here:
- Google began SSL encryption for search users signed into a Google property (ie. gmail) in 2011 to provide privacy to the user.
- Secure search has also meant that while Google will still recognize organic traffic in Google Analytics, it will “not provide” the query term the user searched with within the analytics report.
- Data encryption spread from only users signed in to Google properties, to the omnibox of different browsers in 2012 and now most browsers default to secure search.
Addendum: Not Provided now accounts for practically 90% of organic search traffic in Google Analytics.
On the ground this means marketers cannot tell their clients which keywords their visitors used to find them:
Screenshot taken 6/22/2014 of data that is increasingly Not Provided per session over time in Google Analytics
Analytical Strategies to Compensate for our Losses:
The tools and techniques to counter the Not Provided keyword famine come from within Google and from outside sources. Funny enough, many of Google’s tools compensate for the loss (kind of). There are also third-party solutions who have stepped up to help provide some of the missing data. Here are the best and most bragged about analytical solutions to all your Not Provided problems:
Google Taketh Away and Google Giveth: Strategies Within Google Products
Google AdWords and Google Webmaster Tools: The keywords that are no longer provided in Google Analytics remain available to a certain extent to anyone generating a Search Terms Report (API version still called search query performance report) in Google AdWords or a Search Query Report in Google Webmaster Tools. While it does seem convenient that the data they took away was still available to paid users on their advertising platform, thank goodness it is!
In April, search marketing blogs were fraught with rumors that Google would soon stop giving keyword data not just to Analytics, but also to their paying AdWords customers. Larry Kim cleared that up pretty quickly and wrote on Search Engine Land that not much has changed for SEMs who were previously accessing search query data in their AdWords reports.
Screenshot taken 6/26/2014 from Google’s Search Terms Report Promotional Video
Larry explained that keywords would still be available in AdWords and even in third-party PPC platforms like Acquisio. The only thing that changed was that the search query would no longer be included in the referring URL string. Remember, though, that the tips included in this section are only for the paying AdWords customer.
Google Webmaster Tools (GWT) and AdWords always offered keyword data but was largely out-matched by Analytics. Now they’re that much more valuable following the phase out of organic traffic keywords. Here is an image of a GWT Search Query Report:
Screenshot taken 6/28/2014 from Avinash Kaushik’s Epic Secure Search Blog Post
At first glance, the untrained SEM eye may get wide seeing this data. While there is some reason to get excited, this data is not exactly what it looks like. The GWT Search Query Report is showing how many times your site showed up in Google search results for the query searched and then how many clicks you received for that query. This is highly valuable information for any SEM, but doesn’t replace what has been lost in Google Analytics.
First, this GWT data is based on a different dataset than Google Analytics. The other major difference is that this report says nothing about how those visitors behaved on your site – data you would have used to make major marketing decisions that are no longer there.
That said, if you’re not using AdWords or Google Webmaster Tools, let’s just start with some Google Analytics workarounds.
Landing Page Reports in Google Analytics: Formerly referred to as a hack, the landing pages report technique is more about knowing what you’re doing with filters in Google Analytics.
What is this hack/technique I’m referring to? It is a small tweak in your Analytics set-up and reporting that allows you to connect the Not Provided data to landing pages.
Dan Barker wrote about this Analytics hack on eConsultancy back in 2011 when Not Provided was only up to 33%. He goes through the Analytics setup in great detail, saying the hack looks for the not provided keywords and reports back to the landing page this search visitor landed on, providing a vital clue to site owners. Dan provides an image by image breakdown of what this would look like:
Screenshot taken 6/28/2014 from Dan Barker’s Visual Breakdown of the Landing Page Hack Before and After
Screenshot taken 6/28/2014 from Dan Barker’s Visual Breakdown of the Landing Page Hack Before and After
You can see in Dan’s before and after screenshots above that the inactionable Not Provided data transformed to a list of landing pages that include URL based keywords on those pages. User beware – your site would have to have search-relevant keywords built into the URLs for this to be as easy as it looks above.
Have a look at “np-/cheap-laptops” above for example. If you don’t have clean, search-friendly URLs like this one, this technique hack will require more work. If you have one of those sites with a bunch of “.com/?wuzup+4p-en&prodid=184645” URLS you can still apply this landing-page technique, but you’ll have to go one step further and map the title tags of those landing pages to get the full clue.
You can read the step-by-step break down of the landing page technique written much better than I ever could here. No seriously go to that blog and set up Analytics now so that you can associate Not Provided traffic with the landing pages of your site. What you’ll get is a better branded/non-branded breakdown of your traffic acquisition.
Now, let’s go one step beyond the landing pages.
Avinash Kaushik talks in detail about the landing pages report and actually provides the customized aggregate landing pages report template here, but you have to apply organic to the set-up he provides (I wanted you to try for yourself first! That’s the only way you’ll learn!). Once you get your list of landing pages, Avinash talks about how you can click on each of them to drill down into the keywords that drove the organic traffic to that page.
Unfortunately much of that data isn’t provided either, but you can still see some keywords.
Screenshot taken 6/22/2014 of landing page data that is increasingly Not Provided in Google Analytics
Avinash warns that basing any client advice or strategy on the remaining keywords that are provided is dangerous for SEMs. He says that without knowing what is in the rest of the Not Provided bucket you would be basing your conclusions on assumptions; it would be like assuming you know how a lake formed based on a couple of raindrops you saw fall into it. He also points out that you can still report on your other organic metrics, like “Unique Visitors, Bounce Rates, Per Visit Value and many others, at an aggregated level,” but that’s not great since we know everyone prefers segmentation, even Avinash himself.
Google Analytics workarounds? Check!
User Intent and Behaviour Analysis: Earlier this year I put together a blog post for SEM Rush talking about enriching campaigns using a user behaviour analysis. In it, I talk about the fact that User Segmentation features were added to Google Analytics last year, allowing us to see different user personae in our client’s target audiences, making the point that “effective analysis and marketing is shifting toward user-centric and personalized experiences.” I also refer again to Avinash Kaushik’s popular blog post that discusses creating advanced segments in Google Analytics in addition to keyword research that is more targeted to the region, which in the end is more targeted to the user. This post that I, and many other SEM’s, have referred to is well-rounded and gets us to think past what’s Not Provided.
Avinash begins his post reminding us that website traffic can come down many different channels/funnels before landing on any particular site. With that in mind, we’re reminded that it’s a huge mistake to focus only on the last click before your site. Instead, he suggests getting a sample of which funnel your traffic is coming down and allocating your marketing budget accordingly (direct, paid, organic etc). The same search query that drove traffic to your site organically, could still come down a paid or direct channel. So what if we go to the root of all funnels? What is at this root? The searcher’s intent.
What matters most is getting into the head of your customer here and figuring out what their intention was. Mike Walker wrote on the Conductor blog earlier this year, remarking on the role of SEO and how it has evolved. He says:
It’s true that there are new challenges. Google has encrypted keyword level data. We no longer have the insights of the past. We are seeing a smarter consumer with more expectations. The consumer is in control. Direction is needed beyond one vertical.
SEO today means keyword research based on the bigger picture.
Andrew Davis wrote on Search Engine Watch how to do advanced keyword modelling to research keywords based on user intent. He explains the nature of search types, categorizing them as navigational, informational, and transactional. Andrew talks some sense here by saying many of us may be optimizing for information searches with high volume, while forgetting that the intent behind this type of search will ultimately lead to a low conversion.
So, he advises us to bid on more transactional type of search queries to improve ROI. To figure out the types of transactional queries your customers are using, he suggests simply listening to them. Listening to your customer can be done using software and cloud products like customer experience analytics, but ultimately you will be using this customer data to feed it back into your keyword research and advanced modelling.
Think about what we’ve covered thus far in this article – so many different areas to gather missing data from. All of this data is going to help us understand our site traffic better, our potential clients, their journey to our website and the intention that brought them there in the first place.
All of this data can be factored together to get a larger picture of user intent based on the data we do have. Andrew Davis breaks down this theory in his post:
Screenshot taken 6/28/2014 from Andrew Davis’ Article on Search Engine Watch
Please see Andrew’s full post here to get the step-by-step breakdown of advanced keyword modelling and user intent. Remember that each business will apply a different variation of the model. The point is that your customers are unique and you’re going to have to figure out who they are, how they buy your product, how they are funneled towards your site AND THEN plan and organize your keyword research and strategies around them.
This would have been easier with a list of organic keywords yes, but the upside is that all of this extra leg-work is going to mean you actually understand your customer, not just the word or phrase they typed without any context or understanding of their journey.
Onsite Search: You can also have a look at your onsite searches for keyword discovery (meaning look at what your traffic searched for within your website). Assuming you have a search bar on your website and are using onsite search in Google Analytics, you can get an idea of what types of keywords your site traffic is using to look for your products and services. I didn’t include this technique in the meat of the Google Analytics section because it isn’t referral traffic keywords that led users to the site.
The interesting thing about internal site search is that users are already familiar with the niche of the site they’re on. So more often than not, the internal site searches will help you discover popular long tail keywords within your niche. It can also help you discover keywords you are completely off-radar. For example, let’s say you have a t-shirt site and you notice everyone is searching for “sad t-rex” logos within your site because a meme about that went viral. In this case, research what the fuss is all about and print some “sad t-rex” t-shirts immediately!
Only Half the Battle
Ok, so far in this roundup of digital analytics in the era of Not Provided we’ve covered tools and techniques within Google’s suite of web products (AdWords, GWT and Analytics) that can be used to help you recover some of the organic search data and then we’ve looked at all of these combined in a way that is customer centric. You can probably already assume that if Google is no longer providing you with the keywords your organic traffic used to find you that you won’t be able to look to Google to get that information back in any more of a direct manner than the pages of explanation provided above. That said, where there is a gap, there is an opportunity.
Provided: Strategies Outside of Google Products
You may be asking, isn’t there something that can just analyze search queries like Google Analytics used to? A few well-known SEO giants are making the best of digital analytics in the era of Not Provided and attempting to give us SEMs back the data we lost. There is of course a price to pay for this data. Here are companies that are executing workarounds and applying fancy algorithms for us, in turn giving us back some of our lost Not Provided data with the least amount of effort.
Conductor’s True Traffic Solution: Conductor launched their solution to the organic keyword data gap in 2013. The solution looks at which keywords are being used to drive traffic to your site and how much traffic these words bring to each page of content.
Let’s drill down. Conductor’s True Traffic solution is able to figure out which organic keywords drive traffic to your site in the following four ways previously described on their blog:
- “We take the vast sets of keywords our customers already know are being used to find their sites and we discover if any of their existing pages show up in the Google results for them. This creates a relationship between those keywords and a page from your site.
- We have an enormous index of keywords that we track where we then “reverse” out your content against this index. For example, if in our index is the keyword “digital cameras”, we then identify if you have any content for this keyword here. Once again, we create a keyword to page relationship.
- We leverage partner indexes like SEMrush, and we do the same thing as the above – just with their index.
- And lastly, we leverage the keyword data from other major search engines, like Bing, and map those keywords to your content.”
True Traffic creates a complex map of short-tail keywords that are mapped against your content and then applies an algorithm that estimates the amount of traffic for each keyword (future versions of True Traffic will estimate traffic for very long-tail keywords also). Their algorithm is based on online visibility (rank position), consumer demand (search volume) and consumer behavior models (proprietary analysis of historical data and engine data). Interestingly, anyone who uses True Traffic can also add to the consumer model data with any of their own unique data and insights. According to Conductor “the more you tell the system, the more comprehensive this will be.”
Screenshot taken 6/28/2014 of Conductor’s True Traffic Solution from their Blog
SEO Monitor and gShift make very similar tools and some are more helpful than others. The caveat of these tools is the fact that you have to know which keywords you’re looking for, meaning these tools cannot be used for keyword discovery. So if you want to get really close to pre-Not Provided era, you’ll have to go back to Google’s keyword tool in Adwords to do some research before taking your researched list of keywords to these third-party solutions to track for you.
Analytics Data Enrichment with gShift’s Page Level Position Data Formula: gShift created a tool that provides page-level position data that connects you back to the keywords no longer provided. Their tool does this by factoring together three different variables for each keyword the client tells it to track. These factors are:
- Page-Level Position
- Monthly Search Volume
- Estimated Click Through Rate (CTR)
Analyzing this trifecta of data, gShift calculates the position of your webpage and the search volume for particular keywords you want to track. Here is how they describe it:
We know that this web page is in position #1 for the keyword “barrie condos.” We also know there are 207 searches performed on this keyword throughout the month. Finally, we also know the estimated CTR for Position #1 is 36.4%. Therefore, over the last 30 days approximately 75 organic searchers visited this web page though the keyword “barrie condos.” That’s 207 * 36.4% = 75
Lets take a look at what all this factoring looks like on the SEMs end:
Screenshot taken 6/25/2014 from gShift’s Not Provided Intelligence and Insights Guide
Keylime Toolbox: Ginny Marvin wrote on Search Engine Land earlier this month that Keylime Toolbox and Moz have both debuted tools to assist with the recovery of Not Provided data. Keylime Toolbox is a start-up founded and run by former architects of Google’s Webmaster Tools who recently launched a toolbox. The toolbox claims to “recover” about 40-60% of your Not Provided keywords.
The Keylime toolbox uses Google’s suite of information to gather data, then it aggregates and de-dupes the data. The toolbox can calculate your performance impact and it also segments branded and non-branded search queries.
Performance and trend reporting includes impressions, ranking, click through rates, traffic, bounce rate, time on site, and other metrics by segment and individual query.
Screenshot taken 6/29/2014 from Keylime Toolbox Product Tour
Screenshot taken 6/29/2014 from Keylime Toolbox Product Tour
Keylime is able to give you all this information through Google Webmaster Tools. The idea is that GWT has query logs unaffected by secure search, but will only provide the top queries for a website unless the site owner adds subfolders for the entire site structure.
The problem is that these subfolders have to be set up and the query data has to be downloaded, merged, and de-duplicated…everyday. That’s where Keylime Toolbox comes in to do all that for you, providing you with more organic search queries. Pretty awesome stuff from a new kid on the block, considering this toolbox is more something you’d consider from a lifer like Moz…
Moz’s New Landing Page Report: Similar to the technique I talked about using landing-pages in Google Analytics, Moz has streamlined and added to the technique of tracking Not Provided keyword data by linking them to landing pages with their new New Landing Pages Report. Anyone using Moz Analytics in combination with Google Analytics will have access to these new features that are helping to compensate for lost data.
Screenshot taken 6/29/2014 of Moz’s Screenshot of their New Landing Pages Report in Moz Analytics
The new report features are available in Moz Analytics by clicking on Landing Pages, under the Search tab of your Dashboard. Here you will now find an estimated share of the organic traffic and keywords per page on your site. The report factors most of what other tools are considering in their calculations, including “landing page traffic, keyword ranking position, estimated search volume, and SERP click-through-rates.” You can check out Moz’s sample of the report here, where they also tell you how to get a 30 day free trial.
Time to Round Down
So you woke up one day with no leg to stand on. What are you gonna do about it? Use technology!
We never lost search volume in the Not Provided sweep and this is the most strategic starting point once you’ve really taken the time to understand your customers and their behavior. Put all of these methods together and cross-validate your keywords, your content, and your client strategy.
The techniques and tools shared here give you the actionable advice the keyword data was giving you before Not Provided. There is a difference in the data to say the least, but in the end the decisions you make will be guided by search volume data, Google Adwords, exit pages, page position data, user segmentation, customer analytical enrichments, and even just by buying this information from SaaS providers.
Google is going to keep you on your toes, but you’ll look back one day and thank them for how much you’ve grown over this era.
Do you have a strategy for Not Provided that hasn’t been included here? Please share it with our community below.
All images used with permission; featured image via Unsplash (completely royalty free)
