Data-driven?
Think again.
The sentiment is nice. Makes you sound like an edgy growth hacker who growth hacks, growth hacks… Except there’s only one problem.
Your data is wrong. Not like, some of the time. But like, lots of the time.
Most of it’s not your fault. Problems pop up. Gaps from your third-party tools appear. But the end result is the same: false reporting that leads your response astray.
Some of it, however, is your fault. Inadvertent and innocent, sure. But once again, the end result is the same.
We’re in a results-driven business. Which means we must live and die by the sword. Whether we like it, agree with it, approve of it, or not.
Here are some of the most common reasons your data is wrong and how to sidestep, recover from, or prevent the consequences.
Your Search Data is Wrong
Here’s how SEO used to work.
Client paid you $X. You validated $X by showing them Y rankings and Z search traffic in return.
That worked great. Until it didn’t.
❌ First, the good ol’ Goog started personalizing SERPs, which kicked off a downward spiral of ranking relevance.
❌ Then they doubled down on webmasters with secure search, effectively stripping out all of your keyword referral data and leaving you with a big, hot, steaming pile of “(not provided)”. 💩
Stop smiling, friendly poop emoji. This ain’t funny.
We’ve already got major problems, and we haven’t even touched on link building.
Look.
SEO has never been very straightforward. We all know that when we signed up for this crazy roller coaster of a career.
But today it’s even less so. And that only continues to get worse. Because now you can’t even show how changes here or there are leading to better results over on the other side.
Sure, you can try. You can show them how your work is leading to increases in both organic search traffic and conversions from organic search compared to the prior year.
Except, there’s only one problem. Those numbers are wrong now, too.
Your Direct Traffic is Wrong
Everyone’s favorite startup punchline, Groupon, de-indexed themselves in 2014.
Here’s why they would do something so dumb so helpful in the name of SEO science.
Over the past few years, there’s been a mysterious rise in the number of direct sessions and conversions that websites are seeing. Coincidentally (or not), this started happening around the same time SERP personalization + secure search hit.
In theory, direct is supposed to mean people typing your URL into their address bar and hitting Enter. In reality, though, it’s not.
Groupon wasn’t buying the latest evidence, so they de-indexed themselves from Google in order to see the impact on their page sessions.
Unsurprisingly, traffic quickly dropped like a rock.
The most curious part, though, was how and where those traffic dips were seen. You’d assume that traffic from organic search would fall off.
But they also saw huge dips in “direct traffic” going to their Deal pages (which each featured long, hard-to-remember URL strings). In other words, the pages that almost nobody would every type in manually.
❌ TL;DR? Their findings showed that “60% of direct traffic is actually SEO.”
They had a few possible explanations for this. Certain browsers, especially mobile ones, don’t always report data correctly.
But the point is that on top of the first problem above, you also now have direct eating away at those SEO sessions and conversions that you so rightfully deserve.
Even more depressing is that this isn’t getting better. It’s only getting worse. Because Dark Traffic is taking over other referral source data, too.
Your Referral Data is Wrong
Dark traffic refers to all of the traffic you get that goes unaccounted for.
It might call itself direct. But as we just saw, that’s often a lie.
Organic search traffic, though, isn’t the only casualty. Email and social are, too.
For example, in the past few years:
- ❌ Pinterest traffic has been underreported.
- ❌ Twitter traffic has been underreported.
- ❌ Facebook traffic has been underreported.
Hell, we can’t even rely on email open rates. Haven’t been able to for nearly a decade now.
❌ So of course, email traffic is also underreported, too, if campaign tagging isn’t done properly.
Which means our problem is even worse than we realized.
Because all of our marketing channels – across the board – are wrong. Or at least severely flawed.
Which means we, as marketers, aren’t getting the attribution and the credit (and the resource allocation) we should be getting.
Sometimes, dipping into your pocketbook can solve the problem. Paying for AdWords traffic, for example, is the easiest way to bring that keyword referral data back in a jiffy.
But while taking the plunge into paid media generally provides us with more reliable data. That’s not always the case.
Your Advertising Data is Wrong
In 2015, Invoca analyzed over 30 million phone calls to see if they could identify a few patterns.
One result especially stuck out:
Seventy percent of phone calls come from digital channels.
That’s no small feat, considering that your phone is one of the best converting salespeople in your organization; with a close rate of 30 to 50 percent.
Great, right?
Except for one thing… how many of those phone calls are you accurately attributing right now?
❌ You might have AdWords call extensions set up. But those only track calls directly from the phone number on your ad.
Think about it:
As a percentage of your overall searches, how many times have you called the phone number directly in an advertisement?
Or – have you clicked on the website first, browsed around a little bit, before calling the number on one of the site’s pages?
❌ Chances are, none of those website-based calls were tracked. Even though AdWords is what delivered them.
Oh, wait. You think your conversion numbers are any better? Guess again.
Your Conversions are Wrong
AdWords Campaign #1 drives in five leads. AdWords Campaign #2 only two.
Guess what happens next?
The first campaign is Our Savior. It’s lauded with attention and interest and additional budget.
Except, as we’ve seen already, data doesn’t always tell the truth.
❌ For example, these are leads – not purchases.
What if the second campaign has a higher close rate? Or a higher average order value? Or a higher lifetime value?
One way around all of these problems is through closed loop analysis. You know: Lining up someone’s name and email and phone and credit card against this lead data.
❌ Except, oh shoot. You can’t. Because collecting Personal Identifiable Information (PII) violates Google Analytics’ terms. And they’ll delete your account if they catch it.
Then there’s the issue of this AdWords Campaign in the first place.
What if someone it’s not this person’s first visit? What if they’ve already been here multiple times through different channels?
After all, today’s typical customer journey involves multiple touchpoints before buying, which typically span several different channels.
❌ Except, you aren’t seeing any of that. You’re only seeing the last touch because that’s the default unless you’re some Avinash Kaushik-like analytics ninja.
Then there are the problems on the AdWords Campaign-level that you’re basing decisions from, too.
Trick question as an example: What is an “acceptable” cost per click to pay?
Is it $3 bucks? OR $30?
❌ The answer is that your CPC doesn’t matter. Not as much as you think it does, anyway.
Your Cost Per Lead matters. Your Cost Per Acquisition does even more. If the math works, a higher margin on those will allow you to aggressively bid up CPCs; stealing market share away from competitors because you can make up the profit on the back-end.
In other words, it’s often too short-sighted. Especially when only the long-term profitability is what matters most.
Your Tests are Wrong
A/B testing is fun.
Fun to read about. Fun to blog about. Fun to discuss on conference stages.
Except, of course, when faced with reality.
❌ The first issue is statistical significance. Many tests fail because you need a minimum of 1,000 monthly conversions and 250 per test to get anything remotely accurate.
Otherwise what often happens is that those small fluctuations in your favor ultimately regress back to the mean over time.
Case in point: form fields.
❌ Reduce them and you can instantly increase your conversion rate. BUT, increases in landing page conversion rates end up in lower quality leads in many cases.
That’s like not asking for a credit card on free trials. Your conversion rate will go up, but more and more of those free trials will be “useless.”
Years ago, Moz studied this exact issue.
❌ Contrary to popular belief, customers who converted on the first or second visit were NOT their best customers. They either wouldn’t complete the trial, or they would be among the first to churn.
Instead, those that visited their site over eight times had the highest lifetime value. They were given enough time to browse, learn, interact, and grow affinity prior to handing over their credit card.
And the patient proof was in the bottom line.
Conclusion
The point of this wasn’t to bum you out.
At least, not entirely.
It was to open your eyes. To get you to think.
Too often, we’re data obsessed. Even though data falls victim to many of the same problems and biases.
It ain’t perfect. In fact, as you can see, it’s far from it.
Data matters. To a point. In certain cases.
You know what else matters?
Branding matters. Loyalty matters. Customer experience matters.
And you can’t always get data on any of that.
But it’s OK.
Because data is often wrong, anyway.
Image Credits
Featured Image, Brad Smith June 2017
