Controlled Test Results: Facebook Organic Reach is Under Reported

697 Shares Facebook 279 Twitter 245 Buffer 82 Google+ 49 LinkedIn 35 Pin It Share 7 697 Shares ×

facebook organic reach under reported Controlled Test Results: Facebook Organic Reach is Under Reported

If you read the pages on this website, you know what to expect. Fact and data based information, (mostly) unaffected by emotion.

Since my primary goal is accuracy, I don’t try to be first when there’s breaking news. And I’ll hold off publishing a post until I have all of the facts.

Luckily I took this path recently because I was on the verge of writing a post that would have reported that Total Reach is over reported. And that would have simply added fuel to the fire for anyone upset about a drop in Reach.

But I took my time. I ran a controlled test. And what I discovered was a pleasant surprise.

Facebook Organic Reach is under reported — specifically when you promote posts.

Sure, that’s not necessarily “good” news. We want Facebook to be accurate. And as we’ll see soon, what appears to be inaccurate data may actually be by design. But it’s always nicer to hear that stats are better than you initially thought.

This post is going to break down the following:

  • What I initially found that sparked concern
  • The possible explanations
  • The controlled test
  • The results

What I Found

You may know that I’ve been putting together the lessons for my Insights training course. I had completed the definitions of all important terms, an overview of web Insights and was giving a tour of the new post level export.

In the definitions section, I covered Organic Reach, Paid Reach and Total Reach. I talked about how Total Reach will never equal Organic Reach + Paid Reach because you’re bound to reach people both organically and with your ad.

When I got to the post level export section, I revisited this. I used my data as an example to show how this was the case.

There was one, big problem: In the example I was going to use, Total Reach equalled Organic Reach + Paid Reach.

I looked at more posts within that export. Same thing. Then I exported five months of data.

That export included 390 posts. Of those 390 posts, 71 had been promoted. Guess what? In every single case, Total Reach equalled Organic Reach + Paid Reach.

Keep in mind that this was using the new export. The old export wouldn’t be relevant because of Viral Reach (which is now folded into Organic Reach).

The same problems I spotted in the new post level export were found in web Insights. Here is an example…

facebook web insights reach problem Controlled Test Results: Facebook Organic Reach is Under Reported

This is a post that I first published organically and then promoted only to fans. So in that case, you know there will be overlap between Organic and Paid Reach. But the sum of the two equals Total Reach.

I was immediately alarmed by this because I know it’s not the way things are supposed to work. In fact, I was able to find this entry in the Facebook help center titled, “Why does the sum of paid reach and organic reach differ from the total reach?

Total reach counts the number of unique people who saw your posts, regardless of where they saw it. If your post reaches a person organically and through an ad, that person will count as one for organic reach, one for paid reach and one for total reach.

That entry is three months old at time of this blog post.

Exploring the Possible Explanations

This was a bit alarming. I could think of only three potential explanations:

  1. Users were only being targeted organically or via an ad, not both
  2. Total Reach was being over reported
  3. Organic Reach was being under reported

I immediately eliminated the first option since I know for a fact that users can both be reached organically and see an ad.

Let’s explain a bit more what #2 and #3 mean…

Over Reporting Total Reach
Let’s say that you create a post that reaches 1,000 people organically. You then promote it, mainly to fans, and reach 1,000 people with that ad. We’ll say that 300 of the people reached with the ad were also reached organically.

If Facebook is over reporting Total Reach, they would say that your ad reached 2,000 people. In reality, it would have reached 1,700 people, or 300 of those people twice. In this case, Facebook would be over reporting Total Reach by 17.6%.

Under Reporting Organic Reach
We’ll stick with the same example as above. This time, though, once your ad reaches those 300 people who already saw it organically, Facebook changes those people to Paid Reach only. So you now reach 700 people organically, 1,000 people paid and 1,700 total.

In this case, Total Reach would have been correct, but Facebook was under reporting Organic Reach by 30%.

The more I thought about it, the more I was convinced: Facebook was over reporting the Total Reach of posts when also promoted.

It just seemed most likely. But I knew I was dealing with a very sensitive situation. People are going absolutely nuts regarding the recent drop in Reach. If I wrote that Facebook was over reporting Total Reach — all while people were upset it was too low — things were going to blow up.

But I also think that Reach is a seriously overvalued metric, and I don’t want to be the center of such a firestorm. That is, unless I could verify my theory to be true.

A controlled test was necessary.

The Controlled Test

I held off on the blog post. I knew that it was something I could test and prove (or at least provide convincing evidence for) one way or the other.

Following is what I did…

First, I created an organic post that was very tightly targeted. The expectation was that only my wife would see it (and I, as the admin, would also be able to see it).

facebook post test Controlled Test Results: Facebook Organic Reach is Under Reported

Since post targeting only applies to the News Feed, I immediately hid it from my timeline. You may have seen early tests where I forgot this important step.

I then went to my wife’s computer and checked her Facebook News Feed to see the post.

As you can see in the image above, only two people saw this post. This screen grab was taken four hours after it was published.

It was safe at this point to promote it. So I then went into Power Editor and created a Custom Audience for my wife only. I promoted that post, making sure that she would be the only person seeing the ad.

When this test completes, the following should be true:

  • Organic Reach = 2 (My wife and me)
  • Paid Reach = 1 (My wife only)
  • Total Reach = 2 (My wife and me)

Here are the results I was looking for…

1. Organic Reach (2) + Paid Reach (1) = Total Reach (3). If this happened, Facebook was over reporting Total Reach.

2. Organic Reach (1) + Paid Reach (1) = Total Reach (2). If this happened, Facebook was under reporting Organic Reach.

The Results

Here is a screen grab of that post, 13 hours after it was published and eight hours after it was promoted…

facebook post test 2 Controlled Test Results: Facebook Organic Reach is Under Reported

As you can see, after the promotion, Facebook has lowered this post’s Organic Reach from 2 to 1. Total Reach is correct at 2 and Paid Reach is correct at 1.

I went into the new post level export and confirmed the same data. Organic Reach, which had been at 2 within the export prior to promotion, fell to 1.

So, this confirmed that Facebook wasn’t over reporting Total Reach. They were instead under reporting Organic Reach.

Now, however, I needed to figure out why. Or at least the source.

I found a clue within the old post level export.

facebook old post level export reach Controlled Test Results: Facebook Organic Reach is Under Reported

As you can see, the numbers are precisely as they should be in the old export.

What Happened?

Based on these results and the three month old help center entry, it’s quite clear that Facebook has changed the way they handle Organic Reach.

We already knew that Organic Reach changed. This was as a result of Viral Reach going away, and being folded into Organic Reach.

However, Facebook’s decision not to count Organic Reach for a user once they are reached with an ad is completely new. And this change was applied across web Insights as well as the post level export.

When I asked about this within the Developer forum, the response I received — as difficult as it was for me to understand — was that this was by design.

Why is it by design? I really have no idea.

If it actually is by design, I guess you could say that Organic Reach isn’t under reported at all. It is what it’s supposed to be.

But I counter that with the help center entry. No warning was given to this change. And there’s no good explanation I can think of for why this change makes any logical sense.

So until I hear otherwise, I will continue to refer to Organic Reach as under reported when a post has been promoted.

Now What?

At this point, it’s still not clear if this is a bug or intentional. I did receive one response that indicates it’s intentional, but that’s not an open and shut case. I still think there’s a good chance it’s a bug.

Some will try to connect an under reporting of Organic Reach with Facebook trying to get you to advertise. Stop. That’s not what this is.

If Facebook wanted to intentionally deflate Organic Reach to get you to advertise, they would do so on posts that weren’t promoted. This only impacts posts that have already been promoted.

Rest assured that when you promote posts, your Organic Reach is actually higher than you think it is. Your true Organic Reach will be a mystery since you won’t know how many of those you reached with your ad were also reached organically.

In the end, my guess is that a small percentage of marketers look all that closely at Organic Reach anyway. The focus is likely on Total Reach. And that metric remains accurate.

For me, this is just one more example of why Reach isn’t a metric you need to worry about. Facebook can change the way they report it. They can change it again tomorrow and make you think you reached double the people. It just doesn’t matter.

Focus on the metrics that lead to your business goals. Those could include likes, comments, shares, link clicks, sales and other conversions.

Don’t waste your time with Reach.

Your Turn

What do you think about this revelation? Let me know in the comments below!3

697 Shares Facebook 279 Twitter 245 Buffer 82 Google+ 49 LinkedIn 35 Pin It Share 7 697 Shares ×
About Jon Loomer

Jon Loomer is a digital marketing consultant with a unique perspective on social media. He was introduced to Facebook in 2007 while with the NBA (back before Pages) and has been using Facebook for business ever since. Stay in touch by liking his Facebook Page (Jon Loomer Digital).

  • Lukas Krejca

    Great test Jon! We see that Facebook can change its metrics anytime and it’s hard to catch all these changes. That’s why I think the most important is the end of your article “Focus on the metrics that lead to your business goals. Don’t waste your time with Reach.”

    • http://jonloomer.com/blog Jon Loomer

      You’ve got it, Lukas!

  • http://jasonhjh.com/ Jason HJH

    I like how you conducted such a thorough experiment for most of of the marketers who would be extremely concerned with Reach – and then cut short their celebrations by telling them “don’t waste your time with Reach.” Well played!

    • http://jonloomer.com/blog Jon Loomer

      Haha, yeah. I know Reach is a popular topic. So when I stumbled on this, I knew I had to get to the bottom of it.

      But in the end… the inconsistencies just provide more fodder for why the metric isn’t worth our time.

  • Vincent Vizachero

    Jon, if I’m understanding the impact correctly then ONE upshot would be to lower the cost per impression for Facebook. By counting every paid impression as “Paid” instead of “Organic” (even if the fan also saw the ad organically) then paid reach could be considered to be inflated.

    For instance, in your controlled test there was a third option:

    3. Organic Reach (2) + Paid Reach (0) = Total Reach (2).

    I’m not arguing that Facebook SHOULD count dual-views this way, but they COULD. And the fact that they don’t makes it easier for them to deliver paid impressions at lower cost.

    AND by counting people as paid who already saw the ad organically, they are also giving the ad credit in the ad report for the engagement that came from those people. Who, because they saw it organically, we presume are more inclined to engage.

    Again, I’m not suggesting anything nefarious. But this does make promoted posts look more successful than they would appear under other counting schemes.

  • Phil

    The reason for almost any change at Google or Facebook normally comes down to influencing user behavior for a better business result – for them. Looking at this change through that lens, the reason for this change is like that advertisers looking at these stats are going to believe that they need to promote posts in order for them to receive a large incremental reach versus just an organic post. This will become even more impactful as normal organic reach continues to decline over time, as you reported is Facebook’s plan in one of your recent posts.

    • http://jonloomer.com/blog Jon Loomer

      I don’t see how this is connected to driving advertising since Organic Reach is only under reported on posts that were promoted.

      • Sylvie St-Amand

        Sorry to have to point this out, but Organic Reach is down all over the board – not only with posts that were promoted. Phil’s argument seems pretty solid to me as Facebook appears to be going on all fronts to convince people to buy ads.

        • http://jonloomer.com/blog Jon Loomer

          We know that, Silvie. This post is not related to that.

          • Sylvie St-Amand

            I am certainly not undermining the value of your post, Jon. Your experiment is really interesting and valuable. Simply, everything that has been happening with FB Reach lately, whether it be with organic or paid posts and taken globally, may appear to be going in one direction, which is to ‘promote’ their promoted posts.

          • Phil

            Technically Jon is correct. I do most of my management on the Adwords side and didn’t realize that this was only relating to paid posts on Facebook. That said, I still believe the motivations I mentioned above were likely the driving factors behind the decision of how to display the stats you are discussing in this post, even if its not relevant to posts that aren’t promoted. Displaying the stats this way still makes it look, proportionally speaking, like the paid advertising is more valuable than it is, does it not? And therefore to the advertisers judging the success of their campaign, it will seem necessary to continue advertising to achieve their desired reach, especially as the default organic reach continues to decline.

          • Sylvie St-Amand

            I tend to agree with you Phil. My comment may have seem unclear or beside the point since Jon’s experiment relates to paid posts. I was only referring to the intent and motivation that brought on this specific change, which I can’t help but perceive as a way to favor the advertising side of Facebook.

      • JamesDiGioia

        Because it makes promotion look more valuable (providing more of a boost over just Organic Reach) than it’s actually providing.

        • http://jonloomer.com/blog Jon Loomer

          Not really. Because again, the typical marketer probably wouldn’t break it down this way. They’d probably compare the Total Reach of a post that was promoted to one that wasn’t. That remains accurate.

          • JamesDiGioia

            Except those comparisons aren’t going to be totally accurate because the results are going to be impacted by a whole host of other factors, whereas a comparison between organic and paid reach on a single post could be accurate (if the numbers were calculated accurately).

            I also think that assumes a level of sophistication of those buying Facebook ads – we’re digital marketers, so we know better, but I don’t think everyone buying Facebook ads necessarily does.

          • http://jonloomer.com/blog Jon Loomer

            Bottom line: Reach is an inaccurate and murky stat that can be defined differently on the run, and you’re fighting with fire if you base your strategies or decisions around it.

  • Emeric

    That’s a heck of a test :-) In the end, the issue is hard to solve, when a user sees a post twice, once organically and once sponsored, as “reach” is an overall unique metric, where should she be accounted for. Not sure there is a “right” answer to that. At least, with the way they do it, organic reach only account for the user who only saw the post organically. That’s an angle of view that makes some sense. Sharing your post on our page now :-)

    • http://jonloomer.com/blog Jon Loomer

      The way they had it before was fine! If I reach my wife once organically and once paid, I should know that! That’s still a total reach of 1.

  • Luciano Larrossa

    Great post Jon! The Lima Loomer example was fantastic. Congratulations! ;)

    Best regards,
    Luciano Larrossa

  • Funny Monkey

    I suspected this. Glad to see someone publish on it. (And I have no idea why my name posts as Funny Monkey. I can’t seem to make that go away. lol)

  • http://www.Smead.com/ John F. Hunt

    Reach has its place in marketing metrics because it can be one indicator on how efficient you are at buying an audience. From a paid advertising paradigm it is important for Facebook to count your wife as reached with a paid ad because you may have been buying on a CPM basis versus CPC. It is also important that they lower your Total Reach by one so that number is accurate as well. Let’s face it – in your example your intention was to reach 1 person with your ad buy and Facebook delivered on that goal. Maybe a better angle to look at this is that Facebook does under report organic reach so the paid reach can be correct and accountable to advertisers. And since you don’t pay for organic reach they are not accountable for that metric to you as an advertiser. (Also you didn’t mention if the post/ad showed up again in your wife’s account later after placing the buy?) Media buyers also look at the metric of Frequency as well to judge the effectiveness of an ad buy (reaching one person more than once). But the most important metric, which you always evangelize is that of ROI. Most direct marketers could give a rip about the accuracy of their reach and frequency as long as their ROI makes sense.

  • Michelle Mastro

    Very insightful Jon! I am still skeptical about whether it’s intentional or not. Even though you’ve already paid to promote your post, perhaps they want to reinforce that the paid post reaches more people, so that you will then continue to advertise and spend more money. I just don’t know…

    • http://jonloomer.com/blog Jon Loomer

      I just don’t see it. Nothing really makes sense, though, if it is by design. So it has to be a consideration.

  • Andi Jarvis

    Hi Jon, nice idea for the test. Mrs L must really be trusting to give you access to her Facebook account! about the results, I’ve started to mistrust most reach figures from Facebook recently. I’ve noticed on a number of posts that the reach will be lower than the number of people who have engaged with it. For example reach of 65, but the post has 61 likes and 5 shares – the likes + shares is more than the reach figure and it’s stands to reason that at least some people must have seen if it has been shared. From your Facebook quote above and info I’ve found on the site, I can’t see anything that suggests anyone who engages with a post isn’t counted in the reach, so I’ve assumed it’s quirky/not reliable.

    • http://jonloomer.com/blog Jon Loomer

      Hey, Andi! Don’t tell her, I hacked her account!

      Yeah, Reach is absolutely buggy. A few examples…

      1) Randomly, the Reach reported will be way down temporarily as you describe. Refresh the page, and it’s back to normal.

      2) The Reach on cover photos has been wrong for as long as I can remember. What you describe often happens. You’ll get more engagement than you’ll get Reach, which is impossible.

      3) Facebook admitted to under reporting Reach during a six month period that ended in March of 2013.

      You are correct. Not a metric worth worrying about. Many good reasons why!

  • Claire Chesneau

    Interesting experiment and I wonder if it might spark another question? What you are saying is that some people are being reached twice with the same post but this is not recorded. If engagement (not just reach) for a promoted post is up, does this mean that ‘double whammy’ is vitally necessary, but not being appreciated.
    I know this would show up in the stats but I am only a very small operation and I don’t analyse to that depth.

  • http://www.windycityparrot.com WindyCityParrot

    I think reach is terribly misleading number to begin with because of what I call “sub” reach. for example my page is for pet bird enthusiasts – lots of engagement, double digit increases in reach weekly but here’s where the dilution comes in – I’ve divided my fans in 9 “sub” profiles , examples being fans who used to have birds, international fans (I only sell supplies domestically) fans who have no birds but like the pictures and so forth – so at the end of the day I figure my “true” reach – target customers – is only 20 – 30% of overall reach – any way to measure that?

  • http://www.BlitzMetrics.com Dennis Yu

    Unique metrics that are non-aggregable are super tricky to calculate and interpret. For example, you can’t sum daily reach, nor can you sum organic and paid reach (at least not with any plausible business value). Jon, I agree– stay away from reach, unless you evaluate it in conjunction with engagement rate and conversion rate.

  • Cecile

    I am fairly naive about all of this, but unless there was a way to target an ad so that it did not reach those that would see it organically, then the duplicated organic/paid stat is actually irrelevant. What would be really useful is ad targetting that only reached those who would not have seen it anyway as that would actually be an effective way to grow your market. In some ways, removing the duplicated organic reach is a more honest statistic about the effectivenenss of your ad.

  • Bryan Caporicci

    Awesome. Very much based on facts, reality and statistics. Thanks for clarifying and reporting your findings.

  • JamesDiGioia

    It’s possible this will still incentivize advertising, as it makes posts that have been promoted appear to provide a much bigger boost to its reach in relation to its organic reach than it really does.

    Edited in example: if a Post gets 300 organic, and then gets promoted so organic drops to 150 and promoted ggets 450, you could look at the data and go “promotion tripled my reach” when it only really doubled it.

  • sara

    i have also noticed a big difference in the total reach reported on promoted post in post insights report compared with Ads Manager report — the total reach in Ads manager is much higher than in the insights. has anyone else seen this?

    • Alex

      hi sara, I’m facing the same issue and I still can’t find an answer for it. have you had any luck yet?

  • Mark Williams

    I’m a tad confused. I think you’re saying that the EXPORT of FB Insights actually had the correct reporting – 2 organic reach, 1 paid reach, 2 total reach.

    But the info reported in the status update itself was under-reporting reach.

    Am I reading that correctly? It seems like FB is ultimately giving the correct data, but the ‘real-time’ reporting that displays under a status update is a bit wonky.

    • http://jonloomer.com/blog Jon Loomer

      No, the correct report was the OLD export. The new export (and all of the new info reported on the post and web insights) are under reported.

  • http://blog.abstractedge.com Scott Paley

    Here’s a similar kind of thing happening on Facebook that is suspicious. Perhaps an idea for another test? Basically, I boosted a post for a client to fans and friends of fans. Basic stuff. After a while, I saw the fans were seeing the post too many times (my wife, who is a fan, complained that she had seen it half a dozen times.) So I shut off the ad to just the fans. Friends of fans would still see it, since that group was still performing well.

    But, after I did that, my wife still saw the ad! So is this a similar thing? She’s a fan of the page, sure. But she’s ALSO a friend of fans of the page. Is Facebook showing it to her because she’s in both groups? That’s kind of dumb, because it’s likely that many of the page fans have other friends that are also page fans…

    Thoughts?

697 Shares Facebook 279 Twitter 245 Buffer 82 Google+ 49 LinkedIn 35 Pin It Share 7 697 Shares ×