Controlled Test Results: Facebook Organic Reach is Under Reported

by - 39 Comments

Facebook Organic Reach Under Reported

If you read the pages on this website, you know what to expect. Fact and data based information, (mostly) unaffected by emotion.

Since my primary goal is accuracy, I don’t try to be first when there’s breaking news. And I’ll hold off publishing a post until I have all of the facts.

Luckily I took this path recently because I was on the verge of writing a post that would have reported that Total Reach is over reported. And that would have simply added fuel to the fire for anyone upset about a drop in Reach.

But I took my time. I ran a controlled test. And what I discovered was a pleasant surprise.

Facebook Organic Reach is under reported — specifically when you promote posts.

Sure, that’s not necessarily “good” news. We want Facebook to be accurate. And as we’ll see soon, what appears to be inaccurate data may actually be by design. But it’s always nicer to hear that stats are better than you initially thought.

This post is going to break down the following:

  • What I initially found that sparked concern
  • The possible explanations
  • The controlled test
  • The results

[Tweet “Facebook is under reporting your Organic Reach — at least when you advertise.”]

What I Found

You may know that I’ve been putting together the lessons for my Insights training course. I had completed the definitions of all important terms, an overview of web Insights and was giving a tour of the new post level export.

In the definitions section, I covered Organic Reach, Paid Reach and Total Reach. I talked about how Total Reach will never equal Organic Reach + Paid Reach because you’re bound to reach people both organically and with your ad.

When I got to the post level export section, I revisited this. I used my data as an example to show how this was the case.

There was one, big problem: In the example I was going to use, Total Reach equalled Organic Reach + Paid Reach.

I looked at more posts within that export. Same thing. Then I exported five months of data.

That export included 390 posts. Of those 390 posts, 71 had been promoted. Guess what? In every single case, Total Reach equalled Organic Reach + Paid Reach.

Keep in mind that this was using the new export. The old export wouldn’t be relevant because of Viral Reach (which is now folded into Organic Reach).

The same problems I spotted in the new post level export were found in web Insights. Here is an example…

Facebook Web Insights Reach Problem

This is a post that I first published organically and then promoted only to fans. So in that case, you know there will be overlap between Organic and Paid Reach. But the sum of the two equals Total Reach.

I was immediately alarmed by this because I know it’s not the way things are supposed to work. In fact, I was able to find this entry in the Facebook help center titled, “Why does the sum of paid reach and organic reach differ from the total reach?”

Total reach counts the number of unique people who saw your posts, regardless of where they saw it. If your post reaches a person organically and through an ad, that person will count as one for organic reach, one for paid reach and one for total reach.

That entry is three months old at time of this blog post.

Exploring the Possible Explanations

This was a bit alarming. I could think of only three potential explanations:

  1. Users were only being targeted organically or via an ad, not both
  2. Total Reach was being over reported
  3. Organic Reach was being under reported

I immediately eliminated the first option since I know for a fact that users can both be reached organically and see an ad.

Let’s explain a bit more what #2 and #3 mean…

Over Reporting Total Reach
Let’s say that you create a post that reaches 1,000 people organically. You then promote it, mainly to fans, and reach 1,000 people with that ad. We’ll say that 300 of the people reached with the ad were also reached organically.

If Facebook is over reporting Total Reach, they would say that your ad reached 2,000 people. In reality, it would have reached 1,700 people, or 300 of those people twice. In this case, Facebook would be over reporting Total Reach by 17.6%.

Under Reporting Organic Reach
We’ll stick with the same example as above. This time, though, once your ad reaches those 300 people who already saw it organically, Facebook changes those people to Paid Reach only. So you now reach 700 people organically, 1,000 people paid and 1,700 total.

In this case, Total Reach would have been correct, but Facebook was under reporting Organic Reach by 30%.

The more I thought about it, the more I was convinced: Facebook was over reporting the Total Reach of posts when also promoted.

It just seemed most likely. But I knew I was dealing with a very sensitive situation. People are going absolutely nuts regarding the recent drop in Reach. If I wrote that Facebook was over reporting Total Reach — all while people were upset it was too low — things were going to blow up.

But I also think that Reach is a seriously overvalued metric, and I don’t want to be the center of such a firestorm. That is, unless I could verify my theory to be true.

A controlled test was necessary.

The Controlled Test

I held off on the blog post. I knew that it was something I could test and prove (or at least provide convincing evidence for) one way or the other.

Following is what I did…

First, I created an organic post that was very tightly targeted. The expectation was that only my wife would see it (and I, as the admin, would also be able to see it).

Facebook Post Test

Since post targeting only applies to the News Feed, I immediately hid it from my timeline. You may have seen early tests where I forgot this important step.

I then went to my wife’s computer and checked her Facebook News Feed to see the post.

As you can see in the image above, only two people saw this post. This screen grab was taken four hours after it was published.

It was safe at this point to promote it. So I then went into Power Editor and created a Custom Audience for my wife only. I promoted that post, making sure that she would be the only person seeing the ad.

When this test completes, the following should be true:

  • Organic Reach = 2 (My wife and me)
  • Paid Reach = 1 (My wife only)
  • Total Reach = 2 (My wife and me)

Here are the results I was looking for…

1. Organic Reach (2) + Paid Reach (1) = Total Reach (3). If this happened, Facebook was over reporting Total Reach.

2. Organic Reach (1) + Paid Reach (1) = Total Reach (2). If this happened, Facebook was under reporting Organic Reach.

The Results

Here is a screen grab of that post, 13 hours after it was published and eight hours after it was promoted…

Facebook Post Test

As you can see, after the promotion, Facebook has lowered this post’s Organic Reach from 2 to 1. Total Reach is correct at 2 and Paid Reach is correct at 1.

I went into the new post level export and confirmed the same data. Organic Reach, which had been at 2 within the export prior to promotion, fell to 1.

So, this confirmed that Facebook wasn’t over reporting Total Reach. They were instead under reporting Organic Reach.

Now, however, I needed to figure out why. Or at least the source.

I found a clue within the old post level export.

Old Post Level Export Total Reach

As you can see, the numbers are precisely as they should be in the old export.

What Happened?

Based on these results and the three month old help center entry, it’s quite clear that Facebook has changed the way they handle Organic Reach.

We already knew that Organic Reach changed. This was as a result of Viral Reach going away, and being folded into Organic Reach.

However, Facebook’s decision not to count Organic Reach for a user once they are reached with an ad is completely new. And this change was applied across web Insights as well as the post level export.

When I asked about this within the Developer forum, the response I received — as difficult as it was for me to understand — was that this was by design.

Why is it by design? I really have no idea.

If it actually is by design, I guess you could say that Organic Reach isn’t under reported at all. It is what it’s supposed to be.

But I counter that with the help center entry. No warning was given to this change. And there’s no good explanation I can think of for why this change makes any logical sense.

So until I hear otherwise, I will continue to refer to Organic Reach as under reported when a post has been promoted.

Now What?

At this point, it’s still not clear if this is a bug or intentional. I did receive one response that indicates it’s intentional, but that’s not an open and shut case. I still think there’s a good chance it’s a bug.

Some will try to connect an under reporting of Organic Reach with Facebook trying to get you to advertise. Stop. That’s not what this is.

If Facebook wanted to intentionally deflate Organic Reach to get you to advertise, they would do so on posts that weren’t promoted. This only impacts posts that have already been promoted.

Rest assured that when you promote posts, your Organic Reach is actually higher than you think it is. Your true Organic Reach will be a mystery since you won’t know how many of those you reached with your ad were also reached organically.

In the end, my guess is that a small percentage of marketers look all that closely at Organic Reach anyway. The focus is likely on Total Reach. And that metric remains accurate.

For me, this is just one more example of why Reach isn’t a metric you need to worry about. Facebook can change the way they report it. They can change it again tomorrow and make you think you reached double the people. It just doesn’t matter.

Focus on the metrics that lead to your business goals. Those could include likes, comments, shares, link clicks, sales and other conversions.

Don’t waste your time with Reach.

Your Turn

What do you think about this revelation? Let me know in the comments below!