Over the years, TMC has tried many third party measurement sites and without a doubt, Google Analytics under reports traffic numbers as compared to other independent measurement services by an order of magnitude in many cases. Why is anyone’s guess but this article confirms that Google’s in-house numbers are lower than virtually all other measurement methods. This includes the well-respected Comscore.
In addition as John Battelle points out, there should be natural suspicion of measurement practices of any company which also runs an ad network. Here is an excerpt of his post which discusses Google Ad Planner in detail:
In short: If you were a media planner using Google Ad Planner, and you were looking for larger sites, you would be led to sites that are running Google AdSense, on average, over sites that do not. Net net: This data indicates that Google Ad Planner pushes ad dollars to Google sites over non-Google sites. This makes sense – Google has data on Google users, after all. So that data might naturally bias toward Google-related sites.
But as I said in my coverage: “Such a tool must be neutral and not bias advertisers toward buying on Google properties or those that have Google ads, which of course is going to be a perceived bias in any case. Such is the price of being Very Big.”
So far, not so good on this measure. As Gian and Comscore have long pointed out to me, it takes more than raw data to make for good measurement. Ideally, you weight your data with a lot more knowledge of its context – what kind of machine is creating it (work or home? Man or woman? etc.). While Google once blended Comscore demographic data into its ad network, Comscore confirmed to me that this is no longer the case. And while it is subject to endless criticism, Comscore does have a lot more practice at this game than does Google. At least for now.
This data once again raises the question, long asked, of how Google is measuring in the first place. Most believe Google must be leaning heavily on its Toolbar data (see TC for more here and Danny here), and this data does nothing to counter that argument. The strong bias toward Google network sites is suspicious – one can imagine that folks who might install the Google toolbar are clearly already biased toward visiting Google-related sites, for example.
But Google will not acknowledge any use of the Toolbar. Instead it said in its announcement: “Google Ad Planner combines information from a variety of sources, such as aggregated Google search data, opt-in anonymous Google Analytics data, opt-in external consumer panel data, and other third-party market research.”
In short, it is difficult to trust any measurement numbers unless you know how the numbers are being measured. Until Google explains how it calculates traffic in more detail, there should be natural suspicion. This is especially the case as Google seems to be the only service reporting such low traffic numbers. Perhaps they do know something that everyone else is missing but if they do they should tell us what it is.