Since the search engine is so popular, it is also one of those search engines people obsess about and I have been involved in more and more conversations lately where people have mentioned how a slight change in Google Rank can wreak havoc on a company's web traffic. In fact, even Yahoo and Microsoft, two of the company's biggest competitors rely on Google for massive amounts of traffic.
For all the talk of Microsoft being a big, bad company and controlling the world, the simple fact is a simple algorithm sneeze by Google can wipe out legions of companies in a moment. Some have even concluded that Google is more proprietary than Microsoft.
Microsoft has erased many competitors over the years but this has usually been done by giving something away for free or at a lower price or developing something better than the competition.
But in Google's case, when they do change algorithms and subsequently shift a company's search rank, the company blames it on an algorithmic change designed to serve users better. While there is no real reason to doubt this is the case as no evidence to the contrary has emerged, the secrecy of Google's algorithms is legend and as the company gets bigger, more people are beginning to see Google as a company with just too much power.
It is likely in response to this sentiment that Google has decided to open its search algorithms a bit more. According to a company blog post written by Udi Manber, VP Engineering, Search Quality:
This blog post is part of a renewed effort to open up a bit more than we have in the past.
In a move which will surprise many, Manber explains:
We will try to periodically tell you about new things, explain old things, give advice, spread news, and engage in conversations. Let me start with some general pieces of information about our group. More blog posts will follow.
In addition, Manber goes on to introduce himself. For a company which is so secretive just acknowledging the name of an important person other than the CXO team seems unusual.
But he does go on to introduce himself and moreover gives some information on the company's activities:
I should take a moment to introduce myself. My name is Udi Manber, and I am a VP of engineering at Google in charge of Search Quality. I have been at Google for over two years, and I have been working on search technologies for almost 20 years.
The heart of the group is the team that works on core ranking. Ranking is hard, much harder than most people realize. One reason for this is that languages are inherently ambiguous, and documents do not follow any set of rules. There are really no standards for how to convey information, so we need to be able to understand all web pages, written by anyone, for any reason. And that's just half of the problem. We also need to understand the queries people pose, which are on average fewer than three words, and map them to our understanding of all documents. Not to mention that different people have different needs. And we have to do all of that in a few milliseconds.
The most famous part of our ranking algorithm is PageRank, an algorithm developed by Larry Page and Sergey Brin, who founded Google. PageRank is still in use today, but it is now a part of a much larger system. Other parts include language models (the ability to handle phrases, synonyms, diacritics, spelling mistakes, and so on), query models (it's not just the language, it's how people use it today), time models (some queries are best answered with a 30-minutes old page, and some are better answered with a page that stood the test of time), and personalized models (not all people want the same thing).
Another team in our group is responsible for evaluating how well we're doing. This is done in many different ways, but the goal is always the same: improve the user experience. This is not the main goal, it is the only goal. There are automated evaluations every minute (to make sure nothing goes wrong), periodic evaluations of our overall quality, and, most importantly, evaluations of specific algorithmic improvements. When an engineer gets a new idea and develops a new algorithm, we test their ideas thoroughly. We have a team of statisticians who look at all the data and determine the value of the new idea. We meet weekly (sometimes twice a week) to go over those new ideas and approve new launches. In 2007, we launched more than 450 new improvements, about 9 per week on the average. Some of these improvements are simple and obvious -- for example, we fixed the way Hebrew acronym queries are handled (in Hebrew an acronym is denoted by a (") next to the last character, so IBM will be IB"M), and some are very complicated -- for example, we made significant changes to the PageRank algorithm in January. Most of the time we look for improvements in relevancy, but we also work on projects where the sole purpose is to simplify the algorithms. Simple is good.
There are also some thoughts on search complexity and it seems humans searching for information seem to roughly approximate Moore's Law meaning our queries get more difficult over time.
What this means for the world is we will finally get some insight as to how Google works. Expect those involved in SEO to spend countless hours studying every word that is uttered about how Google searches and expect Yahoo and Microsoft to secretly worship every tidbit they learn.
What Google will be doing - and they know it - is give many companies added ammunition to "game" the system. They also know as they put more information out about how they work, they will be making their own jobs more difficult as they will have to combat search engine spammers who will become better educated.
This brings us to why Google is opening up at all. They must really feel the pressure of their reputation being damaged by their commanding position in the market and also think they are so good at this search thing that even sharing some information with the world won't hurt their results much.
One thing is certain - the information Google will supply could be some of the most important information many companies will ever be presented with and those who are paying attention will be in a much better competitive position than the companies who don't.