If you operate a public web site, there is little doubt that you'd like an occasional visit from search engine minions, known as robots. Robot are little agents that search engines dispatch to your site to scan your page contents and sent them back to the mother ship for cataloguing and finally including in search engine results pages (SERP).
� If you have ever scanned your web logs, you would undoubtedly noticed these agents. They come with different names like googlebot, msnbot, and yahoo slurp.
Continue Reading...