A few days ago I had a discussion with our managing editor for our company's Web site about how crawlers discover and index pages. He was convinced that search engines can somehow find hidden pages on a Web site even if there are no links to those pages. I, on the other hand, wouldn't be persuaded. How could search engines crawl a page if they don't know the page's name and location, i.e. its path? Turns out we were both wrong – and right, depending on how you look at it.
In order for search engines to crawl a Web page, they must first be directed to it. The process of page discovery is generally a hyperlink on another page that the crawler can follow. I'm not sure if search engines also follow plain text URLs, but it is a possibility. A site that wants to publicize a new page would normally have links to the new page from other pages, or the page will be in a directory index which lists all files in a directory when accessed (Web sites normally disable this option though for security reasons). In the absence of a link to a Web page's URL, crawlers would have no idea about the existence of that page (referred to as a hidden or orphaned page). I suppose they could engage in name-guessing, but that's an expensive proposition I suspect most search engines shun.
Then a few days ago I ran into an anomaly that disproved my belief about hidden pages and crawler discovery. I was working on a fairly popular page (Browser Simulator/Emulator) on my personal site. Due to the nature of the page, it has the potential of becoming a tool in the hands of abusers, so it is monitored for abusive activity patterns. I began to notice that the page was being accessed excessively by Googlebot with specific parameters as if a human was commandeering the page. Respecting the privacy of users however, I only monitor general patterns on that page, so I didn't have detailed information about Googlebot's activity.
With my curiosity piqued, I constructed a similar but hidden page in the same folder and switched on full monitoring. Then I began hitting the page, entering various data in the form fields. Sure enough, Googlebot began accessing that page with the same data as I had specified. How could Googlebot discover the hidden page so fast (if at all) and specify the same data as I was? A glance near the top of my Internet Explorer browser found the culprit. It was the Google toolbar, the seemingly innocuous toolbar that most people have installed on their browsers and are oblivious to its operation.
I am certain the Google toolbar comes with a privacy disclosure detailing how and what it gleans from the user's activity. I never bothered to read this and chances are most people ignore it as well. I am also not sure what Google does with the data. I suppose they do use it for ranking purposes, but I am now certain that it crawls the pages surfed on by users. I am, however, still unsure whether the crawled pages ever make it to the Google's index to be displayed as search results. I am also unsure if what the browser displays to the users is sent to Google along with the URLs (this could have potentially disastrous privacy repercussions).
There you have it. If you place hidden pages on your Web folders, don't be too confident about their secrecy, even if those pages are only accessed internally by you and a few trusted people. Anyone with a Google toolbar (or any other toolbar such as Alexa or A9) would be unwittingly sending the URLs of those hidden pages to Googlebot (or other robots/spiders), and potentially exposing the location of those pages to the world.
Technorati
Del.icio.us
Slashdot
DiggIt's a classic tit-for-tat. In a move that is nothing short of an act of reprisal, GM announced today that it will indefinitely suspend advertising in Los Angeles Times. GM cited "factual errors and misrepresentation" for its decision.
The move comes two days after an LA Times columnist criticized the auto-maker for pushing its gas-guzzling SUVs rather than investing in hybrid technologies. Perhaps the columnist went too far by calling for the impeachment of two GM executives, but GM's action is just plain childish.
It's a free press. People express a lot of opinions about a lot of subjects. Some may get a bit scathing, others a bit risqué perhaps, but advertisers should take criticism in stride and understand that the separation between the editorial and advertising departments is what keeps a publication vibrant. An uncontaminated editorial process can only help the credibility of its publication which leads to more interested readers which in turn translates to more eyeballs seeing the ads.
If GM hadn't jumped the gun, they could have used their advertising space to subtly challenge the columnist's position. Instead they decided on this immature action, possibly handing the columnist even more credibility.
I wasn't sure if this was an April fool's joke when I logged in to my Gmail account and saw my total storage at 1324 MB. It had been 1000 MB (which by the way, is not exactly 1GB) since its inception. But then something stranger kept happening. The next couple of times that I logged in during the day, I noticed that the storage limit was increasing. It's like it was a moving target. Right now it's sitting at 1627 MB and I surmise I'll see a higher number late tonight.
I'm not sure what to attribute to this strange behavior. Google already had its April fool's fun with Google Gulp. But I just saw this story and it appears that Gmail is indeed marching towards the 2 GB storage limit. It was only a few days ago that Yahoo! announced plans to augment their email storage to 1 GB.
So it seems that the storage wars are once again heating up between the email titans. Personally I can't see how anyone would exceed the 1 GB capacity, but I suppose there is some value in bragging rights here. Having the highest capacity in the industry is great PR after all.
Meanwhile Hotmail has been quiet during all this. I wonder if they're planning a 1 TB storage offer after the smoke has cleared. That should settle the race, or does it?
Admit it. You have broken the law at least once in your life. Maybe you have gone above the speed limit a time or two, or made a copy of your cousin's music CD. The law doesn't have the patience nor the time to worry about these occasional lapses. Society must rely on people's good characters in some instances, lest be crippled under a mountain of enforcement actions. But when an illegal activity reaches critical mass, it is the duty of the law to step in and curb the abuse.
Such is the case heard before the supreme court regarding the file sharing and file swapping products known as P2P or peer to peer software. No one in their right mind is going to admit that the products they create are intended to encourage people to break the law, but actions speak louder than words.
Napster, the original digital piracy tool wasn't heralded as a bootlegging product either, but once its true usage in swapping pirated music became obvious, the law had no choice but to shut it down.
Proponents of P2P products such as Kazaa, Grokster, and eMule claim that their creators have the right to innovate and can't be held liable for people's misuse of their creations. There is some merit to that argument, but when these wares become tools in the hands of millions to break the law and bilk others of their rightful living, the law has no choice but to intercede. If the purveyors of the P2P products are looking the other way and getting rich on the backs of others' hard work, even if it is inadvertent, they must bear some responsibility.
They would holler just the same if someone found a dubious method to unwittingly hijack their creations and dent their pocketbooks.
Checking my mailbox over the weekend, I was greeted with my tax return documents. Only thing is that those were supposed to be delivered to the darlings at the IRS. What steams me is that I made sure to mail my return from another government entity, the post office. They weighed the enveloped, printed the postage stamp, and sent it on its way. I even paid for delivery confirmation. Thankfully I had a return address on the envelope (I normally don't bother), and the envelope returned to me with an insufficient postage sticker. Nice!
Wonder what would have happened if I was at the deadline date (I wasn't) and I owed money (I didn't). I guess on top of the wasted time and postage money, I would have been slapped with late IRS penalties too.
Moral of the story? I'm not sure. I mean how much more certain can you be than taking your tax return envelope to the post office to send to the IRS? Maybe get a second opinion on the postage from a different post office?
For some, e-file may be an option. I tried it a couple of years ago and it worked out great, but last year I had some hassles with it and ended up mailing in the return. It's back to e-file for me this year. And, what are my chances of recovering my postage fee from the USPS? Yeah, right.
Seeing Bernie Ebbers' picture with a gloomy face after the guilty verdict was announced evoked some feelings of sympathy in me. Perhaps I am too sentimental seeing people in distress. I truly felt bad for him. He just doesn't look like a criminal. But the facts spoke for themselves.
Ebbers' star rose at a dizzying speed during the heyday of the telecom era. From a small-time phone company operator, he became the mighty chief of MCI WorldCom and would have nabbed the number one spot had his merger initiatives with Sprint had come to fruition.
We have been a long-time MCI customer and I suppose we should have foreseen such a bleak day for its chief. From our vantage point their accounting department was nothing short of a complete disarray. I can't tell you how many uncounted times we contacted their accounting people to fix a billing problem, only to call again and again after receiving subsequent erroneous invoices. And the bills themselves were impossible to decipher. It's almost as if they were deliberately designed that way to confuse and thus fleece the customers.
Ebbers' anguished face might engender feelings of pity for many, but in the final analysis, the man was in charge of the company and deeply involved in its operation. He could not shirk responsibility of the massive fraud that had overtaken the company, and the jury was observant enough to conclude his guilt.
Now that blogging has taken the Web by storm, in its true, come-from-behind style Microsoft has joined the fray with its MSN Spaces offering and it is rapidly adding subscribers to this free blogging service which is currently in beta.
Spurred by an enticing link in my Hotmail account, I recently paid a visit to MSN Spaces. All I can say is that this service has a long way to go before it can catch up with the other blogosphere titans such as Blogger and Movable Type (and by extension the TypePad service).
MSN Spaces is designed nicely, but it is cluttered and cumbersome to navigate. There are no options to save the blogs to an external site, and the space provided (10 MB) is inadequate for an active blogger who uses lots of images. I use Movable Type at work and Blogger at home, and both of these products provide a superior blogging experience. They have simpler interfaces, provide better functionality, and have more useful options.
I am certain that with time Spaces will improve. Whether or not it can catch up with the old reliables, we'll have to wait and see. For now, this is the only blog I'm going to place in my Spaces account. I must try everything at least once.
Back in early 2002 when Carly Fiorina was pushing a merger with the rival Compaq, many suspected that she may not make it through unscathed. Instead, she persevered and received the slight nod to merge with Compaq, much to the displeasure of Walter Hewlett. Today, with the announcement of her departure as a result of board disenfranchisement, Walter must be in a high spirits, feeling a vindication of sorts.
And while investors seem to be cheering alongside Walter by sending the HP shares up on the news, one can not help wonder whether things just got better or worse at the company. It's not just Fiorina that was the trouble for HP. For the true culprit, HP should look East to
Unfortunately HP's computer business is not a cohesive unit as Dell's is. HP carries a number of other entities such a Compaq and Digital Equipment (from a previous Compaq take-over) which have come with some heavy baggages of their own as they were sewn into the HP culture. The Dell culture, on the other hand, is one of growth, focus, and steady vision. And with its present momentum, it's a Herculean task to match, let alone beat.
So as investors rejoice Fiorina's departure news today, they'd better be sure to keep their Aspirin bottles nearby. It's gonna be a long hang-over.
Gmail's claim to fame at its point of inception was the large storage (one Gigabyte) it offered its users. It prompted other major email providers to announce their higher storage services and quit bugging their users to pay money for more room. Remember the old 2-Meg Hotmail limit?
I had blogged about all this previously, and as compelling as the move to Gmail was, once Hotmail came through with their 250-Meg storage, I decided to stay with it, although I utilize my Gmail account occasionally.
But now I am finding that competition between providers has created a real boon for the end-user and so far Gmail is ahead of the pact. One of the greatest features offered by Gmail is the ability to forward incoming emails to another account. This is truly a test of confidence on the part of Gmail, as users could simply forward their emails to Hotmail or Yahoo! if they so choose. The other feature is pop-3 email download. This allows those of us with email clients, such as Outlook, to download incoming emails to our local boxes. It would also allow Hotmail and Yahoo users to pull in the Gmail messages as well, if they choose to do so.
The forwarding and pop-3 features would have never been offered by the other guys as they'd fear users skipping their email sites and read their messages elsewhere. Gmail should be applauded for offering these services so fearlessly, allowing their users to interact with their services on their own terms.
Two other Gmail features that are also useful are Gmail Notifier (the tray icon that notifies user of new messages) and the RSS/XML feed of newly arrived messages. Many people utilize RSS readers today, so the RSS/XML feed comes in real handy.
Gmail has sure made it tough to go with any other provider, not necessarily because of technical advantages, but more due to a better understanding of their user base.
As much as Microsoft would like to tout itself as an innovator, its history is largely a legacy of imitation. That is not in any way a negative trait. Most products in use today are imitations of the originals with incremental improvements. and the search technology seems to have caught Microsoft's fancy as of late.
Microsoft's latest push in the search field is clearly an imitation of what Google has been able to achieve. But in this case, the product is just too similar to Google's offering. Take the new and improved MSN search. The pages are clean, the ads are text-based, and the results are presented in almost the same format as Google does. Then there is the IE toolbar plug-in. It's a copy of google's toolbar with the highlight tool, form auto-fill, and Blog posting. And the MSN Deskbar, again the same striking similarity to Google's Deskbar down to the logo watermark in the search field.
I downloaded the products (the toolbar and deskbar are lumped in one install) and used them for a while. They work fine, but I just can't see a compelling reason to leave Google at this time. Perhaps, given some time, Microsoft will be able to improve these products drastically and surpass Google, but for now I just hope the hard competition continues to deliver better products to the consumer. The race is just heating up.
Microsoft's search-related products can be downloaded here.
Did you vote during the last presidential election? I did, and I realized that registering to vote gives you more than a power to voice your choice. It adds you to a list that can be used by a variety of government agencies for a number of purposes. One of those is for jury duty selection, and I was called for it for the first time ever.
Okay, I realize that jury duty is supposed to be an honor and a civic duty and what not. But really, how many of us really look forward to being called for it? I don't. Yet I wonder how after so many years, my name finally made it in the pool. It had to be the registration to vote. The truth is that until the last presidential election, I had never voted, and therefore never bothered to register. Once I did however, my name went into the list, and jury duty found me.
This is not unlike companies selling their customer lists to others for the purpose of marketing, but at least in this case the government is not secretive about it. I should have done my homework to understand the process better. I'm not sure if knowing this information would have stopped me from voting, but since my chosen candidate didn't make it, it makes this jury duty call even harder to bear.
There was a time when blogs (known as Weblogs in those ancient times) were considered a personal diary, relegated to chatty teenagers. Then as more people found out about them, they began typing away. I myself keep a blog on the my personal site and find it therapeutic and liberating to freely express my views on a variety of topics.
As blogging gathered critical mass, companies started to show interest. Now many news and information sites have a heavy blog presence as well, manned by their editors and other contributors. Our own company saw a slow start to blogging, but now we have a number of power bloggers who contribute to the wealth of our online existence.
Many people now get their daily news from blogs, but one of the areas that blogs have begun to encroach on Web pages is what's known as trackback. This is a means by which a blog site can be notified when another blogger makes a mention of them in the form of a url link. Generally, the notification involves a summary of data to be passed back to the mentioned blog site and most sites choose to display this information, in effect providing a link back to the blogger's post. It's a courtesy move in response to the mention. By the way, trackback was devised and promoted by the renowned blog software maker, Movable Type.
Now many sites have decided to incorporate trackback in their own pages, and we now provide this service to our readers. When you read our pages on tmcnet.com, feel free to reference them in your blogs and if your blogging software is equipped with auto-discovery, we will receive your message (known as Ping). To view the trackbacks or get the Ping url (to trackback manually) just click on 
on the our pages.
Remember .NET? The cutting edge, unifying, all encompassing platform Microsoft launched a few years ago?
It was perhaps one of the most confusing and blurry marketing ideas ever to come out of Redmond. And no matter how hard Microsoft tried to clarify this grand vision, the befuddlement just got worse. Finally Microsoft threw in the towel and backtracked on calling everything under the sun, "everything under the sun".NET.
Along with the .NET hoopla, the Passport service was being showcased as the de-facto authentication scheme. It would allow third party Web sites to login users through Passport and leave the authentication and access job to Microsoft. The idea wasn't new by any means. Single sign-on has been the aspiration of the authentication industry for decades, but alas, Passport wasn't meant to be the proverbial silver bullet.
eBay was one of the more prominent companies who initially signed up for Passport services in addition to their own authentication method. But I always wondered, why would a vendor entrust its valuable customer information to Microsoft. I guess eBay must have been having the same doubts. As of January 24th, they are dropping the Passport service from their login screen. If not the death knell, this is a giant blow to an already waning product.
Meanwhile .NET has mostly been forgotten by the public, save one area. It is still much alive in the development community and with the next version of .NET development Framework 2.0, and Visual Studio 2005 on the horizon, it has the development community, myself included, abuzz. The .NET vision has never been clearer.
Desktop Search has become all the rage these days, and for a good reason. After people discovered that they can seemingly search the Web faster and more efficiently than the contents of their own PC's, the search companies decided to use some of their search technologies to help users find stuff in their own backyard, i.e. files stored on their own hard drives.
For years the only choice for Windows users to search their local files has been the standard Microsoft Search program which comes bundled with Windows. It does the job,
But it's minimalist, clunky, and painfully slow. Microsoft tried to address the speed issue by marrying the search program with the Index Server (a service that indexes the local files), but while this has boosted the performance of search it is still woefully lacking in utility, features, and robustness.
There has been a need for a better search software for quite a while. Enter the big three search outfits, Yahoo!, Google, and MSN with their own search programs, known as Desktop Search.
Not too long ago I installed Google's Desktop Search and I was initially happy with its performance and flexibility. But Google's program is limited in the file types it could index and the interface is, well, Googlish, as in plain and simple and no frills. Perhaps Google's strength in its Web search simplicity was just too bland for me when it came to searching locally. I soon lost interest and forgot about desktop search, until today when I downloaded the Yahoo's incarnation of the desktop search program, and my reaction so far is, wow!
Available in beta version, this program does an impressive job of indexing the local files as well as Outlook messages. It has the intelligence to recognize over 200 different file formats and it can index everything in its path. The interface looks fresh, functional, and fast. In little time I had it configured to index my entire hard drive and after it was done with its initial scan, I tested it with a number of keywords. Who knew searching could be so much fun? It's blazing fast and the results are presented in a nice format. I am sold. Yahoo's Desktop Search has now replaced Google's program in my program tray and I feel absolutely empowered having such a tool at my beck and call.
You can download Yahoo! Desktop Search for Windows from here.
Read the press release.
It was only a matter of time, but Microsoft's concerted mobilization against hackers has started to pay dividends. Today, that dividend manifested itself in the form of an anti-spyware program, aptly named Microsoft AntiSpyware. The program (currently in its first Beta) is available for free from Microsoft's Web site.
Windows Users have a number of choices to battle spyware today. They come in free and paid flavors from a number of companies and I bet these companies are not so sanguine about this new release from Microsoft.
Installing and configuring the program was a breeze and then I had it run a quick scan. It identified one low-risk program (with a nice summary) and gave me a few options to deal with it. For the time being I configured it to run a deep scan every mid-night and check for new updates. Exiting the program, it silently went into an icon state, nestling itself with the rest of the resident programs in the Tray area.
For me, the Microsoft AntiSpyware program is too new to have a definitive opinion about, but my first impression was a positive one. One more thing, the program also contains a tool to wipe clean many hidden areas that record user's activity and whereabouts, including Internet Explorer history, visited links, and recently opened documents by other program. It's a great tool for a paranoid like me.
You can download Microsoft AntiSpyware from here.
