~by Danny Sullivan
"At the end of last month, controversy erupted over the US White House preventing portions of its web site from being indexed by search engines. Was the White House doing this as a means to rewrite history unnoticed, or was it an innocent mistake?
At issue is the White House's robots.txt file, a long-standing mechanism used to tell search engines that certain material should not be indexed.
Keith Spurgeon has an excellent analysis of what happened. He outlines how sometime between April and late October of this year, the White House changed its robots.txt file to block access to many pages with the word "Iraq" in their URLs."
Get the Whole Scoop Here
