Augustwest
October 28, 2003, 02:26 PM
This from an Australian news site, via Claire Wolfe's 'blog, via freemarket.net:
By Sam Varghese
October 28, 2003
The White House website effectively prevents search engines indexing and archiving material on the site related to Iraq.
The directories on a site which can be searched by the bots sent out by search engines can be limited by means of a file called robots.txt, which resides in the root directory of a site...
The rest can be found at:
theage.com.au (http://www.theage.com.au/articles/2003/10/28/1067233141495.html)
Not cool, at all.
By Sam Varghese
October 28, 2003
The White House website effectively prevents search engines indexing and archiving material on the site related to Iraq.
The directories on a site which can be searched by the bots sent out by search engines can be limited by means of a file called robots.txt, which resides in the root directory of a site...
The rest can be found at:
theage.com.au (http://www.theage.com.au/articles/2003/10/28/1067233141495.html)
Not cool, at all.