I want to disable the internet archiver from my site, but don't want to stop google.. If I add this to my robots.txt, will this stop the archiver, but still allow google and all other se's?
User-agent: ia_archiver Disallow: /
bufferzone
9:57 am on Jun 20, 2004 (gmt 0)
It should. I would recommend that you take a look at the robots.txt Brett has written for WebmasterWorld. Go to [webmasterworld.com...]
abates
9:59 pm on Jun 20, 2004 (gmt 0)
I'm surprised at some of the names in there as the programs aren't spiders... e.g. GetRight is a downloading utility and wouldn't check robots.txt in any case.
