Page is a not externally linkable
Brian_M - 10:29 pm on Nov 21, 2005 (gmt 0)
If you changed the robots.txt file to the following syntax (in the order shown below), wouldn't that allow only Googlebot in and keep all the other good bots out? Rogue bots will completely ignore the robots.txt anyway, but at least the site search would still work: User-agent: Googlebot User-agent: *
Hi Brett,
Disallow:
Disallow: /