Page is a not externally linkable
- Local
-- Foo
---- Attack of the Robots, Spiders, Crawlers.etc


waynet - 8:12 pm on Nov 25, 2005 (gmt 0)


Anyway, I was thinking if it would be possible to set up a mirror site where posting wasn't allowed and redirecting all known and detected robots over there, would that work? Let the bots chew on a slow mirrored site and keep the real site relatively bot free. When a user logs into the mirror site, have them redirected over to the real site. At least Google will have something to crawl.

I was also thinking one of those Google search appliances would be a good replacement for the site search, but looking at it and the number of pages here it would cost a fortune. Brett, know anybody over there that would give you a good discount on that? Maybe as part of some beta test or something?

[edited by: Brett_Tabke at 9:51 pm (utc) on Nov. 27, 2005]


Thread source:: http://www.webmasterworld.com/foo/9618.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com