Page is a not externally linkable
- Search Engines
-- Ask - Teoma
---- Bot is killing my site, is it worth it?


caine - 3:51 pm on Aug 8, 2003 (gmt 0)


ask yourselve one question, do you want the pages to be indexed?

If the answer is no - robots.txt the pages you don't want crawled

If the answer is yea - then you will have to up your bandwidth to deal with it.


Thread source:: http://www.webmasterworld.com/ask_jeeves_teoma/316.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com