Page is a not externally linkable
shilmy - 3:41 pm on Jun 23, 2005 (gmt 0)
The problem with BecomeBot in my site is they eat about 100MB bandwidth in a snap. The crawl heavy and deep. You know, my sites is using a script to showcase a products from amazon. If it is not stopped, can you imagine a bot crawling the whole amazon store at once? I'm a newbie in robots.txt, is there a way to limit so the bot do not crawl the whole amazon store at once?
May be you are right, we need more SE so we don't depend only on The Big G.