become spider

be aware of become.com

[tomasz]

Hello,
I have a dedicated server with couple of database driven web sites, today became.com came thru and took my server down. It took me a while to figure it out and I ended up excluding become.com in IIS.
It was ignoring my robots.txt

Are there any more spiders you guys recommend to exclude?

[fiestagirl]

A resource to help make your own decisions about what to ban.
[webmasterworld.com...]

[pendanticist]

I can certainly say your experience seems out of character for the become bot that I have come to know. This bot request robots.txt and a few files over a ten-minute period and has never sucked down any of my bandwidth.

[webmasterworld.com...]

Are you on Apache where you can view the access_log files?

How many files did it take, per second or minute?

Did it repeat requests for certain files?

Are these files especially large?

Can you provide us with a few of the UA Strings that adequately represent your statement?

I am thinking there has to be something more to this.

[bcolflesh]

Become.com's bot has been relatively behaved on my sites - I'm suspending a ban until their index is public.

[tomasz]

I have several thousands of db generated files pages. It requested around 60 pages a second and it locked my SQL server which run at 100% of CPU.
I need to take one more look at one of mine stored procedures which is taking a litle to long to pull the data,