Forum Moderators: open

Message Too Old, No Replies

Yahoo Spider consuming large amounts of bandwidth

         

servoweb

5:39 pm on Aug 31, 2005 (gmt 0)

10+ Year Member



Hi there,

The Yahoo spider "crawl1.rides.auto.mud.yahoo.com" has been consuming some pretty big bandwidth on one of my sites lately... Almost 7GB for each of the last two months.

To give you an idea as to the signifigance of this, the next highest spider consumed 270MB this month, and Yahoo traditionally has consumed about 50MB per month.

The site has always had pretty good traffic and rankings with Yahoo but recently the bandwidth surge has me a little confused. This site has about 23,000 pages in the Yahoo database.

Anyone else seeing unusually high spider traffic from Yahoo lately, or have ideas on how to tell this spider to take a little easier on my site? =)

Thanks!

wilderness

11:03 pm on Aug 31, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



"have ideas on how to tell this spider to take a little easier on my site"

DNS tools provides this IP.
209.191.64.227

Dan keeps this list relatively current and does NOT provide the above range ad Yahoo.
[iplists.com...]

Don

volatilegx

3:09 pm on Sep 1, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Looks like I should add that one.

Thanks.