Welcome to WebmasterWorld Guest from 54.166.178.177

Forum Moderators: martinibuster

Message Too Old, No Replies

Excessive Crawling

     
4:45 pm on Jun 14, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Yahoo is the biggest bandwidth hog on one of my sites.

Each month 20% of all bandwidth (pure pages - no graphics) is used by Yahoo Slurp. Or another figure: 77% of all major SE crawling was performed by Y.

And yet only 5% of visitors come from Y. This doesn't seem right?
11:35 pm on Jun 20, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Do you have any randomly generated links on your site?
8:09 am on Jun 21, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



A while ago I noticed the same thing on one of my sites, crawling 500+ pages over and over again yet NO visitors.
When I searched Y for indexed pages only 3 showed up so I blocked Y from crawling, they're wasting my bandwidth.
10:28 am on Jun 21, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Not randomly generated links but there are a few thousand pages with a small amount of content which is updated each day.

It just seems a lot of waste crawling the site for so little visitor traffic.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month