Welcome to WebmasterWorld Guest from 54.225.16.10

Forum Moderators: martinibuster

Message Too Old, No Replies

Excessive Crawling

     
4:45 pm on Jun 14, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 17, 2002
posts:1187
votes: 6


Yahoo is the biggest bandwidth hog on one of my sites.

Each month 20% of all bandwidth (pure pages - no graphics) is used by Yahoo Slurp. Or another figure: 77% of all major SE crawling was performed by Y.

And yet only 5% of visitors come from Y. This doesn't seem right?
11:35 pm on June 20, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 5, 2002
posts: 1318
votes: 0


Do you have any randomly generated links on your site?
8:09 am on June 21, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 24, 2002
posts:894
votes: 0


A while ago I noticed the same thing on one of my sites, crawling 500+ pages over and over again yet NO visitors.
When I searched Y for indexed pages only 3 showed up so I blocked Y from crawling, they're wasting my bandwidth.
10:28 am on June 21, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 17, 2002
posts:1187
votes: 6


Not randomly generated links but there are a few thousand pages with a small amount of content which is updated each day.

It just seems a lot of waste crawling the site for so little visitor traffic.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members