homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
Forum Library, Charter, Moderators: martinibuster

Yahoo Search Engine and Directory Forum

Excessive Crawling

 4:45 pm on Jun 14, 2010 (gmt 0)

Yahoo is the biggest bandwidth hog on one of my sites.

Each month 20% of all bandwidth (pure pages - no graphics) is used by Yahoo Slurp. Or another figure: 77% of all major SE crawling was performed by Y.

And yet only 5% of visitors come from Y. This doesn't seem right?



 11:35 pm on Jun 20, 2010 (gmt 0)

Do you have any randomly generated links on your site?


 8:09 am on Jun 21, 2010 (gmt 0)

A while ago I noticed the same thing on one of my sites, crawling 500+ pages over and over again yet NO visitors.
When I searched Y for indexed pages only 3 showed up so I blocked Y from crawling, they're wasting my bandwidth.


 10:28 am on Jun 21, 2010 (gmt 0)

Not randomly generated links but there are a few thousand pages with a small amount of content which is updated each day.

It just seems a lot of waste crawling the site for so little visitor traffic.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved