Welcome to WebmasterWorld Guest from

Forum Moderators: martinibuster

Message Too Old, No Replies

Yahoo Excessive Crawling.

crawling 500+ pages over and over again yet NO visitors



12:01 pm on Jul 20, 2010 (gmt 0)

5+ Year Member

A while ago I noticed that on one of my sites, crawling 500+ pages over and over again yet NO visitors. i cheked and its yahoo crawler so I blocked Y from crawling, they're wasting my bandwidth. Each month 35% of all bandwidth is used by Yahoo Slurp. Pl. suggest what we do to slow a yahoo crawler.


3:30 am on Jul 22, 2010 (gmt 0)

5+ Year Member

I experience that same thing too. please do suggest ways to slow down yahoo crawler.


10:39 pm on Jul 31, 2010 (gmt 0)

WebmasterWorld Senior Member zeus is a WebmasterWorld Top Contributor of All Time 10+ Year Member

yep same here was like ddos attack server went totally slow


3:15 pm on Aug 2, 2010 (gmt 0)

5+ Year Member

Ranking and crawling are two entirely different things. Unfortunately if you want to be found you have to pay for bandwidth. So blocking a search engine is not the answer. The key is to build links so that you can also raise your website's rankings.

Featured Threads

Hot Threads This Week

Hot Threads This Month