Welcome to WebmasterWorld Guest from

Forum Moderators: martinibuster

Message Too Old, No Replies

Yahoo Excessive Crawling.

crawling 500+ pages over and over again yet NO visitors

12:01 pm on Jul 20, 2010 (gmt 0)

New User

10+ Year Member

joined:July 22, 2006
votes: 0

A while ago I noticed that on one of my sites, crawling 500+ pages over and over again yet NO visitors. i cheked and its yahoo crawler so I blocked Y from crawling, they're wasting my bandwidth. Each month 35% of all bandwidth is used by Yahoo Slurp. Pl. suggest what we do to slow a yahoo crawler.
3:30 am on July 22, 2010 (gmt 0)

New User

5+ Year Member

joined:Apr 4, 2010
votes: 0

I experience that same thing too. please do suggest ways to slow down yahoo crawler.
10:39 pm on July 31, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member zeus is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 28, 2002
votes: 18

yep same here was like ddos attack server went totally slow
3:15 pm on Aug 2, 2010 (gmt 0)

Preferred Member

10+ Year Member

joined:June 5, 2006
votes: 0

Ranking and crawling are two entirely different things. Unfortunately if you want to be found you have to pay for bandwidth. So blocking a search engine is not the answer. The key is to build links so that you can also raise your website's rankings.

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members