A while ago I noticed that on one of my sites, crawling 500+ pages over and over again yet NO visitors. i cheked and its yahoo crawler so I blocked Y from crawling, they're wasting my bandwidth. Each month 35% of all bandwidth is used by Yahoo Slurp. Pl. suggest what we do to slow a yahoo crawler.
Msg#: 4173261 posted 3:15 pm on Aug 2, 2010 (gmt 0)
Ranking and crawling are two entirely different things. Unfortunately if you want to be found you have to pay for bandwidth. So blocking a search engine is not the answer. The key is to build links so that you can also raise your website's rankings.