Excessive Crawling

[Frank_Rizzo]

Yahoo is the biggest bandwidth hog on one of my sites.

Each month 20% of all bandwidth (pure pages - no graphics) is used by Yahoo Slurp. Or another figure: 77% of all major SE crawling was performed by Y.

And yet only 5% of visitors come from Y. This doesn't seem right?

[Hester]

Do you have any randomly generated links on your site?

[Staffa]

A while ago I noticed the same thing on one of my sites, crawling 500+ pages over and over again yet NO visitors.
When I searched Y for indexed pages only 3 showed up so I blocked Y from crawling, they're wasting my bandwidth.

[Frank_Rizzo]

Not randomly generated links but there are a few thousand pages with a small amount of content which is updated each day.

It just seems a lot of waste crawling the site for so little visitor traffic.