The economics of spidering

There has to be a mutual benefit here

[webdevsf]

my website gets spidered - upwards of tens of thousands of hits - literally every day. I don't mind this - its great, because these spiders are supposed to provide some service.

Googlebot for instance, is welcome anytime, as is Inktomi.

But TEOMA, Fast and the like? Geesh, I get maybe 1-2 hits a day.

I think about the bandwidth and think - why bother letting them in? I guess I hope that someday, they'll get their acts together and drive some traffic. :)

Anybody blocking these underperformers?

[jdMorgan]

Hmmm... Teoma feeds AsJeeves, which is popular with a certain demographic, and ATW feeds Lycos.

I don't mind if the number of referrals is greater than the number of spider hits - I'm open for business as long as there's some quid pro quo.

Now, all those others that spider for "employee monitoring," those that mine data for sale to corporations, and the sneaky spam-bots can go eat 403's...

Everybody's site is different, and everyone has to decide for themselves...

Jim