abbeyvet - 2:40 pm on Jul 7, 2005 (gmt 0)

Without completely stopping it :(

In a way this is a follow on from this thread which made me look at my ASK traffic more carefully:
[webmasterworld.com...]

This from my stats for this month:

Jeeves1238722.32 GB
Inktomi Slurp212941.75 MB
Googlebot (Google) 80012.43 MB

Last month it indexed 159129 pages and used 2.04 GB, so the pace seems to be accelerating if anything. It is there all the time.

Google has it about right - I have somewhere between 900 and 1000 pages in the site proper.

What Ask is so busily indexing is basically Amazon. I have an Amazon shop that is really only intended to have about 5-6 sections of books relevant to my topic, organised in a way very different from the way Amazon does it, but Ask is following everything - every obscure author you (n)ever heard of.

It does send traffic in reasonable numbers but it's mostly irrelevant traffic looking for obscure authors for whom I now rank very highly in Ask - without ever meaning to! They do not buy anything much, maybe a couple of books a month - I have checked the pages they land on and mostly the books are not available - and the Adsense income from those pages is not brilliant.

I am caught between disallowing Ask from that directory altogether and leaving well enough alone - in a way I hate to lose ranking, even relatively useless ones!

I could plaster Adsense all over it in a very aggressive way, but I really do not want to do that for the pages that have value to my users.

Has anyone else experienced this? Does it eventually slow down? What would you do?