90% of my traffic is spiders and bots.

What to do -

[webpro00801]

So I am finally realizing that since the latest Google fiasco that 90% of my traffic is now spiders and bots. A ton is from Yahoo, they have 400k pages of mine indexed, but treat us like crap, etc. The MSN bot is another offender. Ask.com, etc. What should I do? This is to the point of costing me serious bandwith every month. Should I ban all for awhile as a test? Or just leave the Google bot to run? I have never used a robots.txt before. Thanks -

[Quadrille]

Run xenu, and check all your internal navigation; set up a site map, and think about a Google site map.

Search and replace /index.html with / (as appropriate)

301 from non-www to www.domain.com

It may be that spiders are having problems zooming through your site, and just keep trying.

Have you done any strange linking 'tricks' that might be causing trouble?

To get good listings, you need to be spider-friendly; and to get visitor numbers, you need those listings.

So think carefully before using robots.txt - use it to eliminate duplicate URLs and other unnecessary pages so often produced by databases - but be sure that 'content' pages don't get eliminated too.

I had a directory once with 37 categories, that produced a Google count of 700 and poor listings. Once I configured my robots.txt to ignore the bumph, I got a leaner, cleaner search result, and more visitors - because they stopped seeing nonsense pages in the serps. I suspect that poorly-structured dynamic sites do create unnecessary work for spiders, which eventually bites you in the butt.

[webpro00801]

I really don't think I am having any of the issues you mentioned. I can see the spiders, looking at my log live, and they just come through every page, and I have a ton of pages.

I have also never done anything but pretty simple and straight forward HTML - no tricks at all. We have a nice directory structure, have been on line for over 11 years, and have already enjoyed great love by most search engines. It just seems that these things come through without mercy day after day...

[JAB Creations]

What exactly is high bandwidth for you? You can get a terabyte of bandwidth for as cheap as $8 a month. What is your current bandwidth allowance and how much are the spiders using combined on average?

- John