jdMorgan - 7:59 pm on Aug 22, 2009 (gmt 0)

> to address your concern

Not really my concern, since the answer wouldn't change regardless of the site... :) Someone else reading this thread in the future may have a site that differs remarkably from yours (or mine).

I tried to answer in-depth because it's important to be able to ake a step back and evaluate your site and content and links objectively. Sadly, this is the ability most lacking among Webmasters taken as a group, and the reason we get a lot of complaints directed at search engines in posts like, "Google banned my (spammy copied-misspelled-stolen-content stuffed-with-keywords and plastered-with-PPC-advertising) site for no reason! - No reasons at all!" A degree of detachment and pragmatism is needed.

> a quality list of good search engine agents
Well, it would start with googlebot, Slurp, msnbot, Teoma, etc.

If you want to get more specific, then a bit of time spent mining your server access logs for the past year will likely give you a fairly good list of these 'bots and their many incarnations.

That's really the problem: It's possible to use just the single, general names as in my short list there, or to find dozens of variations of many of those 'bots; MS and Yahoo especially favor the 'bot du jour approach, apparently exercise little control over naming conventions used by their various dev groups, and therefore tend to unleash multiple, inconsistently-named 'bots at a fairly high rate. (In fact, I had a laugh just yesterday when I got some logged requests from a 'kitchen sink' 'bot from Yahoo, incorporating just about every 'bot name they (or their predecessor Inktomi) had ever used.)

Jim