So, what's the bestest, mostest well behaved bot on the Planet?

Why - "Ask Jeeves Teoma" of course.

[pendanticist]

Some time ago, in a post about stupid spiders. [webmasterworld.com...] (msg#3) where I said:

As you can see below, Ask Jeeves/Teoma needs a little work too.

################# - - [21/Jan/2003:10:43:18 -0800] "GET /1PRezzies.html HTTP/1.0" 301 259 "-" "Mozilla/2.0 (compatible; Ask Jeeves/Teoma)"
################# - - [21/Jan/2003:10:43:33 -0800] "GET /Presidential.html HTTP/1.0" 200 8727 "-" "Mozilla/2.0 (compatible; Ask Jeeves/Teoma)"

Again, week after week, the same thing - endlessly droning on and on but not quite 'learning' the file has been moved.

Apparently, judging from the overnight activity, they must have done some good things because Ask Jeeves/Teoma has improved substantially.

Check it out. Asks for just what's on my site and no longer even referencing to moved pages (via 301). Nice, non-load bearing demands evenly spaced requests.

egspd428.teoma.com - - [05/Apr/2003:03:11:51 -0800] "GET /Quotations.html HTTP/1.0" 200 2621 "-" "Mozilla/2.0 (compatible; Ask Jeeves/Teoma)"
egspd428.teoma.com - - [05/Apr/2003:03:15:38 -0800] "GET /Archaeology.html HTTP/1.0" 200 13349 "-" "Mozilla/2.0 (compatible; Ask Jeeves/Teoma)"
egspd428.teoma.com - - [05/Apr/2003:03:18:41 -0800] "GET /Webmastery.html HTTP/1.0" 200 8705 "-" "Mozilla/2.0 (compatible; Ask Jeeves/Teoma)"
egspd428.teoma.com - - [05/Apr/2003:03:21:37 -0800] "GET /Vocabulary.html HTTP/1.0" 200 4181 "-" "Mozilla/2.0 (compatible; Ask Jeeves/Teoma)"
egspd428.teoma.com - - [05/Apr/2003:03:24:07 -0800] "GET /Mega_Sites.html HTTP/1.0" 200 13642 "-" "Mozilla/2.0 (compatible; Ask Jeeves/Teoma)"

Hey, they could all traverse my domain in this fashion. :)

Kudos to the Ask Jeeves/Teoma team!

It'sa pleasure seeing you've learned some.....Now, what about the rest of you?

Pendanticist.

[Oaf357]

They could slow their spider down a little. It would bother me at all. It just seems that it is trying to suck everything down a little too quickly.

[pendanticist]

Am I supposed to chuckle, or are you being facetious?

Not sure who you mean by 'They'....

Teoma? or, the others?

I just wish all the spiders/bots were as improved as Teoma has been acting of late.

Pendanticist.

[Oaf357]

They = Teoma

And yes I was being serious.

[NorthernStudio]

Always a polite and well-behaved bot, today Ask Jeeves/Teoma read the robots.txt file of an under construction site where we've banned all bots and dove headfirst into our spider-trap directory, gobbling up bait files until it hit our ASP-generated list of 15,000 bogus URLs.

I was shocked and awed and rechecked my robots.txt.

We always enjoy good refererals from Mr. Jeeves and hope his spider doesn't cause strange listings due to it's momentary lapse.

Wayne

[redzone]

I'm not impressed w/ Teoma's spidering que system a bit.
We host many domains, and when Teoma hit's in full swing, it creates quite a bit of load.

They needlessly re-crawl the same URL multiple times....

Google used to be this way. IMO Google has tuned their spidering system, and remains the most network friendly spider compared to AV/Teoma/Ink...

[pendanticist]

I now concur with the both of you. It would seem that AJK has relapsed.

<sigh> And I has such high hopes too......

While it is true that having visitors come to my work via AJT is nice (as are all the others), I just wish they's shorten the referrer a bit. I mean, nine (9) or ten (10) lines is a bit much.

Pendanticist.