Jeeves/Teoma and Robots.txt

Jeeves/Teoma does not follow the robots exclusion protocol

[renee]

Jeeves/Teoma has been spidering my site including directories excluded through my robots.txt file. Is anybody else seeing this or just me? I sent an inquiry to their support people and never got a response.

[caine]

renee,

sorry for such a late reply, its wroth checking your robots.txt file, with Brett's tool at Search Engine World. Also worth a read of the article as well.

Robots.txt exclusions [searchengineworld.com]

btw, Welcome to webmasterworld

[WebJoe]

Renee

I have observed the same thing and decided to ban the bot from my site entirely, after I havent heard from their support personel for several months.

I am aware that this will worsen the ranking of my site with Jeeves/Teoma, but I believe that since the content of my site is only German and written for a very specific group of people the impact will not have an affect to the number of visitors.

[caine]

WebJoe, Welcome to Webmasterworld.

I think with a site written in German, that certainly as it stands, your decision is not going to have a profound affect with Teoma for a while.

[AnswerGuy]

You can email me any examples of failures. For this problem I need a sample of the URLs which were downloaded not respecting robots.txt.

[e-mail]

[WebJoe]

@caine: Thanx. There are other "regular" bots I banned too for I didn't approve with their behaviour and figured the negative impact (of not being listed) can be ignored...if anyone is interested on which UAs and IPs I have banned I can give the URL.

@AnswerGuy: Sent you a sticky mail with example URL...

EDIT: That was quick....I see you found the trap :)

[jdMorgan]

AnswerGuy,

First things first, unless you want your e-mail flooded... :)

What is the correct User-agent string for use in robots.txt?
And where is this information available on the AJ site?
How about Teoma? - Is the User-agent "Teoma-agent" retired now?

I too have had disallowed pages crawled, but have not yet made the assumption that I've got everything correct on my end. I strongly suggest you consider putting pages on your AJ and Teoma Web sites to deal with the robots exclusion protocol and specify your User-agent names on those pages. If such pages exist, I certainly can't find them. Webmasters cannot assume that the UA string for robots exclusion is the same as the UA string in their site access logs - that rule has been broken by too many search providers to be considered a rule, and demonstrably doesn't work with AJ.

The only way I've been able to keep "Ask Jeeves/Teoma" from getting caught in my bad-bot traps has been to feed that User-agent a blank link-to-home page instead of disallowed content. It adds complication to my sites, it is technically cloaking, it slows down the servers for all requests, and I don't like doing it.

For User-Agent strings in robot.txt, I've tried:
Ask Jeeves
Ask Jeeves/Teoma
ask jeeves
ask jeeves/teoma

I've had no problems with other legitimate robots and my robots.txt files validate with every validation tool I can find. However, they are non-trivial files, due to the different ways that search engines treat disallows in robots.txt and on-page meta-robots tag exclusions, and combinations of the two. So maybe it is on my end, and the UA string is the most suspect.

Please publish this info!

Thanks for your time,
Jim