Forum Moderators: open
Accept: text/*
User-Agent: YodaoBot/1.0 (http://www.yodao.com/help/webmaster/spider/; )
Previous version in 2007
60.191.80.*
User-Agent: Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )
From the headers supplied they are only looking for text based results, not images. They never made it past my Robots.txt. The bot does obey Robots.txt it seems from my small sampling (3 fetches of Robots.txt total). Don't have anything else to say about the bot good or bad.
[edited by: incrediBILL at 3:54 am (utc) on April 9, 2008]
[edit reason] delinked URL per posters request [/edit]
We dont service china so ..
Verify that it got a 200-OK response each time. If you gave it a 403-Forbidden, you should expect "hammering" from all but the major 'bots. Some of the third-tier 'bots can be fairly crudely-written, and will keep trying 'til they successfully fetch robots.txt.
Jim
