Forum Moderators: goodroi

Message Too Old, No Replies

Baiduspider and multiple user-agent text file.

         

Vimes

7:48 am on Aug 7, 2007 (gmt 0)

10+ Year Member



Hi,

Can anyone confirm if Baiduspider handles a multiple user agent robots text file ok.

Or should I just deliver a file for them only?

Vimes.

new_seo

8:09 am on Aug 7, 2007 (gmt 0)

10+ Year Member



See,I don't have any experience of setting up relation between robots.txt and baiduspier,but in a post at webmasterworld I heard that baiduspiders doesn't follow robots.txt properly.
If I was in your position I would deliver a file for them only .

Vimes

9:11 am on Aug 7, 2007 (gmt 0)

10+ Year Member



Hi,

i've read that post if its the one dated a long time ago, Baidu has i hope progressed since those days, hey but i could be completely wrong :(

i'm seeing different patterns from the chinese bot and the japanese bot
i was wondering if any one had any experience with dealing with these bots recently.

Vimes.

goodroi

12:04 pm on Aug 7, 2007 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



I haven't had a recent reason to deal with them. I would try a regular robots.txt and then use htaccess as the fall back solution.