Forum Moderators: open

Message Too Old, No Replies

Baiduspider - does it obey robots.txt ?

         

1script

6:56 pm on Aug 5, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I'm getting Baiduspider+ (and less frequently Baiduspider) pummeling my sites and I would like to disallow it. I've added this to my robots.txt two days ago:

User-agent: Baiduspider
Disallow: /

They've read it 5 times already, still haven't stopped. The user agent string leads to an error page:

"Baiduspider+(+http://www.baidu.com/search/spider.htm)"

And of course it's in Chinese (Mandarin?) so I can't tell what it says.

So, do you guys know if they just don't obey robots.txt or they just take their sweet time to adjust to a robots.txt change? I'm tempted to just firewall them out but that would mean that they won't read any robots.txt changes and will keep pounding.

So, anyone knows how best to stop Baiduspider ?

P.S. I've added this today, will see what happens:

User-agent: Baiduspider+
Disallow: /

Pfui

6:07 pm on Nov 11, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Your filter will keep these relentless fakers at bay. From ChinaTelecom Group Beijing in recent days:

180.149.130.17 [projecthoneypot.org...]
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9) Gecko Minefield/3.0

180.149.130.145
Baiduspider+(+http://www.baidu.com/search/spider.htm)

180.149.130.25
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9) Gecko Minefield/3.0

robots.txt? NO

PHP link for the first IP shows this UA, too:

Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)

lucy24

9:46 pm on Nov 11, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Heck, let's just make it 180.149.128.0/19 unless you know some perfectly nice people at 180.149.131 ;) Or shall we go all-out and say 180.149?

lenne_dk

12:17 pm on Nov 18, 2011 (gmt 0)

10+ Year Member



I made every baiduspider useragent get a 301 redrect for a page about Liu Xiaobo, the chinese nobel price winner.

That will probably get them a visit by the chinese thoughtpolice and some years of reeducation, thereby getting them off my servers :-]

dstiles

10:37 pm on Nov 18, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



> nobel price winner

I rather like that concept. :)
This 34 message thread spans 2 pages: 34