Forum Moderators: open

Message Too Old, No Replies

baiduspider indexing like crazy

How to limit the crawling speed

         

mainspot

2:40 pm on Sep 15, 2008 (gmt 0)

10+ Year Member



I have one site with some of the contents in chinese. And baiduspider is crawling this site like crazy, so far in 5 days it's already 7gigs bandwidth.

Is there anyway to define crawling speed for baiduspider in robot.txt?

Thanks in advanced.

incrediBILL

5:23 am on Sep 19, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Here's the translation of the pages about Baidu spider FAQ [translate.google.com] and what they support in robots.txt [translate.google.com]

Not according to their site, it's supposed to self-regulate.

You'll find an email address on the FAQ page where it claims if you write to them and send log files they'll try to help with the problem.

mainspot

6:02 am on Sep 19, 2008 (gmt 0)

10+ Year Member



Thanks for the info. But still not found any info about crawling speed.
I guess we have to email them directly.