Forum Moderators: open
The behavior is the same on both sites - baidu is crawling deep pages on the site and then sends 80 - 100 requests to my home page over a 30 second span.
I don't usually have complaints about Baidu's behavior, and I don't want to restrict them from crawling the home page or any other page, but the behavior is rediculous. I don't mind them sending many requests, but there is no point to hitting the home page like this.
I don't currently use if-modified-since on the home page - does anyone know if Baidu sends that header?
Any advice is appreciated.