Welcome to WebmasterWorld Guest from 188.8.131.52
Forum Moderators: goodroi
However, I found the crawl-delay seems to be set per domain, so if I have a site with 50K subdomain hosting on a single server, crawl-delay = 1 is useless, seem they can fetch 50K requests per second.
I belive they know as the 50K subdomain are on a single IP.
I want to ask, is it possible to limit the crawl-rate by server IP, rather than domain / subdomain.
How do you solve it if you have many subdomains?
It may be that it takes some time to associate all the domains and subdomains. Has any change taken place recently, such as a new IP address, or more subdomains added to your 'collection' on the single IP address?
Ultimately, the decision of how many (sub)domains to host on a server should take crawling into account. Fifty thousand is a very high number -- about 125 times higher than a 'normal' shared hosting maximum for medium- to low-traffic sites. So, you might consider setting each crawl-delay to at least 120 if you really intend to host that many sites on one server.