Forum Moderators: goodroi
What about allowing deep/rapid crawling only by bots from the known IP ranges of the major SEs? For any other IP that is attempting a rapid and deep crawl launch an error message?
Has it been tried? Failed or worked? Tried with XYZ consequences?