Page is a not externally linkable
Mokita - 1:35 am on Dec 3, 2010 (gmt 0)
wilderness wrote:
If your website is either RIPE or APNIC based and you denied access to North American-based-major-SE's while allowing their bots to spider from their perspective RIPE or APNIC ranges?
Most of our sites are APNIC based. I don't think I have ever seen a verifiable, googlebot or bingbot/msnbot crawling from anything other than ARIN based IPs. Blocking those would be incredibly self-defeating (unless you don't want any traffic).
Yahoo only crawls from ARIN based IPs or Chinese - and I have been blocking Slurp China (APNIC based) for a very long time, with no apparent effect on our listings in Yahoo.
HTH.