Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- of pipl and robots


Mokita - 7:46 pm on Dec 1, 2011 (gmt 0)


@aleksl

Our blocking of Softlayer (or any server farm) would only affect you or other webmasters (regular or not) if you or they are attempting to crawl / scrape our sites.

We are blocking traffic coming FROM the servers, not going TO the servers.

Most "regular webmaster folks" don't have any need or desire to crawl our sites.

[edited by: Mokita at 7:48 pm (utc) on Dec 1, 2011]


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4367193.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com