Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- of pipl and robots


wilderness - 3:32 am on Mar 10, 2012 (gmt 0)


Had a request for robots.text and two requests for the main page from:

159.253.143.zz - - [10/Mar/2012:02:50:00 +0000] "GET / HTTP/1.0" 301 234 "-" "Mozilla/5.0 (compatible; SWEBot/1.0; +http://swebot-crawler.net)"

The interesting part is that the IP is returned as

netname: NETBLK-SOFTLAYER-RIPE-CUST-MB27388-RIPE
descr: Hosting Services Inc. (Midphase)
country: US

Started doing requests to see how far the Class C went and still belonged to Softlayer.
After four verifications of small Class D blocks (to different orgs) up to 111, I stopped and just denied that entire Class B, which may not be beneficial to everyone.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4367193.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com