Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Baidu's IPs - Which Are Legit?
rowan194 - 3:28 pm on Aug 15, 2012 (gmt 0)Thread source:: http://www.webmasterworld.com/search_engine_spiders/4475767.htm
I understand that robots.txt doesn't block a site. What I meant was that Baidu don't seem to respect my requests in that file for them to not go anywhere on my site. Thus, I moved towards blocking at the IP level.
Brought to you by WebmasterWorld: http://www.webmasterworld.com