Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- 150.70. revisited


SteveWh - 5:49 am on Sep 18, 2011 (gmt 0)


Ok...

"They are going to consider any URL that their user is able to visit to be fair game for them to fetch and analyze for malware."

That's what I meant.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4363886.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com