Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- SonicWALL Firewall Blocks Spiders


EliteWeb - 3:50 pm on May 4, 2005 (gmt 0)


True robots.txt leads to people putting in URLs that shouldn't be spidered, but the spiders wouldn't even be able to obey them.

Requesting v. number and how this got part of the security package from admin :)


Thread source:: http://www.webmasterworld.com/robots_txt/631.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com