Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- SonicWALL Firewall Blocks Spiders

SEOMike - 6:59 pm on May 4, 2005 (gmt 0)


This is pretty ridiculous. Anyone that is familiar with robots.txt should know what it does and why it's important. Maybe they should have had the firewall READ the robots.txt and block spiders from the directories that were identified in it. Think that would work?

Thread source:: http://www.webmasterworld.com/robots_txt/631.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com