Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- SonicWALL Firewall Blocks Spiders


moltar - 1:33 am on May 10, 2005 (gmt 0)


Interestingly enough their own site is well spiered.

site:www.sonicwall.com shows 31 pages in Y and 4,390 in G :)

They also use robots.txt and I tried requesting it and didn't get banned.


Thread source:: http://www.webmasterworld.com/robots_txt/631.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com