- - Search Engines
- -- Sitemaps, Meta Data, and robots.txt
- ---- SonicWALL Firewall Blocks Spiders
mrMister - 6:31 pm on May 4, 2005 (gmt 0)
|Blocking any UA (spider or browser) that accesses robots.txt is surely not a solution. |
I agree with you there. But its no different to many other things that some of these "top end" firewalls "protect" you against.
They have to justify the extreme cost of these devices somehow!
Thread source: http://www.webmasterworld.com/robots_txt/631.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com