Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Mozilla/4.* (Compatible


g1smd - 7:29 pm on May 20, 2012 (gmt 0)


Yes. I meant to say "MSIE 6.0 and before".

I see a lot of scraper bots, form filling spammer bots and little or no evidence of real human accesses.

If you mean versions 1 to 6 then
(1-6) should be [1-6] here.

MSIE/\ should be MSIE\ too. The slash should not be escaped.

Finally,
RewriteRule 403.php [F]
should be
RewriteRule !^403\.php - [F]
or similar.


Thread source:: http://www.webmasterworld.com/robots_txt/4429094.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com