Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- robots.txt


Alternative_Future - 7:33 pm on Sep 23, 2003 (gmt 0)


Superb post/reply Jim,

Many thanks for the commented example I can now make better sense of it :)
I have basically copied and pasted the example on to my server and run the tests, it works perfectly.

Would I be correct in saying that the error lay at this point:
RewriteCond %{HTTP_HOST}
!^www\.domain\.info where the exclamation point acted as the NOT? Therefore the condition was checking the domain was NOT the dot info extension, moving onwards and falling into the second condition:
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/.*\ \(Slurp/
which of course fell true each time I tested it replicating the UA Slurp?

I cannot thank you enough again Jim.


Thread source:: http://www.webmasterworld.com/robots_txt/149.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com