Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- robots.txt


jilla - 11:20 pm on Sep 21, 2002 (gmt 0)


I followed an online tutorial where I think I correctly setup the robots text file to disallow slurp to a few pages on the site. What I am confused about though is that my index.html page to my site has a robots tag on it about following. Am I supposed to now remove the tag on my index page about following or can I still keep it? Will it conflict with the disallow sentences in my robots.txt file? Thanks.


Thread source:: http://www.webmasterworld.com/robots_txt/80.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com