jilla - 11:20 pm on Sep 21, 2002 (gmt 0)
I followed an online tutorial where I think I correctly setup the robots text file to disallow slurp to a few pages on the site. What I am confused about though is that my index.html page to my site has a robots tag on it about following. Am I supposed to now remove the tag on my index page about following or can I still keep it? Will it conflict with the disallow sentences in my robots.txt file? Thanks.