Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: goodroi
A couple of months ago I decided to set up a robots.txt file to ban all bots from visiting a couple of pages on a couple of sites I run, I set it up as follows:
However I find that all ot these pages are still being crawled, I have done a bit more research and now changed the file to read:
The robots.txt file is in the root folder, and can be accessed by www.mysite/robots.txt
Is the leading / the problem, or would this not normally affect it? Any other suggestions?
Cheers for any advice,
The standard says to put a leading slash every time, so that's what I do. I don't see anything wrong with your syntax, so I don't think you should have any problems.