Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

robots.txt question

8:51 pm on Mar 28, 2003 (gmt 0)

New User

10+ Year Member

joined:Mar 5, 2003
votes: 0

Do you have to insert any thing in the head of a page to let people know that their is a robots.txt file?
9:01 pm on Mar 28, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
votes: 0


You either have a robots.txt file or you don't.

If you don't have one, you will get lots of 404 errors in your log files, as search engine spiders attempt to look at it. So even if you do not wish to keep spiders out of certain pages, it's a good idea to have a robots.txt file, even if it is blank.

You can also control robots' behaviour by including a meta tag in each of your pages, but this is a second method, unrelated to the robots.txt method.

For more information on robots.txt and the robots meta tag, see the robots exclusion standard [robotstxt.org], and follow the links as well.