Welcome to WebmasterWorld Guest from 54.227.83.19

Forum Moderators: goodroi

Message Too Old, No Replies

robots.txt question

     

deltron23

8:51 pm on Mar 28, 2003 (gmt 0)

10+ Year Member



Do you have to insert any thing in the head of a page to let people know that their is a robots.txt file?

jdMorgan

9:01 pm on Mar 28, 2003 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member



No.

You either have a robots.txt file or you don't.

If you don't have one, you will get lots of 404 errors in your log files, as search engine spiders attempt to look at it. So even if you do not wish to keep spiders out of certain pages, it's a good idea to have a robots.txt file, even if it is blank.

You can also control robots' behaviour by including a meta tag in each of your pages, but this is a second method, unrelated to the robots.txt method.

For more information on robots.txt and the robots meta tag, see the robots exclusion standard [robotstxt.org], and follow the links as well.

HTH,
Jim

 

Featured Threads

Hot Threads This Week

Hot Threads This Month