Welcome to WebmasterWorld Guest from 54.145.208.64

Forum Moderators: goodroi

Message Too Old, No Replies

What is Robot.txt?

What is robot.txt,

   
6:57 am on Aug 20, 2008 (gmt 0)

5+ Year Member



Hello everyone, i want to know what is robot.txt, what is this, how it is work?, i dont know anything about robot.txt
pls reply me
9:24 am on Aug 20, 2008 (gmt 0)

5+ Year Member



Hi Deepk,

Robot.txt also known as the Robots Exclusion Protocol and function as a request that specified robots ignore specified files or directories in their search.

For example:

User-agent: *
Disallow:

This will allows all robots to visit all files because the wildcard "*" specifies all robots:

And,

User-agent: *
Disallow: /

This will keeps all robots out.

9:34 am on Aug 20, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



User-agent: *
Disallow: /

This will keeps all robots out.

Only the ones that respect the exclusion file. You can also get file specific, for example:

User-agent: *
Disallow: /images/
Disallow: /DirectoryName/page.htm

and so on.

Marshall

9:34 am on Aug 20, 2008 (gmt 0)

5+ Year Member



what is the use, where we use this files?
9:55 am on Aug 20, 2008 (gmt 0)

5+ Year Member



The robots.txt file must reside in the root of the domain and must be named "robots.txt". Bots only check for this file in the root of the domain. For instance, http://www.example.com/robots.txt is a valid location.
9:56 am on Aug 20, 2008 (gmt 0)

5+ Year Member



You need a robots.txt file only if your site includes content that you don't want search engines to index. If you want search engines to index everything in your site, you don't need a robots.txt file.
2:17 pm on Aug 20, 2008 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



*** robot.txt ***

Note that it is robots.txt and not as specified in the original question.

7:16 am on Aug 21, 2008 (gmt 0)

5+ Year Member



The Actual name of file is robots.txt