What is Robot.txt?

[deepk]

Hello everyone, i want to know what is robot.txt, what is this, how it is work?, i dont know anything about robot.txt
pls reply me

[Rsssingh]

Hi Deepk,

Robot.txt also known as the Robots Exclusion Protocol and function as a request that specified robots ignore specified files or directories in their search.

For example:

User-agent: *
Disallow:

This will allows all robots to visit all files because the wildcard "*" specifies all robots:

And,

User-agent: *
Disallow: /

This will keeps all robots out.

[Marshall]

User-agent: *
Disallow: /

This will keeps all robots out.

Only the ones that respect the exclusion file. You can also get file specific, for example:

User-agent: *
Disallow: /images/
Disallow: /DirectoryName/page.htm

and so on.

Marshall

[deepk]

what is the use, where we use this files?

[Rsssingh]

The robots.txt file must reside in the root of the domain and must be named "robots.txt". Bots only check for this file in the root of the domain. For instance, http://www.example.com/robots.txt is a valid location.

[Rsssingh]

You need a robots.txt file only if your site includes content that you don't want search engines to index. If you want search engines to index everything in your site, you don't need a robots.txt file.

[g1smd]

*** robot.txt ***

Note that it is robots.txt and not as specified in the original question.

[Rsssingh]

The Actual name of file is robots.txt