Rsssingh
msg:3726955 | 9:24 am on Aug 20, 2008 (gmt 0) |
Hi Deepk, Robot.txt also known as the Robots Exclusion Protocol and function as a request that specified robots ignore specified files or directories in their search. For example: User-agent: *
Disallow: This will allows all robots to visit all files because the wildcard "*" specifies all robots: And, User-agent: *
Disallow: / This will keeps all robots out.
|
Marshall
msg:3726963 | 9:34 am on Aug 20, 2008 (gmt 0) |
User-agent: *
Disallow: / This will keeps all robots out. |
|
Only the ones that respect the exclusion file. You can also get file specific, for example: User-agent: *
Disallow: /images/
Disallow: /DirectoryName/page.htm and so on. Marshall
|
deepk
msg:3726964 | 9:34 am on Aug 20, 2008 (gmt 0) |
what is the use, where we use this files?
|
Rsssingh
msg:3726970 | 9:55 am on Aug 20, 2008 (gmt 0) |
The robots.txt file must reside in the root of the domain and must be named "robots.txt". Bots only check for this file in the root of the domain. For instance, http://www.example.com/robots.txt is a valid location.
|
Rsssingh
msg:3726971 | 9:56 am on Aug 20, 2008 (gmt 0) |
You need a robots.txt file only if your site includes content that you don't want search engines to index. If you want search engines to index everything in your site, you don't need a robots.txt file.
|
g1smd
msg:3727181 | 2:17 pm on Aug 20, 2008 (gmt 0) |
*** robot.txt *** Note that it is robots.txt and not as specified in the original question.
|
Rsssingh
msg:3727887 | 7:16 am on Aug 21, 2008 (gmt 0) |
The Actual name of file is robots.txt
|
|
