homepage Welcome to WebmasterWorld Guest from 54.237.249.10
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
What is Robot.txt?
What is robot.txt,
deepk




msg:3726896
 6:57 am on Aug 20, 2008 (gmt 0)

Hello everyone, i want to know what is robot.txt, what is this, how it is work?, i dont know anything about robot.txt
pls reply me

 

Rsssingh




msg:3726955
 9:24 am on Aug 20, 2008 (gmt 0)

Hi Deepk,

Robot.txt also known as the Robots Exclusion Protocol and function as a request that specified robots ignore specified files or directories in their search.

For example:

User-agent: *
Disallow:

This will allows all robots to visit all files because the wildcard "*" specifies all robots:

And,

User-agent: *
Disallow: /

This will keeps all robots out.

Marshall




msg:3726963
 9:34 am on Aug 20, 2008 (gmt 0)

User-agent: *
Disallow: /

This will keeps all robots out.

Only the ones that respect the exclusion file. You can also get file specific, for example:

User-agent: *
Disallow: /images/
Disallow: /DirectoryName/page.htm

and so on.

Marshall

deepk




msg:3726964
 9:34 am on Aug 20, 2008 (gmt 0)

what is the use, where we use this files?

Rsssingh




msg:3726970
 9:55 am on Aug 20, 2008 (gmt 0)

The robots.txt file must reside in the root of the domain and must be named "robots.txt". Bots only check for this file in the root of the domain. For instance, http://www.example.com/robots.txt is a valid location.

Rsssingh




msg:3726971
 9:56 am on Aug 20, 2008 (gmt 0)

You need a robots.txt file only if your site includes content that you don't want search engines to index. If you want search engines to index everything in your site, you don't need a robots.txt file.

g1smd




msg:3727181
 2:17 pm on Aug 20, 2008 (gmt 0)

*** robot.txt ***

Note that it is robots.txt and not as specified in the original question.

Rsssingh




msg:3727887
 7:16 am on Aug 21, 2008 (gmt 0)

The Actual name of file is robots.txt

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved