homepage Welcome to WebmasterWorld Guest from 54.211.34.105
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
What is Robot.txt?
What is robot.txt,
deepk

5+ Year Member



 
Msg#: 3726894 posted 6:57 am on Aug 20, 2008 (gmt 0)

Hello everyone, i want to know what is robot.txt, what is this, how it is work?, i dont know anything about robot.txt
pls reply me

 

Rsssingh

5+ Year Member



 
Msg#: 3726894 posted 9:24 am on Aug 20, 2008 (gmt 0)

Hi Deepk,

Robot.txt also known as the Robots Exclusion Protocol and function as a request that specified robots ignore specified files or directories in their search.

For example:

User-agent: *
Disallow:

This will allows all robots to visit all files because the wildcard "*" specifies all robots:

And,

User-agent: *
Disallow: /

This will keeps all robots out.

Marshall

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3726894 posted 9:34 am on Aug 20, 2008 (gmt 0)

User-agent: *
Disallow: /

This will keeps all robots out.

Only the ones that respect the exclusion file. You can also get file specific, for example:

User-agent: *
Disallow: /images/
Disallow: /DirectoryName/page.htm

and so on.

Marshall

deepk

5+ Year Member



 
Msg#: 3726894 posted 9:34 am on Aug 20, 2008 (gmt 0)

what is the use, where we use this files?

Rsssingh

5+ Year Member



 
Msg#: 3726894 posted 9:55 am on Aug 20, 2008 (gmt 0)

The robots.txt file must reside in the root of the domain and must be named "robots.txt". Bots only check for this file in the root of the domain. For instance, http://www.example.com/robots.txt is a valid location.

Rsssingh

5+ Year Member



 
Msg#: 3726894 posted 9:56 am on Aug 20, 2008 (gmt 0)

You need a robots.txt file only if your site includes content that you don't want search engines to index. If you want search engines to index everything in your site, you don't need a robots.txt file.

g1smd

WebmasterWorld Senior Member g1smd us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3726894 posted 2:17 pm on Aug 20, 2008 (gmt 0)

*** robot.txt ***

Note that it is robots.txt and not as specified in the original question.

Rsssingh

5+ Year Member



 
Msg#: 3726894 posted 7:16 am on Aug 21, 2008 (gmt 0)

The Actual name of file is robots.txt

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved