homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

meta name = robot
What does that do?

 8:44 pm on Jun 29, 2003 (gmt 0)

What exactly does <meta name="robots" content="index, follow"> do? and how does it differ from robots.txt?



 8:46 pm on Jun 29, 2003 (gmt 0)

In my opinion nothing, the robots will follow the links on your webpage whether you have this tag or not.

The robots.txt file will tell the robots which files you wish to restrict access to.


 8:53 pm on Jun 29, 2003 (gmt 0)

For all the info on the tag have a look here;


As for how it differs to robots.txt well...

Robots.txt is at a site level and simply acts as a signpost saying "please don't crawl these" and list of paths, however the robots meta-tag allows you to be more granular - for instance "index,nocrawl" would tell them "index this page but don't crawl any of the links that it uses" - something which can't be done with robots.txt.

Oh and welcome to WebmasterWorld [webmasterworld.com] aroye!

- Tony

...and if I do say so myself a fitting way to hit my 500th post.


 8:54 pm on Jun 29, 2003 (gmt 0)

Hi aroye, welcome to WebmasterWorld!

In addition to Symbios' reaction: here is some more or less official documentation: [robotstxt.org...]


 8:56 pm on Jun 29, 2003 (gmt 0)


Very fitting and very useful. I'm looking forward to my 50th post.


 9:05 pm on Jun 29, 2003 (gmt 0)

Thanks Guys!


 5:57 pm on Jun 30, 2003 (gmt 0)

Some readers of this thread might find this thread [webmasterworld.com] interesting. If you want to control the "presentation" of your pages in search engine results, there are very good reasons to use a mixture of both robots.txt and the robots meta tag.


Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved