Welcome to WebmasterWorld Guest from 54.196.244.206

Forum Moderators: goodroi

Message Too Old, No Replies

meta name = robot

What does that do?

     
8:44 pm on Jun 29, 2003 (gmt 0)

New User

10+ Year Member

joined:June 15, 2003
posts:2
votes: 0


What exactly does <meta name="robots" content="index, follow"> do? and how does it differ from robots.txt?
8:46 pm on June 29, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:May 28, 2003
posts:420
votes: 0


In my opinion nothing, the robots will follow the links on your webpage whether you have this tag or not.

The robots.txt file will tell the robots which files you wish to restrict access to.

8:53 pm on June 29, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 25, 2002
posts:872
votes: 0


For all the info on the tag have a look here;

[searchengineworld.com...]

As for how it differs to robots.txt well...

Robots.txt is at a site level and simply acts as a signpost saying "please don't crawl these" and list of paths, however the robots meta-tag allows you to be more granular - for instance "index,nocrawl" would tell them "index this page but don't crawl any of the links that it uses" - something which can't be done with robots.txt.

Oh and welcome to WebmasterWorld [webmasterworld.com] aroye!

- Tony

...and if I do say so myself a fitting way to hit my 500th post.

8:54 pm on June 29, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 22, 2002
posts:1749
votes: 0


Hi aroye, welcome to WebmasterWorld!

In addition to Symbios' reaction: here is some more or less official documentation: [robotstxt.org...]

8:56 pm on June 29, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:May 28, 2003
posts:420
votes: 0


Dreamquick,

Very fitting and very useful. I'm looking forward to my 50th post.

9:05 pm on June 29, 2003 (gmt 0)

New User

10+ Year Member

joined:June 15, 2003
posts:2
votes: 0


Thanks Guys!
5:57 pm on June 30, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


Some readers of this thread might find this thread [webmasterworld.com] interesting. If you want to control the "presentation" of your pages in search engine results, there are very good reasons to use a mixture of both robots.txt and the robots meta tag.

Jim