meta name = robot

What does that do?

[aroye]

What exactly does <meta name="robots" content="index, follow"> do? and how does it differ from robots.txt?

[Symbios]

In my opinion nothing, the robots will follow the links on your webpage whether you have this tag or not.

The robots.txt file will tell the robots which files you wish to restrict access to.

[Dreamquick]

For all the info on the tag have a look here;

[searchengineworld.com...]

As for how it differs to robots.txt well...

Robots.txt is at a site level and simply acts as a signpost saying "please don't crawl these" and list of paths, however the robots meta-tag allows you to be more granular - for instance "index,nocrawl" would tell them "index this page but don't crawl any of the links that it uses" - something which can't be done with robots.txt.

Oh and welcome to WebmasterWorld [webmasterworld.com] aroye!

- Tony

...and if I do say so myself a fitting way to hit my 500th post.

[RonPK]

Hi aroye, welcome to WebmasterWorld!

In addition to Symbios' reaction: here is some more or less official documentation: [robotstxt.org...]

[Symbios]

Dreamquick,

Very fitting and very useful. I'm looking forward to my 50th post.

[aroye]

Thanks Guys!

[jdMorgan]

Some readers of this thread might find this thread [webmasterworld.com] interesting. If you want to control the "presentation" of your pages in search engine results, there are very good reasons to use a mixture of both robots.txt and the robots meta tag.

Jim