Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Is there a "best" robot txt to get all pages ranked?


EandA - 5:46 am on Jul 13, 2003 (gmt 0)


Hello,

I a SEM newbie and wanted to know what is the absolute best robot txt coding one should use to insure Google and the major SE's spiders index all of my sites pages?

I've seen the following used:

<meta content="index,follow" name="robots">

<meta name="robots" content="all">

<meta content="all" name="robots">

Which one is best? Or are they all equally good and does it even matter which one I use?

Next, is there an even better one than these three? If so, what is it?

Lastly, is there any other key coding/files I need to insure spiders index all of my sites page? If so, please list with enough details on how to implement for us newbies!

Many thanks,

Randall


Thread source:: http://www.webmasterworld.com/robots_txt/132.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com