homepage Welcome to WebmasterWorld Guest from 54.166.66.204
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / WebmasterWorld / New To Web Development
Forum Library, Charter, Moderators: brotherhood of lan & mack

New To Web Development Forum

    
How does a spider see all my pages?
Do I need the "index, follow" tag on all pages?
Monkscuba




msg:970745
 10:32 am on Dec 11, 2002 (gmt 0)

Is it better to have the following on all pages of my site:

<meta name="robots" content="index, follow">

or just the index page?

If it's on all pages, will this be easier for robots to spider the site? We have a very good internal link system already, but i wonder if this would be a good move...

 

pendanticist




msg:970746
 10:44 am on Dec 11, 2002 (gmt 0)

I use <META NAME="Robot" CONTENT="ALL"> on every page, including index.html.

It would seem if you want them to visit, 'ya gotten tell 'em.

Pendanticist.

SmallTime




msg:970747
 10:49 am on Dec 11, 2002 (gmt 0)

I use a robots.txt file to dissallow any folders I don't want them to spider. I do not use a meta robots tag on the page.

Mohamed_E




msg:970748
 2:35 pm on Dec 11, 2002 (gmt 0)

Never use it, spiders have had no difficulty indexing my whole site. The main purpose of that tag is to prevent spiders from exploring parts of your site, following links is their default behavior.

JamesR




msg:970749
 5:58 pm on Dec 11, 2002 (gmt 0)

mohamed has it right. I don't think they pay attention to follow tags, just no index ones (and even then it is suspect). Spiders usually follow links to other pages and will index your site if you have given them a clear path from page to page.

pendanticist




msg:970750
 6:42 pm on Dec 11, 2002 (gmt 0)

I don't think any one solution is right. Obviously both work.

Pendanticist.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / WebmasterWorld / New To Web Development
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved