Welcome to WebmasterWorld Guest from 54.234.38.8

Forum Moderators: brotherhood of lan & mack

Message Too Old, No Replies

How does a spider see all my pages?

Do I need the "index, follow" tag on all pages?

     
10:32 am on Dec 11, 2002 (gmt 0)

Full Member

10+ Year Member

joined:Sept 2, 2002
posts:210
votes: 0


Is it better to have the following on all pages of my site:

<meta name="robots" content="index, follow">

or just the index page?

If it's on all pages, will this be easier for robots to spider the site? We have a very good internal link system already, but i wonder if this would be a good move...

10:44 am on Dec 11, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 27, 2002
posts:1685
votes: 0


I use <META NAME="Robot" CONTENT="ALL"> on every page, including index.html.

It would seem if you want them to visit, 'ya gotten tell 'em.

Pendanticist.

10:49 am on Dec 11, 2002 (gmt 0)

Preferred Member

10+ Year Member

joined:Sept 20, 2001
posts:478
votes: 0


I use a robots.txt file to dissallow any folders I don't want them to spider. I do not use a meta robots tag on the page.
2:35 pm on Dec 11, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 14, 2002
posts:1192
votes: 0


Never use it, spiders have had no difficulty indexing my whole site. The main purpose of that tag is to prevent spiders from exploring parts of your site, following links is their default behavior.
5:58 pm on Dec 11, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 23, 2000
posts:1277
votes: 0


mohamed has it right. I don't think they pay attention to follow tags, just no index ones (and even then it is suspect). Spiders usually follow links to other pages and will index your site if you have given them a clear path from page to page.
6:42 pm on Dec 11, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 27, 2002
posts:1685
votes: 0


I don't think any one solution is right. Obviously both work.

Pendanticist.