homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Marketing and Biz Dev / Deprecated - Search Engine Submission
Forum Library, Charter, Moderator: open

Deprecated - Search Engine Submission Forum

Better to Exclude part of Site?
spider individual pages or index pages?

WebmasterWorld Senior Member 10+ Year Member

Msg#: 367 posted 8:24 am on May 25, 2002 (gmt 0)

I have a site that happens to be very large. (70+meg, 8000+ pages)
Because of this, I have "index" pages that link to individual pages.
Would it be wise to exclude either the "index" pages or the individual pages using robots.txt? I was thinking of excluding the individual pages just because I doubt many spiders will really bother to go through all of the site.
The site itself is also highly interrelated, often having dozens on links to other pages within the site on an individual page.

site on profile


brotherhood of LAN

WebmasterWorld Administrator brotherhood_of_lan us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Msg#: 367 posted 2:33 pm on May 25, 2002 (gmt 0)

hello dcheney

its a good question. In my instances, the only content I disallow is content not relevant to the site (ie contact us/feedback/buy this) sort of thing.

You have to consider that if you exclude certain parts of your site, that you will prevent the bot from navigating your whole site, and it will affect your overall navigation (as far as certain search engines and page rank is concerned)

im sure one of the experienced ones in here will be able to chip in. My largest site is about the quarter of the size of yours and its pretty much left open to spidering.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Marketing and Biz Dev / Deprecated - Search Engine Submission
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved