Welcome to WebmasterWorld Guest from

Forum Moderators: LifeinAsia & httpwebwitch

Message Too Old, No Replies

"Crawler" Pages

What should be in a crawler page


Michael Souder

3:46 am on Nov 2, 2001 (gmt 0)

I read that it is a good idea to include a crawler page that is obscurely accessible from your home page. The crawler page should "contain only links, not text or tags." I read that it can be used by spiders to index relavent pages on your site.

My question: What exactly should appear on the crawler page? I read that I should omit all text and tags and to include only links. What is the syntax of those links? Should I use anchor tags or something else? An example would be very helpful.


8:17 am on Nov 2, 2001 (gmt 0)

10+ Year Member

You are talking about a hallwaypage.
I always make sure that all my pages kan be reached in two clicks.

I make hallwaypages with text, tags and a little graphics.
I group my pages after content and make one hallway for each group (optimized for one keyword). All the hallways are interlinked and I have links to my hallways on most of my pages.
Since my hallways are human freindly, all the links to the hallways are visible (preferably with keywords in the links)


4:51 pm on Nov 2, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Hello Michael, welcome to WebmasterWorld!

Here is a breif summary of what I believe is a good strategy:

Make your hallway page into a "site map" type of page. Each link to a page should include a description (keyword rich, of course) of what you will find on that page. It is also a great idea to arrange the page in a directory fashion. For example

Cool-Widgets directory
-this site is designed to bring you the best information about cool widgets. Yadda Yadda...

-we have a large selection of cool-blue widgets. Whether you are looking for light blue or dark blue widgets, we can help you find what you are looking for.

-check out our new cool-light-blue-widgets. These are great for widgeting on special occasions. We have sizes S-XL.

Basically it becomes a directory page with real value to your customers/users. They should be able to quickly and easily navigate the page.

And....you get a page that spiders will (hopefully) spend lots of time jumping from.

HERE [webmasterworld.com] is lots of great info, and where I learned a lot of this stuff.

Michael Souder

6:20 pm on Nov 2, 2001 (gmt 0)

HI Travoli,

Thanks for the response and for the link to addtional info. I have briefly looked at the topic that you referred me too, but have not had a chance to read it in detail.

Your approach - and that of others who have commented - appears to be a little different than theone I originally read (by the way, from an article by Paul Boutin). You advocate making the page more than a gateway to other pages by including keyword rich descirptions for the links, correct? If this hallway page is chock full of keywords is it going to appear on search engine results and, if so, is that an issue? If it will appear in results should I include text beyond the index to give users an idea of what my site and product is about? Will the hallway page help other pages (where the real content is located) get ranked higher?

By the way, I also submitted a question about separate pages for major keyword phrases. Any input to that question is also greatly apprecited.


6:52 pm on Nov 2, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Hi Michael,

I haven't read the article you mentioned, but I'd have to disagree with the idea of only including links on the page. First of all, having a site map listed in a search engine isn't a bad thing. Having someone enter on a page that provides a clear, annotated list of all the site's content can result in the visitor being able to quickly find what they're looking for.

Also, some search engines clearly avoid indexing and following links on pages that contain no other text. Adding the additional content greatly increases the chances of a spider actually crawling all the pages listed.


3:17 pm on Oct 17, 2002 (gmt 0)

10+ Year Member

This sould like a great idea... although I wonder if this would closely resemble 'link-farms' which, I would assume, that search engines dislike...

I would hate to be banned due to usage... any thoughts or news about this method?


4:23 pm on Oct 17, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Lots of pages have a "site index" page. Google would not do well penalizing all sites with a page like this. It is common practice.

Featured Threads

Hot Threads This Week

Hot Threads This Month