homepage Welcome to WebmasterWorld Guest from 54.211.219.68
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / WebmasterWorld / Professional Webmaster Business Issues
Forum Library, Charter, Moderators: LifeinAsia & httpwebwitch

Professional Webmaster Business Issues Forum

    
"Crawler" Pages
What should be in a crawler page
Michael Souder

10+ Year Member



 
Msg#: 238 posted 3:46 am on Nov 2, 2001 (gmt 0)

I read that it is a good idea to include a crawler page that is obscurely accessible from your home page. The crawler page should "contain only links, not text or tags." I read that it can be used by spiders to index relavent pages on your site.

My question: What exactly should appear on the crawler page? I read that I should omit all text and tags and to include only links. What is the syntax of those links? Should I use anchor tags or something else? An example would be very helpful.

 

bufferzone

10+ Year Member



 
Msg#: 238 posted 8:17 am on Nov 2, 2001 (gmt 0)

You are talking about a hallwaypage.
I always make sure that all my pages kan be reached in two clicks.

I make hallwaypages with text, tags and a little graphics.
I group my pages after content and make one hallway for each group (optimized for one keyword). All the hallways are interlinked and I have links to my hallways on most of my pages.
Since my hallways are human freindly, all the links to the hallways are visible (preferably with keywords in the links)

Travoli

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 238 posted 4:51 pm on Nov 2, 2001 (gmt 0)

Hello Michael, welcome to WebmasterWorld!

Here is a breif summary of what I believe is a good strategy:

Make your hallway page into a "site map" type of page. Each link to a page should include a description (keyword rich, of course) of what you will find on that page. It is also a great idea to arrange the page in a directory fashion. For example

*www.cool-widgets.com
____________________________________________
Cool-Widgets directory
-this site is designed to bring you the best information about cool widgets. Yadda Yadda...

Cool-Blue-Widgets
-we have a large selection of cool-blue widgets. Whether you are looking for light blue or dark blue widgets, we can help you find what you are looking for.

Cool-Light-Blue-Widgets
-check out our new cool-light-blue-widgets. These are great for widgeting on special occasions. We have sizes S-XL.

________________________________________________________
Basically it becomes a directory page with real value to your customers/users. They should be able to quickly and easily navigate the page.

And....you get a page that spiders will (hopefully) spend lots of time jumping from.

HERE [webmasterworld.com] is lots of great info, and where I learned a lot of this stuff.

Michael Souder

10+ Year Member



 
Msg#: 238 posted 6:20 pm on Nov 2, 2001 (gmt 0)

HI Travoli,

Thanks for the response and for the link to addtional info. I have briefly looked at the topic that you referred me too, but have not had a chance to read it in detail.

Your approach - and that of others who have commented - appears to be a little different than theone I originally read (by the way, from an article by Paul Boutin). You advocate making the page more than a gateway to other pages by including keyword rich descirptions for the links, correct? If this hallway page is chock full of keywords is it going to appear on search engine results and, if so, is that an issue? If it will appear in results should I include text beyond the index to give users an idea of what my site and product is about? Will the hallway page help other pages (where the real content is located) get ranked higher?

By the way, I also submitted a question about separate pages for major keyword phrases. Any input to that question is also greatly apprecited.

WebGuerrilla

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 238 posted 6:52 pm on Nov 2, 2001 (gmt 0)


Hi Michael,

I haven't read the article you mentioned, but I'd have to disagree with the idea of only including links on the page. First of all, having a site map listed in a search engine isn't a bad thing. Having someone enter on a page that provides a clear, annotated list of all the site's content can result in the visitor being able to quickly find what they're looking for.

Also, some search engines clearly avoid indexing and following links on pages that contain no other text. Adding the additional content greatly increases the chances of a spider actually crawling all the pages listed.

wangdy

10+ Year Member



 
Msg#: 238 posted 3:17 pm on Oct 17, 2002 (gmt 0)

This sould like a great idea... although I wonder if this would closely resemble 'link-farms' which, I would assume, that search engines dislike...

I would hate to be banned due to usage... any thoughts or news about this method?

Travoli

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 238 posted 4:23 pm on Oct 17, 2002 (gmt 0)

Lots of pages have a "site index" page. Google would not do well penalizing all sites with a page like this. It is common practice.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / WebmasterWorld / Professional Webmaster Business Issues
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved