Forum Moderators: open

Message Too Old, No Replies

Insuficient Spidering

Spidering only 1800 of our 6500 "catalog" pages

         

Valhalla

9:57 pm on Mar 25, 2003 (gmt 0)

10+ Year Member



Regarding Google spidering only 1800 of our 6500 "catalog" pages: the home page, www.davidrumsey.com <http://www.davidrumsey.com> , has a page rank of 7. All 6500 catalog pages are in the same directory as the home page - each one represents a catalog record in our online library - an example is www.davidrumsey.com/maps200.html <http://www.davidrumsey.com/maps200.html> You can see the list of these catalog pages by going to the bottom of www.davidrumsey.com <http://www.davidrumsey.com> and clicking on the link "spg pages" or just go to [davidrumsey.com...] Each of the 1800 spidered catalog pages has a page rank of 3. We are puzzled by two things: why the low page rank, given the home page rank of 7, and more importantly, we see the spider stop each visit when it gets to about the 1800th page on the list of catalog pages - why does it not spider the entire list? The catalog pages are numbered sequentially, ie www.davidrumsey.com/maps150.html <http://www.davidrumsey.com/maps150.html> and going up to about www.davidrumsey.com/maps6613.html <http://www.davidrumsey.com/maps6613.html> . These pages have been up for over a year now.

sun818

10:17 pm on Mar 25, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



The site doesn't seem to be spider friendly with odd navigation and dynamic URLs. You can take a look at the SimSpider on SearchEngineWorld yourself:

[searchengineworld.com...]