Forum Moderators: open

Message Too Old, No Replies

Dynamic Site - Spidering Deeper Pages

Linking From Html pages to get deeper spidering -Or Spam?

         

robbo

7:32 am on Sep 18, 2004 (gmt 0)

10+ Year Member



Hi,
I have a dynamic e-commerce site and am having problems getting my deeper product pages spidered.
I have recently added a HTML links section to the site with about 30 individual pages.
I have links to the main product categories and for each separate page at the bottom I have also included 15 Anchor links to the deeper individual products.I have about 75 products and figured that these links pages would supply about 6 links to each product.
I did read somewhere that google suggests that it is beneficial to direct links from HTML pages to dynamic to enable them to more thoroughly spider these pages.
My concern is that this may be considered spam , if there is any doubt I will remove these links.
If this is the case are there any other suggestions to achieve this.(I have an all products page and am now working on building up my links).
Any advice would be really appreciated.
Robbo

Dave_A

2:48 am on Sep 19, 2004 (gmt 0)

10+ Year Member



As a search engine we would see that as a subdomain or a replica (Mirror) to increase traffic and the chance to get more hits.
Our spider goes in three levels down on the first visit picking up fifty links per level. Each time it re visits, it goes in deeper each time.
One web site I know of has around 30,000 pages and our spider took four goes to get it all. We have to be careful not to block the server, if we get our nose bag into a web site and dig to deep, to quick we can almost shut the server down.
Most web sites are stored in a host that allows around 100mb a month of bandwidth, if we dive in and start pulling heaps of data, it can eat at Bandwidth measurements but not show as visits.
Google from what I have seen of it's operation will only go in at around four levels and take around five to ten links per level so it may take a few visits before it follows all your links but it should get down to the bottom at some stage of your web site.
Spiders will follow all links, html or java and get there in the end.
All the best
Dave Andrews

robbo

4:35 am on Sep 19, 2004 (gmt 0)

10+ Year Member



Thanks for your reply Dave.
perhaps I wasn't too clear - my new links pages are directory pages of other sites, with each one relating to a particular niche of my own industry ie. relevant to my site and possibly of interest to my customers.As such I don't believe they are mirror pages.
My own internal links I am talking about are ones placed at the bottom of the page and go directly to individual products within my site - these pages are about 4 levels deep which at present are in googles index, but only the title has been spidered.
The site is about 7 months old. I just wondered if by adding these links directly from an HTML page as opposed to PHP it would help me get more of the site i.e these contents rich product pages more deeply spidered,or is this method considered spammy.
If so does anyone have any other suggestions to achieve this.
Thanks Robbo