Forum Moderators: open
We have about 1600 listings which are typically displayed via this template page:
http://www.example.com/homepage.aspx?SPID=7986
http://www.example.com/homepage.aspx?SPID=9718
…etc
In an effort to restructure the site for better performance on Google, we have created a static ‘browse’ system based on the geographic location of each accommodation establishment.
The geographic locations are broken down into:
-Country
-Province
-Town
-suburb.
…resulting in the creation of about 6000 new pages, with an example URL from the new structure looking like this:
http://www.example.com/.../Camps-Bay/
Where:
Country = South-Africa
Province = Western Cape
Town = Cape Town
Suburb = Camps Bay
(We have put everything under an extra directory called ‘travel-accommodation’ in order to include 2 major keywords which were otherwise missing.)
We feel that this is a very logical structure; the user experience of surfing this data is great, and the vertical ‘themes’ down the tree are clearly defined.
Our worry is that much of our good content is about 5 directory levels deep, and we suspect that Google only spiders to a certain depth. (This makes sense, if Google can’t index the entire web then its better to index all the sites partially, rather than only some of the sites in their entirety).
SO… we are looking to restructure the browse in one of these ways:
OPTION 1) Create the 6000 files all at the root, with long file names, like this:
http://www.example.com/...Cape-Cape-Town.html
OPTION 2) Create the 6000 files one directory off the root, with long directory names, and using the default file of the webserver to process the result, like this
http://www.example.com/...Cape-Town-Camps-Bay/
My questions are:
a) Does google limit its crawl based on directory depth?
b) Which is the better option from the 2 options above?
(Why? are files closer to the root scored better?)
c) Is this better than our current multi-level-deep directory structure?
Any advice greatly appreciated!
Matthew
[1][[b]edited by[/b]: Brett_Tabke at 1:00 pm (utc) on Mar. 24, 2003][/1]
[1][edit reason] generic urls... [/edit][/1]
Note that Google suggests a maximum of about 100 links per page, hence the suggestion of a hierarchy of link pages rather than one huge one.
Our worry is that much of our good content is about 5 directory levels deep, and we suspect that Google only spiders to a certain depth.
As Mohammed says: the amount of clicks/links away is important and the amount of links per page. I would add that the higher the pagerank, the bigger the chance of getting very deep spidering/indexing. Also some external inbound links into deeper pages would help as well IMO.
Yahoo has categories that are nine levels deep for example.
Added: did Alltheweb ditch the directory level search?
[webmasterworld.com...]
I agree with Mohamed_E and vitaplease that Google doesn't care about your directory structure. For a site with PR6 on the homepage, having several directory levels like above won't be a problem for Google. If you want to put it in an add or so, it is way too long for a user to enter it from a keyboard.
When there is only one accommodation in a country or province, a lot of pages with only 1 link to the next page, will dilute the PR. So the final page will have low PR. In such a case you should consider a direct link.
After looking at the site, I would like to ask you to think about the following 2 points:
1. If there is more than one accommodation in a suburb, adding a link from the every page in a suburb to the overview of all accommodations in this suburb should be helpful for the user. Since the user might like the location, but the accommodation is not what the user is looking for (some kind of facility is missing, too expensive, already full in high season, etc), going up one or more levels in your site structure is welcome. Just like the blue bar at the top of this page, or the top of a DMOZ page.
2. Having a direct link from all these 6000 pages to 'About Us', 'Terms of Use' and 'Copyright' pages is PR-wise not a good idea. You better use an indirect link (JavaScript or so) if you want to add these links on every page.
I agree that we are polluting our clean structure with too many links to unimportant pages.
Do you think its best to:
a) put those unimportant pages in an area which is not spiderable (robots.txt)
or
b) make the links non-spiderable (javascript?)
I thought that google can follow JavaScript links...
Are there any other ways to force a spider to NOT follow a particular link?
Takagi: Thank you so much for the dmoz help. I really appreciate that. I see that many people are experiencing the same issues as me with regard to Dmoz, I wonder what the problem is? weird...
It seems there is a maximum of PR transfer from many identical links on the same site. If you look at the bottom of this page you see a link to BestBBS. Searching
site:www.webmasterworld.com bestbbs
with Google will show you 43,500 pages. Searching
link:www.bestbbs.com
gives 8,100 pages. That is, 8,100 pages with at least a PR4 (up to PR7). But still the PR of this site (which consist of only 1 page) is PR4!
If (almost) identical links are coming from many different sites, there seems to be no limit. A good example is a 'competitor': phpBB. Most of the 73,000 links from pages with PR4 or higher have a "Powered by phpBB" link at the bottom at many many forums. But they have a PR10.
Your first proposal to prevent PR going to unimportant pages like 'About Us', 'Copyright' and 'Terms of Use', is to make these pages not spiderable using robots.txt. I'm not sure if that will help the PR of other pages linked on the same page, if the links to the unspiderable pages are still there.
Your second proposal is using JavaScript. As far as I know, Google doesn't follow links in JavaScript at this moment. If you are afraid Google will in the future (a few years ago, Google couldn't read the text in PDFs, now they even follow links in PDFs), you could put the real URL in a JavaScript include file, and make this include file not spiderable.
I would do the last one, and only add a real link to the unimportant pages from a low ranked site map page. In that case, Google would still index these pages, but with minimal PR loss.
