I have a couple of sites that have been running for about a year using a CMS I built myself in ASP. One of the sites does not seem to be indexed beyond the home page, the other seems to be reasonably well indexed.
Firstly the sites
Example1.com - this is the site that is not being indexed)
Example2.com - this is well indexed
If you look at example1.com, you will see that google sees the home page and 3 old links.
site:www.example2.com/ - Google has numerous links
Both sites (main page) have a PR of 2 according to googlebar, but nothing elsewhere.
Checking links via a search tool
<specific tool removed>
I used the above tool for both sites and the URLs appear to be well-formed
Both sites have a sitemap that is reasonably up to date and valid.
Having just looked at the web logs on october 2nd (when the cache for exmple1.com was updated) I have the following lines
2006-10-02 04:13:29 W3SVC1497435083 10.216.12.64 GET /robots.txt - 80 - 18.104.22.168 HTTP/1.1 Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html) - - 404 0 2 1795 246 93
2006-10-02 04:13:30 W3SVC1497435083 10.216.12.64 GET /default.asp - 80 - 22.214.171.124 HTTP/1.1 Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html) - - 200 0 0 10109 269 468
404 = Not found (correct I haven't got one)
200 = OK. So page returned OK.
Can anyone suggest to me why one site has been reasonably read by google and the other has been largely ignored.
Given that the software system is the same on both sites and they have existed similar length of time, have some similar incoming links and a similar page rank, what am I missing?
Yahoo and MSN have both indexed lots of pages on each site for quite a while now, this suggests a factor other than the software (ie. it confirms to me that the links my CMS generates are perfectly acceptable for a search engine).
Any suggestions greatfully recieved, I have tried to be thorough and scientific about this so that we can narrow down the issues.
[edited by: tedster at 6:17 pm (utc) on Oct. 10, 2006]
[edit reason] use example.com [/edit]