Many URL Listings in site: search

Will the pages fully get into the index?

[MarkJH]

If I do a search thus:

site:www.mydomain.com inurl:www.mydomain.com/subfolder

I get thousands of results back with no title or description and just the link 'Similar pages' underneath.

Are these pages set be indexed, are they blacklisted or what?

[MarkJH]

Nobody?

I'm just curious as to why Google has these pages 'somewhere' in it's database.

[Lorel]

No title and no description can result from several factors:

- it's a dynamic site (change to a static url)
- because of robots.txt or meta tags excluding Google or all engines, (i.e., noindex/nofollow in meta tag).
- could be a double listing www.domain.com and domain.com

I found this on 3 of my pages and the only thing I could find wrong was they weren't validated.

anyone else have anything to add to this list?

[MarkJH]

- it's a dynamic site (change to a static url)

The pages in question are all static URL's.

- because of robots.txt or meta tags excluding Google or all engines, (i.e., noindex/nofollow in meta tag)

Google is not disallowed in the robots.txt and all of the pages are index, follow.

- could be a double listing www.domain.com and domain.com

Not seeing any signs of this happening either.

[abates]

Mark: Are you using the noarchive tag on the pages in question? I've noticed Google seems to have trouble of this nature if you tell it not to cache the pages.

[ken_b]

MarkJH;

I have a similar issue on a five year old site. You can read about it in this thread [webmasterworld.com].

Some of the advice I've gotten might be shed some light for you.

[skipfactor]

Checked for duplicate content?

[steveb]

A couple dozen threads like this. Make sure your internal linking is consistent.

[MarkJH]

> Mark: Are you using the noarchive tag on the pages in question?

No, I'm not using that tag.

> I have a similar issue on a five year old site. You can read about it in this thread.

Thing is, all of the pages that it showing without title, description or cache are actual pages. They all exist on my site.

> Checked for duplicate content?

These are template created pages but they all use different keywords and the content is different. About 1/8th of these pages are indexed correctly whilst the others appear as described.

> Make sure your internal linking is consistent.

The site has (IMHO) very good internal linking. It's about as good as I can get it without using an 8,000 link strong sitemap!

I'm at a loss.

[steveb]

My three minutes of detective work for the day... seems like the site is offline, but good chance its the the linking as some links shown in the cache go to /index.asp instead of just /

[MarkJH]

> seems like the site is offline

Not sure why you are thinking that. Unless you discovered the site (if you have the right one!) during a server reboot, or something.

> but good chance its the the linking as some links shown in the cache go to /index.asp instead of just /

How would this stop the pages from being fully indexed?

[MarkJH]

There is only one link in the site's footer that points to /index.asp, I've now changed it to just / and see if that helps. If anyone can explain how this would help indexing, I'd love to know.

[TrumanTiger]

I had a similar situation on one site and, after weeks/months of frustration, got it fixed. I can only guess we had some kind of inadvertent "overoptimization" and/or “duplicate content” penalties.

I have no idea which of the following fixed it, or if the bug simply ran its course and the problem would have been corrected without me touching the site:

--Changed page titles on the affected pages to unoptimized, dull, virtual nothingness.

--Asked a couple vendors with nice, natural (but keyword rich) hyperlinks to my site to just change them to "click here" or to just list the domain. (Imagine, asking someone who had linked to you unsolicited to change their link!)

--Changed the names of directories and files that contained keywords (again, these were naturally occurring names that related to the content of our site; we changed them nonetheless).

--Expanded our sitemap to include links to every page (previously had gone to just the top couple levels).

--Kept the new pages fresh and uploaded on a regular basis.

--Changed the way some product URLs redirected to the site (from meta-refresh to a change at the DNS level). As we built our site (consisting of 4 products) over 24 months, we had built one product at a time on separate URLs. When finished, we moved the content to a single URL and re-directed the 4 others with a meta-refresh. We thought we could have created a "duplicate content" error because the content on the new, single site had once been indexed on the 4 separate content site URLs which still existed (but with no content, just meta-refresh).

Again, I have no idea what worked. We just did it all because we couldn't sit around with a partially indexed site any longer.