Forum Moderators: Robert Charlton & goodroi
I noticed that there are about 8 pages that are URL only. I've never seen this happen before. I did a standard Google search for my "green widgets" category page by searching for [green widgets]. It used to appear somewhere on page 1 but it was no longer there. I clicked to page 2 and I found my "history of green widgets" page sitting at the top of page 2. The "history of green widgets" page is the only subcategory page that is still listed as non-URL only. I checked my logs and Google has yet to crawl this page.
The site itself is a small site (about 80 pages). It's been going in various guises for about 10 years and therefore has a fair amount of inbound links (mainly to internal pages rather than the home page). For most categories it gets to page 1 of the SERPs for two work keyphrases. I gave it a design overhaul about a year back to convert it from tag soup to clean valid HTML and CSS. I changed the linking structure and the URLs using 301s. I also improved the prominence of Adsense ads (that I'd been trailing for about a year previous) It weathered the change fine (some previous number 1 results dropped down a few places but nothing major).
There have been no major changes since then. I've added a few categories. It's all been fine, every page has been indexed in a timely fashion.
However a week ago, I signed up to Google Sitemaps. I am wondering if this is connected in any way with the URL-only pages that I'm starting to see.
Does anyone have any thoughts or ideas?
Now my other site that has been around for a year, I used google site maps as well and I have been seeing the URL only for it. So I am kind of mixed on it. Googlebot is visiting it regularly but it is not wanting to index quickly.
Ironic how a new website can index completely in a week yet my old site is having some issues still.
I have 4 sites in site maps, some old some new and its all same old same old really.
No better, no worse for me - even crawling looks the same in my experience.
I think your problem would lay elsewhere, not because of sitemaps - just my 2c
If anything Google might be trying to force people into Site Maps, another one of their snoop tools, with poor indexing.
The spider still has to pick up the title, content and elements from the page to display. How it gets there is of no concern really with how it displays your information except in the case of hijacking.
There are no URL only results but about 20 sub pages are missing from the index completely.
Try 64.233.179.104 and see if results are any different.
64.233.179.104 shows the same number of pages as 216.239.37.104
Do the missing pages have the same template design, and similiar content?
The content is very different on each page.
Each document starts off in life as an XML document and is transformed using XSL, so each one shares a similar HTML structure, but I must stress that the design is very lightweight.
It's all CSS and DIVs, the pages don't even have a navigation bar (all navigation on these pages is via a breadcrumb trail). The only repetition baring a 3 word footer is in the placement of the HTML tags (H1, DIV, P etc.) There's no visible content that is repeated.
Judging by people's responses, it's clear that it's not the sitemaps that are at fault.
One thing I can think of. I changed the XSL templates slightly recently. Because of this, every document in each class of page (category, subcategor and article) now has exaxtly the same last modified time.
I'm wondering if this might set off alarm bells with Google as automated spam would show a similar characteristing (although each page on my site is hand written, I don't even have a CMS)
Does anyone agree or disagree that identical last modified dates could cause a problem?
are sitemaps necessary to rank in google? Please advise.
No, they're not. Please don't hijack other people's threads.
Yes, Googlebot is visiting as normal. It visited the pages that disappeared and it got a 200 status code from those pages. All pages on the site are valid HTML4.01 Transitional.
MrMister it sounds like you're already doing pretty well with a small Adsense site. Few if any rank well in areas I frequent and they are competitive areas. My url info came back today after a two week hiatus. Could change tomorrow with an engine that seems to be bursting at the seams. Give it some time. Sometimes you can set off a filter when you haven't touched a site in a while.
However a week ago, I signed up to Google Sitemaps. I am wondering if this is connected in any way with the URL-only pages that I'm starting to see.
MrMister: I don't really know what the problem is, but there have been other posts referring to the same scenario over the last few months. So, maybe it's not the sitemap that did it, but maybe it was. I've wondered if it's a canonical issue when I read the other ones. Sorry that this post is relatively useless, but I thought I'd chip in and add that it may be the sitemap (it may not be too).
When Google takes the sitemap into consideration it probably compares the new understanding of your site structure with the one it had before the site map. Two cases are possible.
1. Sitemap adds nothing to the old Google understanding of the site structure. Than nothing happens, SERP is not recalculated etc. It is actually the most typical case probably.
2. Sitemaps adds something new to the previous Google understanding of your site structure. In this case SERP is recalculated.
Whether result is positive or negative depends on your site, Google algo but not on the sitemap.
Sitemap just opens the door.
Vadim.
