How Does Google's Sitemap Indexing Work?.

[Tom_Cash]

Hey peeps,

Where I work we have multiple websites in multiple languages.

Each website is identical, apart from the language of the website, the contact details of the company, the team member's page and the offers displayed on the home page.

Each website was launched at different times, so there are different amounts of pages in Googles index for each website.

However, the other day we launched three new websites targeted at Romanian, the Netherlands and Denmark.

All three were launched in the same hour. All three were submitted to Google at the same time...

Romania:
/sitemap.php - 29 of 31 submitted
/sitemap_manufacturers.php - 36 of 348 submitted
/sitemap_products.php - 12,302 of 50,000 submitted

Netherlands:
/sitemap.php - 12 of 31 submitted
/sitemap_manufacturers.php - 229 of 348 submitted
/sitemap_products.php - 229 of 50,000 submitted

Denmark:
/sitemap.php - 28 of 31 submitted
/sitemap_manufacturers.php - 14 of 348 submitted
/sitemap_products.php - 16,673 of 50,000 submitted

As you can see, what Google has in the index for each of them is vastly different and I was wondering why?

Does anyone know how Google goes about their indexing and how I could speed up the indexing of our slower websites?

Thanks in advance,
Tom.

[tedster]

There are definitely some mysteries in how Sitemap indexing works. Part of it is an algorithm that the crawl team uses to allocate their resources.

All that I know you can do is make sure all the URLs in your sitemaps resolve 200 OK - without any redirects. My supposition is that too many problematic URLs means that sitemap might get lower priority.

But again, this is just somewhat educated guesswork from me. As you've noticed, every situation is unique and wide variations are commonplace. And some of that variation is only in how the reporting worsk - URLs sometimes get indexed and not reported well, too.

[netmeg]

I think you're probably right, tedster; my own recent experience lines up with that.

I launched two sites last week, both very similar but with slightly different targets. One got picked up immediately, and was ranking and sending traffic in less than 48 hours. The other one - crickets. I went and did some investigating after a day or two, and found a URL issue that needed to be resolved. Fixed it, and still crickets - although at least the error messages from the snafu are starting to show up in GWT now, so I have hopes it'll get figured out eventually. Meanwhile the good site is going gangbusters - indexed urls growing and growing, traffic growing and growing.

[Tom_Cash]

Thanks for the replies, Fellas.

All the pages work fine. Sites that have been up for more than 1-2 months are more or less fully indexed now.

So, what it's a mystery aye? Frustrating haha...

[lucy24]

Conversely: If you replace a sitemap to reflect pages that are renamed, moved or flat-out gone, it may take forever for google to notice the subtractions. (Additions go much faster.) Until a few days ago, my "crawl errors" included an In Sitemap group-- complaining about pages that ceased to exist and were removed from the sitemap at least half a year ago.

[netmeg]

Until a few days ago, my "crawl errors" included an In Sitemap group-- complaining about pages that ceased to exist and were removed from the sitemap at least half a year ago.

Yea, me too.