Welcome to WebmasterWorld Guest from 35.172.195.49

Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

How Does Google's Sitemap Indexing Work?.

     
11:37 am on Jun 2, 2011 (gmt 0)

Junior Member

10+ Year Member

joined:June 26, 2008
posts:178
votes: 0


Hey peeps,

Where I work we have multiple websites in multiple languages.

Each website is identical, apart from the language of the website, the contact details of the company, the team member's page and the offers displayed on the home page.

Each website was launched at different times, so there are different amounts of pages in Googles index for each website.

However, the other day we launched three new websites targeted at Romanian, the Netherlands and Denmark.

All three were launched in the same hour. All three were submitted to Google at the same time...

Romania:
/sitemap.php - 29 of 31 submitted
/sitemap_manufacturers.php - 36 of 348 submitted
/sitemap_products.php - 12,302 of 50,000 submitted

Netherlands:
/sitemap.php - 12 of 31 submitted
/sitemap_manufacturers.php - 229 of 348 submitted
/sitemap_products.php - 229 of 50,000 submitted

Denmark:
/sitemap.php - 28 of 31 submitted
/sitemap_manufacturers.php - 14 of 348 submitted
/sitemap_products.php - 16,673 of 50,000 submitted

As you can see, what Google has in the index for each of them is vastly different and I was wondering why?

Does anyone know how Google goes about their indexing and how I could speed up the indexing of our slower websites?

Thanks in advance,
Tom.
6:23 pm on June 2, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 26, 2000
posts:37301
votes: 0


There are definitely some mysteries in how Sitemap indexing works. Part of it is an algorithm that the crawl team uses to allocate their resources.

All that I know you can do is make sure all the URLs in your sitemaps resolve 200 OK - without any redirects. My supposition is that too many problematic URLs means that sitemap might get lower priority.

But again, this is just somewhat educated guesswork from me. As you've noticed, every situation is unique and wide variations are commonplace. And some of that variation is only in how the reporting worsk - URLs sometimes get indexed and not reported well, too.
6:39 pm on June 2, 2011 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member netmeg is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Mar 30, 2005
posts:13012
votes: 222


I think you're probably right, tedster; my own recent experience lines up with that.

I launched two sites last week, both very similar but with slightly different targets. One got picked up immediately, and was ranking and sending traffic in less than 48 hours. The other one - crickets. I went and did some investigating after a day or two, and found a URL issue that needed to be resolved. Fixed it, and still crickets - although at least the error messages from the snafu are starting to show up in GWT now, so I have hopes it'll get figured out eventually. Meanwhile the good site is going gangbusters - indexed urls growing and growing, traffic growing and growing.
7:44 am on June 3, 2011 (gmt 0)

Junior Member

10+ Year Member

joined:June 26, 2008
posts:178
votes: 0


Thanks for the replies, Fellas.

All the pages work fine. Sites that have been up for more than 1-2 months are more or less fully indexed now.

So, what it's a mystery aye? Frustrating haha...
7:37 pm on June 3, 2011 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15941
votes: 889


Conversely: If you replace a sitemap to reflect pages that are renamed, moved or flat-out gone, it may take forever for google to notice the subtractions. (Additions go much faster.) Until a few days ago, my "crawl errors" included an In Sitemap group-- complaining about pages that ceased to exist and were removed from the sitemap at least half a year ago.
8:32 pm on June 3, 2011 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member netmeg is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Mar 30, 2005
posts:13012
votes: 222


Until a few days ago, my "crawl errors" included an In Sitemap group-- complaining about pages that ceased to exist and were removed from the sitemap at least half a year ago.


Yea, me too.