|Is there anyway for me to get G to crawl my site map but not index it? |
Placing the following in the site map's <head> should do the trick.
<meta name="robots" content="noindex,follow">
Thanks 4serendipity, will do.
(I hope G replaces my site map with the appropriate page!)
Ah, very interesting!
Some follow up questions on this topic:
1. So this should be on any page used as an index?
2. Does G handle this page differently in any way other than not indexing it but following the links?
3. For example, will it follow more than the 100 links or allow a larger page than 100k?
4. Will it increase the weight of the links?
5. Does it pass PR the same way? I would have to think it would be different since the PR of that page indicates how much PR is passed?
Im a little confused with all this sj stuff. we have moved from number 1 on the serps in www.google.com for widgets uk to about number 5 on page 2 in www-sj. and for widgets on its own we cannot be found on sj even though we are on page 1 in google.com. The results for the particular searches on sj seem to be throwing up results from last year. Is anyone else seeing this?
Just for the record our site name contains the keyword as does most of the anchor text linking to us.
GoogleGuy (or anyone else who might have an idea) is this because most of the data is not in sj yet? Will these results pass over to www? Any help here always appreciated.
www is now showing something entirely different-- it's not SJ and it's not FI and it's different from www2 and www3.
www now showing the same results as:
but www2, www3, www-sj, and www-fi are different.
This is all getting really confusing!
I agree, www is now showing something different. We are suddenly No 1 for about the biggest term you can get for our market (prev. No 7), yet on www2 we kind of disappeared. I guess www is the one that counts, though.
I think a little bit of knowledge can be a dangerous thing in this game. All us amateurs here are looking at things like www-sj.google.com and panicking about the SERP's but we don't know what the hell it all means, and joe surfer is never going to see it anyway.
Next update time, I am not going to look at all these servers. When it changes on www, let me know.
Yep, big changes on www.
In my case the site map has now disappeared and the correct page is being shown (and promoted to page #1).
Looks like www has May 7 freshdates showing. The changes could just be the normal Everflux.
What is the significance of this "www-sj.google.com" I keep hearing about?
As the normal Google updates rollout, they typically start on one datacenter and then migrate to the rest. "www-sj" is just one of the datacenters that Google uses. Check my above post for the list of others. By using specific URLs you can target your query to a specific datacenter and see what results it serves up vs. other arms of Google.
www is definitely showing different results, but i don't see new pages. I had titling changes on my entire site, and in www it is still showing older pages (full site), whereas fi and sj are showing newly crawled pages (but 10% less).