Okay i have changed my robots.txt file to disallow robots from indexing my sitemap.
It looks like this:
Can search engines still scan this file for all my links if i changed my robots.txt file to disallow it. (from indexing)---i still want them to scan it however.
Thank you for any comments...
If you disallow the site map page in a robots.txt, the spiders won't visit the page at all. If you want the spiders to visit the page and index the links, but not index the page itself, them you need remove the robots.txt ban and use a robots meta tag in the head of the site map page instead:
<meta name="robots" content="noindex,follow">
Always use a sitemap.
The most reasonable explanation for the low PR on the new pages is that they're new.
Sitemaps are the best way to get all of your pages indexed with the fewest hops from your root url.
In addition, the PR will probably go up once the rest of your site is re-indexed and the backlinks are recounted for the new pages. The sitemap most likely one of the only backlinks being counted to the new pages since other internal links may be further down in your website structure.
Yes i think i will allow them to follow the sitemap, but not index it.
My sitemap is getting higher rankings that the actual pages.
Does anyone think otherwise?
Has indexing sitemaps cause anyone else probs., similar to this?
Oy, this is the sort of thing that causes urban myths. I was tempted to post saying, 'yeah, me too,' but the voice on my left shoulder wouldn't let me do it. ;-)
Maybe they've got it indexed but not fully valued yet. Maybe they're just taking their sweet time. Maybe the floods affected their algo. Maybe the pages are not very valuable. Maybe they've instituted G-like aging factors. I dunno.
Whatever it is, site maps are a good thing.
<meta name="robots" content="noindex,follow">
I understand the "noindex" if you don't want the spiders to index / cache etc. the sitemap file,
but what is the "nofollow" for?
Isn't "nofollow" a directive to NOT follow and index the links listed from sitemap?
I would think that the guy wants noindex alone.
Do I have this all wrong? What precisely is the meaning of "nofollow"? - Larry
If you read the code, it says "noindex,follow". Nowhere does it say "nofollow". So.. in essence, you're right, but it doesn't apply here.
Sitemap = Table of Contents
Sitemap = Index of Products
Sitemap = Index of Services
Sitemap = Index of Locations
Sitemap = Index of ...
No, a site map will not harm your rankings. Site maps and/or index pages are an integral part of any well planned site. They are the glue that holds everything together.
For larger sites, well organized and segmented site maps are an added benefit. Think of your site maps as index pages for each specific area of your site. Break them down to their smallest common denominator. Then, link to those smaller entities from the main site map. Don't try to stuff everything on one page, it loses its value. Break them down and focus each site map.
So are you saying that telling the robots to follow the sitemap but not index it, wouldn't make a difference?
Or should i test it out and see if my rankins for those pages get better.
It is really weird...I have # 1 rankings for alot of keyphrases.
But i am finding on some pages (pages where i have spent hours optimizing) that the sitemap page gets a better ranking that the page. I couldn't even find some my pages in the first 10 pages. But the sitemap is on the 1st page on some of them.
I run about 20+ domains and they all have sitemaps. So far 90% of them have high rankings, so I would not block sitemaps all together.
As a matter of fact I have higher rankings for my sitemaps in Google than in yahoo, etc.
So try it out and see if it works for you.
|But i am finding on some pages (pages where i have spent hours optimizing) that the sitemap page gets a better ranking that the page. I couldn't even find some my pages in the first 10 pages. But the sitemap is on the 1st page on some of them. |
That is because your new pages have not yet migrated to the search results, but your site map is considered an authority page for your site, so is showing up because it contains links to your new pages.
Wait for an update and then check again.
pageoneresults ... excellent post and very good advice! I intend to change my site map tomorrow using your formula. Many thanks! ;)
I think with most of this stuff patience is the best bet. Check a few times over a period of weeks before panicking, some things take time to settle down and every now and then all SEs have a wobble :)
I see no reason why a site map page could hurt your site - unless of course you're using black hat methods. If your site is being penalized for something it's probably for something other than your site map.