Forum Moderators: open
-The pages that got indexed before the sitemap all rank high.
However- pages that got indexed after the site map--all rank extremely low.
Should i disallow robots from indexing my sitemap?
I added a sitemap to this site so it would be faster and easier to index my pages.
***If i set my robots.txt file to disallow my sitemap, can the robots still scan all of its links?
I ask this because i know duplicate keywords from the same domain results in one of the pages to be penalized.
In this case i would rather have the sitemap being penalized instead of the actual page i want customers to click on.
Keep in mind that all pages in the website contain all different targetted keywords. The only duplicate content is links to all my pages on my sitemap.
It looks like this:
User-Agent: *
Disallow: /sitemap.htm
My question:
Can search engines still scan this file for all my links if i changed my robots.txt file to disallow it. (from indexing)---i still want them to scan it however.
Thank you for any comments...
<meta name="robots" content="noindex,follow">
The most reasonable explanation for the low PR on the new pages is that they're new.
Sitemaps are the best way to get all of your pages indexed with the fewest hops from your root url.
In addition, the PR will probably go up once the rest of your site is re-indexed and the backlinks are recounted for the new pages. The sitemap most likely one of the only backlinks being counted to the new pages since other internal links may be further down in your website structure.
My sitemap is getting higher rankings that the actual pages.
Does anyone think otherwise?
Has indexing sitemaps cause anyone else probs., similar to this?
Maybe they've got it indexed but not fully valued yet. Maybe they're just taking their sweet time. Maybe the floods affected their algo. Maybe the pages are not very valuable. Maybe they've instituted G-like aging factors. I dunno.
Whatever it is, site maps are a good thing.
I understand the "noindex" if you don't want the spiders to index / cache etc. the sitemap file,
but what is the "nofollow" for?
Isn't "nofollow" a directive to NOT follow and index the links listed from sitemap?
I would think that the guy wants noindex alone.
Do I have this all wrong? What precisely is the meaning of "nofollow"? - Larry
No, a site map will not harm your rankings. Site maps and/or index pages are an integral part of any well planned site. They are the glue that holds everything together.
For larger sites, well organized and segmented site maps are an added benefit. Think of your site maps as index pages for each specific area of your site. Break them down to their smallest common denominator. Then, link to those smaller entities from the main site map. Don't try to stuff everything on one page, it loses its value. Break them down and focus each site map.
Or should i test it out and see if my rankins for those pages get better.
It is really weird...I have # 1 rankings for alot of keyphrases.
But i am finding on some pages (pages where i have spent hours optimizing) that the sitemap page gets a better ranking that the page. I couldn't even find some my pages in the first 10 pages. But the sitemap is on the 1st page on some of them.
As a matter of fact I have higher rankings for my sitemaps in Google than in yahoo, etc.
So try it out and see if it works for you.
But i am finding on some pages (pages where i have spent hours optimizing) that the sitemap page gets a better ranking that the page. I couldn't even find some my pages in the first 10 pages. But the sitemap is on the 1st page on some of them.
That is because your new pages have not yet migrated to the search results, but your site map is considered an authority page for your site, so is showing up because it contains links to your new pages.
Wait for an update and then check again.
pageoneresults ... excellent post and very good advice! I intend to change my site map tomorrow using your formula. Many thanks! ;)
