Page is a not externally linkable
- Google
-- Google SEO News and Discussion
---- Pages are indexed even after blocking in robots.txt
shaunm - 7:07 am on Sep 3, 2012 (gmt 0)
@tedster
Thank you so much!
| Google then constructs a title and snippet for the URL just from references rather than by crawling the page directly. |
|
But it doesn't look like a snippet, it just looks how other pages are displayed in Google's search results. I have have come through this snippet stuff for other websites though.
| And remember to change your robots.txt file so you now ALLOW googlebot to crawl the page. Unless they crawl, they won't ever read the robots meta tag. |
|
Oops, I wouldn't have done that if you hadn't informed me. Thanks a ton!
@not2easy
Thanks buddy!
| If the URL is in your sitemap, the page will be crawled. |
|
Are you sure that even though I may block a web page using noindex meta tags, the page will still be indexed if the URL has been included in the SITEMAP?!? Because I have never heard of this before. Can you give me some references or share your personal experiences? THanks
@aakk9999
Thanks for that mate! I will keep those things in mind as well :)
@tedster
Thanks again, ted :)
@atlrus
Thank you!
@Robert
Thank you for replying on my post and for the provided URL reference :)
Thread source:: http://www.webmasterworld.com/google/4490125.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com