Page is a not externally linkable
-- Google SEO News and Discussion
---- Some help with https vs http
shaunm - 7:04 am on Jul 19, 2013 (gmt 0)Thread source:: http://www.webmasterworld.com/google/4589848.htm
How would you normally block all the https: requests through Robots.txt? Is there a specific syntax for it?
|In my case the HTTPS pages (all within a folder) are blocked via robots.txt and meta robots as well. |
Thanks again :-)
haha yeah, it does never make sense doesn't it? I thought there might be some shopping cart type of pages or any pages that require secure log in in the website I am talking about. But there isn't any. So I can block all the https:// requests right? Can you please explain me how do I start it over?
|it wouldn't be normal if www.example.com was intended to be secure content |
|you would ideally design your url structure so that you can easily distinguish secure and non-secure content and then use mod_rewrite techniques (for apache) or various techniques as your environment requires to make sure all non-canonical request are redirected to the canonical url. |
|there are several recent discussions about robots.txt-excluded urls which appear in the index. |
Interesting! So, it's just not the snippet where Google used to show the URL only version of a blocked content? Is is showing the complete page now? I also got to know from the same forum that if we that page in the sitemap it will get crawled and indexed no matter if we block it in robots.txt or not. Is that also true?
Thanks you all for the overwhelming knowledge! :-)
Brought to you by WebmasterWorld: http://www.webmasterworld.com