Page is a not externally linkable
shaunm - 12:29 pm on Sep 3, 2012 (gmt 0)
@Shaddows
Thanks buddy!
Could you please explain this to me? It starts accumulating PageRank, and all the other externally defined factors that exist in Google's world
Below is my robots.txt structure. I just put it as it is, only removing the actual URLs.
As for the 'Noindex' command in robots.txt, I know that there is a long debate on this topic. While this is strongly not a directive some claim that this is indirectly respected by Google bots even though not a directive. So I didn't remove the 'Noindex' section which was created by someone else before me.
#--------------------------------
#-- VERSION 08.15.2012.1 --
#-- LAST MODIFIED DATE 08-12-2012 --
User-agent: *
Disallow: /perf/folder/
Disallow: /this-is-a-page/test.aspx
Disallow: /this-is-another-page/example.aspx
User-agent: Googlebot
Noindex: /abc.example.com/folder/
Noindex: /abc.example.com/forum/folder/
Noindex: /abc.example.com/blog/folder/
Sitemap: http://www.example.com/sitemap.xml
Sitemap: http://www.example.com/de/sitemap.xml
#--------------------------------