Page is a not externally linkable
- Google
-- Google SEO News and Discussion
---- robots.txt - Google's JohnMu Tweets a tip


g1smd - 6:06 pm on May 30, 2010 (gmt 0)


I'd not expect there to be a refresh of robots.txt data before every such page request, but I do wonder what happens if the page is crawled and then robots.txt is pulled between then and before the page would have showed up in the SERPs. Is the data dumped, or does it make it into the index only to be removed again within hours or days? I have always disallowed the URLs for stuff several days before publishing it.


Thread source:: http://www.webmasterworld.com/google/4143083.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com