---- page is noindexed, but still shows in SERP with a Google notice
phranque - 10:43 am on Jun 30, 2013 (gmt 0)
then in a day or two those pages will be in the SERPs, with aforementioned "description is blocked by robots.txt". Then there will be some sort of data update/refresh and the pages are gone.
you won't see the pages - only the urls. and when you say "the pages are gone" are you sure they aren't filtered out? try adding &filter=0 to the google search url and see if those urls reappear.
i don't think you want googlebot requesting robots.txt first for every resource requested. iirc googlebot caches robots.txt for up to 24 hours. what was the elapsed time for those 147 requests of robots.txt?
Am I missing something here?
the part i see missing is where you have verified that googlebot has actually requested a url in the /merchant/ directory and if so that you checked the IP of the visitor to verify that it is in fact googlebot and not a spoofed user agent.
it has been mentioned numerous times in this thread that the noindex directive is irrelevant when you have excluded googlebot from crawling that url.