Convergence - 9:56 pm on Jun 30, 2013 (gmt 0)
- You have confirmed via WMT that URLs with pattern /merchant/ are blocked via robots.txt
- However, you have positively identified in your logs (via IP address and user agent) that Googlebot has requested an URL with the pattern /merchant/, i.e. in your logs there was a line something like :
GET /merchant/ with 200 OK, IP address from Googlebot and user agent Googlebot
Are you absolutely sure that this URL was requested by Googlebot and not some other bot from Google (e.g. AdsBot-Google treats robots.txt differently, see note below)
Yes. We also REFUSE to have Adsense on our web properties. That would compete with OUR PPC ad network :)
If so, how odd...
Yes. That's why I posted what I have.
There is an important distinction between crawling and indexing. Robots.txt controls crawling, but not indexing
However, saw plain as day, the 200 header response in the logs.
It what it is, lol...