Convergence - 12:35 am on Jun 30, 2013 (gmt 0)
Has anyone seen the googlebot-- the real thing, not a spoofer-- snuffling around where it doesn't belong?
to which bots are you referring?
We have specific pages that we do not want indexed, IE: Merchant profile pages. The Googlebot shows up in logs all the time as having accessed those pages. These pages are properly blocked in robots.txt and have noindex in the headers. These pages show up in the SERPs from time to time with the "description blocked by robots.txt" statement.
The only way the Googlebot is accessing these URLs is from internal pages as we have yet to add rel=nofollow to them...