Page is a not externally linkable
httpwebwitch - 7:37 pm on May 4, 2009 (gmt 0)
at 3:30pm Googlebot is still requesting those pages. how long until G takes another look at my robots.txt? Or do I need to prompt it somehow? Hey G-bot, please take a look at my robots.txt, I changed it... please stop crawling... I could put some cloaking rules in my .htaccess, but that seems heavyhanded IMHO Anyone have experiences to share?
I noticed this afternoon that Googlebot is visiting URLs that it shouldn't visit. So I added those URLs to the robots.txt at 2:30pm