Forum Moderators: open
The above ip addresses have hit my site's cgi-bin about 5000 times over the past month. It appears to have come from Googlebot, at least according to my logs. The ip address is not familiar to me.. Does anybody know if it is really Googlebot?
The reason I ask is that I thought my robots.txt file, which has been the same for months, prevented Google from accessing my cgi-bin. Perhaps my robots.txt file is wrong, or Google is trying to find out more information than it should?
Here it is:
=========================
User-agent: *
Disallow: /cgi-bin/
User-agent: SurveyBot
Disallow: /
User-agent: Googlebot
Disallow: /sitemap.cgi
=========================
