GoogleBot is caught in firewall :: bad bad

Googlebot caught in firewall Black Ice

[brockster]

I just noticed the other day that the googlebot was caught up in our firewall because of the type of commands the googlebot threw at the server:

"HTTP_Zero_Hex_Code"
crawl-66-249-64-195.googlebot.com
URL=/%2500&evasions=contains_embedded_null;&server=www.example.org&http-server=Microsoft-IIS/5.0&accessed=yes&code=200

"HTTP_Squid_ACL_Bypass"
crawl-66-249-64-195.googlebot.com
URL=/%2500&evasions=contains_embedded_null;&server=www.example.org&http-server=Microsoft-IIS/5.0&accessed=yes&code=200

These are just two examples, however, this makes me believe that we are keeping our sites from being crawled at the risk of keeping hackers out of our iis/M$.

I of coursed added this IP to our approved list, but I know the bot probably uses multiple IPs. Any suggestions here? We are using Black Ice Server, with the most recent updates.

brockster

[edited by: ciml at 10:21 am (utc) on Dec. 13, 2004]
[edit reason] Examplified [/edit]