I have noticed googlebot visiting the robots.txt file and then creating a fantasy URL similar to /kjakuknaaldfu.html
Did anyone see something like this?
tedster
11:11 pm on May 26, 2008 (gmt 0)
One reason is that Google likes to check your server's handling of 404 errors. If there is a problem, such as not returning a 404 http status in the server header, you'll get a note in your Webmaster Tools account.
g1smd
7:38 pm on May 27, 2008 (gmt 0)
I see them ask for a filename starting with /noexist_... sometimes too.