Googlebot requesting ./robots.txt and /robots.txt/?

[martin]

What's Googlebot trying to find in /robots.txt/, I know that it *should* look at /robots.txt from time to time but why it requests it with a trailing slash.

The second request is just a second later than the first one and from the same IP.

[Visit Thailand]

I have seen this where it looks for page.htm/robots.txt and also page.htm/

Strange would be interesting to know what people here think.

[jdMorgan]

martin,

You might want to be sure that any redirection you have set up on your server is not inadvertently
redirecting requests for robots.txt. You can test by requesting robots.txt from your own site, and
checking to see if anything weird happens in the address bar (or elsewhere).

Otherwise, this is pretty strange, and hopefully Googleguy will kick this over the the Googlebot
crew for an answer.

Interesting mystery...

Jim

[Hemsell]

Was it actually looking to index your robots.txt becasuse you linked to it?
This websites robots.txt gets indexed as well as "followed"

[216.239.51.100...]

[google.com...]

[ciml]

Martin, what happens when you type yourdomain.com/robots.txt into a browser? Are you redirected to /robots.txt/ ?

Although it seems unlikely to be the reason here, most servers would issue an HTTP redirect from the former to the latter if there is a directory named /robots.txt

Can you see what the HTTP status of your server's response is? It shuold be in the logs.

[martin]

Wow, sorry about that.

This is my wildcard redirect for files in the root directory that are not found - I intented it for requests for directories without a trailing slash.