Reasons that an IP requests Robots.txt

[palmpal]

Hello,

What are some reasons that an IP would request only my robots.txt file?

My logs show an entry from a company that offers Internet services to their clients. The only thing they viewed was my robots.txt file. There was no referrer information. Should I ban them?

Thanks

[Dreamquick]

There are quite a few reasons to request a robots.txt file from another site, banning someone just because they looked at it seems a little harsh.

It could just be professional curiousity where they want to see what others are doing / how they they are doing it - I suspect most people here have read the robots.txt for at least one website recently or have pulled their competitors to see if they are doing anything "interesting".

Equally there are those who see robots.txt as a magnet for hackers, spammers etc. because if it's approached badly your exclusion file can make for juicy reading material that hands out a list of non-public pages to a potential attacker.

-Tony