Welcome to WebmasterWorld Guest from

Forum Moderators: phranque

Message Too Old, No Replies

Unidentified robots.txt access

Any reason not to ban them?

7:51 pm on Aug 20, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 1, 2002
votes: 0


I see a lot of "whatevers" (bots probably, real humans, I doubt) reading my robots.txt... with no user agent.

Any reason not to just write a new rule in and just instantly ban anyone who reads robots.txt with no UA?

5:01 am on Aug 21, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2001
votes: 3

Seem to recall you participating in Forum 11 (Search Engine Sipder ID?)

There are many threads and multiple criteria to both permit and deny blank UA's, however each webmaster must determine what is beneficial or detrimental to their own site (s).

A search [google.com]

One result, your own inquiry from 2002 [webmasterworld.com]


Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members