Welcome to WebmasterWorld Guest from 54.161.157.73

Forum Moderators: DixonJones & mademetop

Message Too Old, No Replies

Unidentified Bots

Any reason not to ban them?

     
6:17 pm on Aug 20, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 1, 2002
posts:774
votes: 0


Hi:

I see a lot of "whatevers" (bots probably, real humans, I doubt) reading my robots.txt... with no user agent.

Any reason not to just write a new rule in and just instantly ban anyone who reads robots.txt with no UA?

7:49 pm on Aug 20, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 1, 2002
posts:774
votes: 0


Sorry- wrong area. Will repost in the correct forum.