Welcome to WebmasterWorld Guest from 54.198.222.129

Forum Moderators: DixonJones & mademetop

Message Too Old, No Replies

Unidentified Bots

Any reason not to ban them?

     
6:17 pm on Aug 20, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Hi:

I see a lot of "whatevers" (bots probably, real humans, I doubt) reading my robots.txt... with no user agent.

Any reason not to just write a new rule in and just instantly ban anyone who reads robots.txt with no UA?

7:49 pm on Aug 20, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Sorry- wrong area. Will repost in the correct forum.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month