| Welcome to WebmasterWorld Guest from 220.127.116.11 |
register, login, search, subscribe, help, library, PubCon, announcements, recent posts, open posts,
|Subscribe and Support WebmasterWorld|
Any reason not to ban them?
| 6:17 pm on Aug 20, 2008 (gmt 0)|
I see a lot of "whatevers" (bots probably, real humans, I doubt) reading my robots.txt... with no user agent.
Any reason not to just write a new rule in and just instantly ban anyone who reads robots.txt with no UA?
| 7:49 pm on Aug 20, 2008 (gmt 0)|
Sorry- wrong area. Will repost in the correct forum.
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld ® and PubCon ® are a Registered Trademarks of Pubcon Inc.
© Pubcon Inc. 1996-2012 all rights reserved