| Welcome to WebmasterWorld Guest from 18.104.22.168 |
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
|Pubcon Platinum Sponsor|
|Unidentified robots.txt access|
Any reason not to ban them?
| 7:51 pm on Aug 20, 2008 (gmt 0)|
I see a lot of "whatevers" (bots probably, real humans, I doubt) reading my robots.txt... with no user agent.
Any reason not to just write a new rule in and just instantly ban anyone who reads robots.txt with no UA?
| 5:01 am on Aug 21, 2008 (gmt 0)|
Seem to recall you participating in Forum 11 (Search Engine Sipder ID?)
There are many threads and multiple criteria to both permit and deny blank UA's, however each webmaster must determine what is beneficial or detrimental to their own site (s).
A search [google.com]
One result, your own inquiry from 2002 [webmasterworld.com]
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
© Webmaster World 1996-2013 all rights reserved