homepage Welcome to WebmasterWorld Guest from 54.145.191.14
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Hardware and OS Related Technologies / Website Technology Issues
Forum Library, Charter, Moderators: phranque

Website Technology Issues Forum

    
Unidentified robots.txt access
Any reason not to ban them?
carfac

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3727539 posted 7:51 pm on Aug 20, 2008 (gmt 0)

Hi:

I see a lot of "whatevers" (bots probably, real humans, I doubt) reading my robots.txt... with no user agent.

Any reason not to just write a new rule in and just instantly ban anyone who reads robots.txt with no UA?

 

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 3727539 posted 5:01 am on Aug 21, 2008 (gmt 0)

carfac,
Seem to recall you participating in Forum 11 (Search Engine Sipder ID?)

There are many threads and multiple criteria to both permit and deny blank UA's, however each webmaster must determine what is beneficial or detrimental to their own site (s).

A search [google.com]

One result, your own inquiry from 2002 [webmasterworld.com]

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Hardware and OS Related Technologies / Website Technology Issues
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved