homepage Welcome to WebmasterWorld Guest from 54.198.33.96
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
FTAgent
Anyone seen this one?
mvl22




msg:397645
 1:14 pm on May 14, 2003 (gmt 0)

User-Agent string is FTAgent
From 61.52.74.49

Just generated several 404s within too short a time frame for a human.

It basically took the file/directory names from one site but the domain name from a different virtualHost. Neither site is the default virtualHost or real IP address, so it's not that it doesn't understand HTTP1.1. I suspect it's having issues with server-side redirection.

I'm going to ban it unless someone can point to it having a legitimate status.

 

wilderness




msg:397646
 5:38 pm on May 14, 2003 (gmt 0)

mvl22,
Unless far east traffic is beneficial to your site?
Consider denying the entire IP range.
61.52.0.0 - 61.54.255.255

To even take it further?
You might consider denying the ENTIRE 61. block with the EXCEPTION of the following ranges;
61.8.0.0 - 61.8.31.255

mvl22




msg:397647
 2:14 pm on May 15, 2003 (gmt 0)

I must admit to being very wary of banning IPs, as I feel that's just overkill - effectively saying that everyone in that range is using spambots and the like.

It's an educational site, so will attract some traffic from the far east.

In any case, there are 22 sites on the server and I try to use a global anti-spam config, which wouldn't be possible if I went along the IP-banning route.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved