homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

Anyone seen this one?

 1:14 pm on May 14, 2003 (gmt 0)

User-Agent string is FTAgent

Just generated several 404s within too short a time frame for a human.

It basically took the file/directory names from one site but the domain name from a different virtualHost. Neither site is the default virtualHost or real IP address, so it's not that it doesn't understand HTTP1.1. I suspect it's having issues with server-side redirection.

I'm going to ban it unless someone can point to it having a legitimate status.



 5:38 pm on May 14, 2003 (gmt 0)

Unless far east traffic is beneficial to your site?
Consider denying the entire IP range. -

To even take it further?
You might consider denying the ENTIRE 61. block with the EXCEPTION of the following ranges; -


 2:14 pm on May 15, 2003 (gmt 0)

I must admit to being very wary of banning IPs, as I feel that's just overkill - effectively saying that everyone in that range is using spambots and the like.

It's an educational site, so will attract some traffic from the far east.

In any case, there are 22 sites on the server and I try to use a global anti-spam config, which wouldn't be possible if I went along the IP-banning route.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved