homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

odd behavior

 7:45 pm on Nov 6, 2012 (gmt 0)

no referrer, hitting base directory structures (our site doesn't allow that or link that way.




 9:33 pm on Nov 6, 2012 (gmt 0)

There was some rather recent discussion on ViaWest, although I was unable to locate it.


 10:29 pm on Nov 6, 2012 (gmt 0)

Block -


 10:32 pm on Nov 6, 2012 (gmt 0)

I've got them flagged as "colo and servers" which generally translates as "shoot to kill" ;)

And it looks like I've met your identical robot. Wearing the same clothes--

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.215 Safari/535.1

-- and living in the same crack house, down to the last 101.

base directory structures

On its last visit, almost all attempts got logged in the form

/directory 301
/directory 403

which translates as:

1. wrong form of domain name redirected by host
2. directory name, still missing final slash, meets my htaccess before mod_dir gets a chance to do its stuff

It tried to fool me by requesting one named file followed by /errorstyles.css which is usually the mark of a wrongly excluded human. (Also Yahoo! Slurp and the plainclothes bingbot, but those are auto-ignored.) Yah. Nice try.


 6:34 am on Nov 7, 2012 (gmt 0)

Yea, I had a rash of them since setting up our new server. Hadn't had the time to go back through my ipblock list to reverify ranges so we were a bit wide open. I got some really interesting logs if anyone really wants to see the mess bots can do on a 100k+ page ecomm site.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved