homepage Welcome to WebmasterWorld Guest from 54.204.94.228
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
shell.com
Pfui

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 4383033 posted 1:40 pm on Nov 3, 2011 (gmt 0)

Last month, this Host/IP behaved itself after requesting robots.txt where it's Disallowed. This month? equested, and ignored:

134-163-253-248.shell.com [projecthoneypot.org...]
Goldfire Server

02:34:38 /robots.txt
02:34:57 /

134.163.253.248 is a.k.a. irongrove.net [robtex.com...]

"Goldfire Server" is not a completely uncommon UA (although I reckon it's wrongly configured when it's the UA). So why is shell.com spidering, I wonder? SERPs from 2000-2005 suggest it was an employee-accessed server, or some such. [g6ftpserver.com...] Now? Dunno. Anyone else seeing this?

 

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4383033 posted 7:23 pm on Nov 3, 2011 (gmt 0)

Pretty much the same experience, however unsure about the range. I'd see Goldfire Server once every several weeks. Behaved and of no concern. Then last week it ignored robots.txt and crawled a few pages. Now blocked by UA.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved