Welcome to WebmasterWorld Guest from

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies




1:40 pm on Nov 3, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Last month, this Host/IP behaved itself after requesting robots.txt where it's Disallowed. This month? equested, and ignored:

134-163-253-248.shell.com [projecthoneypot.org...]
Goldfire Server

02:34:38 /robots.txt
02:34:57 / is a.k.a. irongrove.net [robtex.com...]

"Goldfire Server" is not a completely uncommon UA (although I reckon it's wrongly configured when it's the UA). So why is shell.com spidering, I wonder? SERPs from 2000-2005 suggest it was an employee-accessed server, or some such. [g6ftpserver.com...] Now? Dunno. Anyone else seeing this?


7:23 pm on Nov 3, 2011 (gmt 0)

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Pretty much the same experience, however unsure about the range. I'd see Goldfire Server once every several weeks. Behaved and of no concern. Then last week it ignored robots.txt and crawled a few pages. Now blocked by UA.

Featured Threads

Hot Threads This Week

Hot Threads This Month