homepage Welcome to WebmasterWorld Guest from 54.197.65.82
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Robot from Cisco Systems?
msgraph

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 275 posted 2:21 pm on Jan 10, 2001 (gmt 0)

Has anyone noticed this robot on any of their sites?

UA: CE
IP: 171.69.236.219

It's running off of Cisco Systems, Inc.

First it went for my robots.txt file, then my root page, then tried to access another domain within this site.

171.69.236.219 - - "GET / HTTP/1.0" 200 robots.txt HTTP/1.0" 302 214 "-" "CE"

171.69.236.219 - - "GET / HTTP/1.0" 200 "-" "CE" "-"

171.69.236.219 - - "GET / HTTP/1.0" 200 - "http://www.domain.com:80/" "CE" "-"

 

Cisco

10+ Year Member



 
Msg#: 275 posted 3:21 pm on Jan 10, 2001 (gmt 0)

I have noticed this on and off for some time now. I have not been overly concerned because it hit random domains on a sporadic basis.

I run a robot alert cgi on my logs every half hour that looks for certain user agents as well as any hits to the robots.txt file. Then the script compares the ip against a know robot ip list. It will email me if anything is amiss. Today, 171.69.236.219 hit every domain I own (lots). Now I am concerned.

I can't find anything about this ip other than it is owned by Cisco Systems.

msgraph

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 275 posted 3:30 pm on Jan 10, 2001 (gmt 0)

Have the same type of thing set up on our systems. Works great for catching people trying to steal your pages or new search engine robots!

Since a large part of the net is running off Cisco equipment, I imagine they are just running server/connection integrity checks. If they are just doing small checks I'll leave them be, but if they get out of control they're going to get tossed in the banned user agent pile.

BoneHeadicus

10+ Year Member



 
Msg#: 275 posted 5:36 pm on Jan 10, 2001 (gmt 0)

Cisco,
That sounds like something I might be interested in...is there a link to that script?

Cisco

10+ Year Member



 
Msg#: 275 posted 5:54 pm on Jan 10, 2001 (gmt 0)

It's something that I wrote for my own use after missing an entire Google spidering blitz because I wasn't paying attention. Needless to say, it took me awhile to recover.

Sorry

BoneHeadicus

10+ Year Member



 
Msg#: 275 posted 6:07 pm on Jan 10, 2001 (gmt 0)

You know whats really cool is I understand what has to take place in that type of situation..I just dont know how to code it out from scratch....I'm getting good at hacking stuff tho'.

Thanks anyway.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved