Welcome to WebmasterWorld Guest from 54.157.7.205

Forum Moderators: Ocean10000 & incrediBILL & keyplyr

Message Too Old, No Replies

shelob v1.0 - No robots.txt

From Juniper Networks

     
4:41 pm on Jun 13, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


Saw this:

208.223.208.*** - - [13/Jun/2007:11:29:26 -0500] "GET / HTTP/1.0" 403 666 "-" "shelob v1.0"

IP address resolves to a research facility belonging to networking equipment maker Juniper Networks.

Two things their researchers should take note of: The robots.txt standard, and the fact that Shelob was an evil spider... at least according to Tolkien.

For non-compliant spiders, no tasty hobbitses to eat here, only 403s. :(

Jim

[edited by: volatilegx at 11:11 pm (utc) on June 13, 2007]
[edit reason] obfuscated ip address [/edit]

6:54 pm on June 15, 2007 (gmt 0)

Moderator This Forum from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:6984
votes: 167


I've had it banned for nearly two years. It musta done something wrong :)
10:00 pm on June 15, 2007 (gmt 0)

Administrator from US 

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 25, 2005
posts:14650
votes: 94


First contact with it was on 6/2/07 daily thru 6/15/07 so far. It hails from security-lab1.juniper.net which has just encountered my security lab that has been feeding it garbage pages since first encounter.
7:00 pm on June 17, 2007 (gmt 0)

Preferred Member

10+ Year Member

joined:May 14, 2002
posts:378
votes: 0


According to my records these guys lost their privileges in 4/2006, after scraping with a Python UA.

208.223.208.***
python-urllib/1.16