Welcome to WebmasterWorld Guest from 54.163.35.238

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

shelob v1.0 - No robots.txt

From Juniper Networks

     

jdMorgan

4:41 pm on Jun 13, 2007 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Saw this:

208.223.208.*** - - [13/Jun/2007:11:29:26 -0500] "GET / HTTP/1.0" 403 666 "-" "shelob v1.0"

IP address resolves to a research facility belonging to networking equipment maker Juniper Networks.

Two things their researchers should take note of: The robots.txt standard, and the fact that Shelob was an evil spider... at least according to Tolkien.

For non-compliant spiders, no tasty hobbitses to eat here, only 403s. :(

Jim

[edited by: volatilegx at 11:11 pm (utc) on June 13, 2007]
[edit reason] obfuscated ip address [/edit]

keyplyr

6:54 pm on Jun 15, 2007 (gmt 0)

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



I've had it banned for nearly two years. It musta done something wrong :)

incrediBILL

10:00 pm on Jun 15, 2007 (gmt 0)

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



First contact with it was on 6/2/07 daily thru 6/15/07 so far. It hails from security-lab1.juniper.net which has just encountered my security lab that has been feeding it garbage pages since first encounter.

fiestagirl

7:00 pm on Jun 17, 2007 (gmt 0)

10+ Year Member



According to my records these guys lost their privileges in 4/2006, after scraping with a Python UA.

208.223.208.***
python-urllib/1.16

 

Featured Threads

Hot Threads This Week

Hot Threads This Month