Welcome to WebmasterWorld Guest from 126.96.36.199 , register , free tools , login , search , pro membership , help , library , announcements , recent posts , open posts Become a Pro Member
shelob v1.0 - No robots.txt From Juniper Networks jdMorgan
208.223.208.*** - - [13/Jun/2007:11:29:26 -0500] "GET / HTTP/1.0" 403 666 "-" "shelob v1.0"
IP address resolves to a research facility belonging to networking equipment maker Juniper Networks.
Two things their researchers should take note of: The robots.txt standard, and the fact that Shelob was an
evil spider... at least according to Tolkien.
For non-compliant spiders, no tasty hobbitses to eat here, only 403s. :(
[ edited by: volatilegx at 11:11 pm (utc) on June 13, 2007] [edit reason] obfuscated ip address [/edit]