rdfbot drops "Nutch"

Forum Moderators: open

Message Too Old, No Replies

rdfbot drops "Nutch"

UA Change

caribguy

7:55 pm on Nov 28, 2008 (gmt 0)

67.202.58.nn - - [27/Nov/2008:21:09:19 -0600] "GET /robots.txt HTTP/1.0" 403 284 "-" "rdfbot/Nutch-1.0-dev"
75.101.236.nnn - - [27/Nov/2008:22:50:41 -0600] "GET /robots.txt HTTP/1.0" 403 284 "-" "rdfbot/Nutch-1.0-dev"
75.101.195.nnn - - [28/Nov/2008:02:08:42 -0600] "GET /robots.txt HTTP/1.0" 403 284 "-" "rdfbot/Nutch-1.0-dev"

75.101.195.nnn - - [28/Nov/2008:02:20:48 -0600] "GET /robots.txt HTTP/1.0" 403 284 "-" "rdfbot/1.0 (rdfbot-support@rediffmailpro.com)"
67.202.58.nn - - [28/Nov/2008:09:37:41 -0600] "GET /robots.txt HTTP/1.0" 403 284 "-" "rdfbot/1.0 (rdfbot-support@rediffmailpro.com)"

Edit, added more ranges

Mokita

9:48 pm on Nov 28, 2008 (gmt 0)

This bot also comes from 202.137.236.nn which belongs to Rediff.com India

keyplyr

9:46 am on Nov 30, 2008 (gmt 0)

So far the new rdfbot has obeyed robots.txt.