Forum Moderators: open

Message Too Old, No Replies

Riddler honors robots.txt

         

keyplyr

3:49 am on May 14, 2014 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month





UA: Riddler (http://riddler.io/about.html)
Robots.txt: yes

Looks like either a distro bot or botnet. So far coming from:

Amazon (server farm)
54.72.0.0 - 54.95.255.255

Amazon (server farm)
54.192.0.0 - 54.255.255.255

Sonera (broadband Pool)
88.195.0.0 - 88.195.255.255

Nianet (server farm)
130.185.128.0 - 130.185.143.255

Hetzner (server farm)
176.9.0.0 - 176.9.255.255

Very aggressive despite eating 403s. 3 days hourly non-stop so far.

keyplyr

6:19 am on May 16, 2014 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Update: Put up a deny in robots.txt and it's now leaving me alone.

Keep forgetting that some bots actually obey robots.txt... who knew?