Welcome to WebmasterWorld Guest from 54.226.33.117

Forum Moderators: Ocean10000 & incrediBILL & keyplyr

LittleScraper

     
8:44 pm on Jan 27, 2018 (gmt 0)

Moderator This Forum from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:10646
votes: 630



UA: LittleScraper 0.1
Protocol: HTTP/1.1
Robots.txt: Yes
Host: Google Cloud
35.192.0.0 - 35.207.255.255
35.192.0.0/12
12:37 am on Jan 28, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:14431
votes: 576


With a name like that, what could possibly go wrong?

Robots.txt: Yes
I've seen at least one scraping tool that has a user-configurable option: to honor robots.txt or not. This strikes me as vaguely analogous to a conscientious, morally upright burglar who will only rob a place if the door happens to be unlocked.
12:53 am on Jan 28, 2018 (gmt 0)

Moderator This Forum from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:10646
votes: 630


IMO some UAs request robots.txt to circumvent being blocked.