Welcome to WebmasterWorld Guest from 54.198.55.167

Forum Moderators: Ocean10000 & keyplyr

Message Too Old, No Replies

LittleScraper

     
8:44 pm on Jan 27, 2018 (gmt 0)

Moderator This Forum from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12765
votes: 873



UA: LittleScraper 0.1
Protocol: HTTP/1.1
Robots.txt: Yes
Host: Google Cloud
35.192.0.0 - 35.207.255.255
35.192.0.0/12
12:37 am on Jan 28, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15173
votes: 679


With a name like that, what could possibly go wrong?

Robots.txt: Yes
I've seen at least one scraping tool that has a user-configurable option: to honor robots.txt or not. This strikes me as vaguely analogous to a conscientious, morally upright burglar who will only rob a place if the door happens to be unlocked.
12:53 am on Jan 28, 2018 (gmt 0)

Moderator This Forum from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12765
votes: 873


IMO some UAs request robots.txt to circumvent being blocked.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members