Forum Moderators: open

Message Too Old, No Replies

Quick-Crawler

         

keyplyr

7:01 pm on Nov 6, 2017 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month




UA: Quick-Crawler (+https://www.scrapinghub.com/)
Protocol: HTTP/1.1
Robots.txt: No
Host: hetzner.de
136.243.0.0 - 136.243.255.255
136.243.0.0/16

Scraper for hire. Went straight for sitemap.xml.

lucy24

5:39 am on Nov 7, 2017 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Heh. They came by a few days ago, picked up robots.txt--redirected, in fact--waited 8 hours and then came back for the front page. Where they were promptly denied on the grounds of one small header anomaly. I note with interest that they gave the host as EXAMPLE.COM, which is never encouraging.

To nobody's surprise, last month's Contacts-Crawler exhibited identical behavior.

keyplyr

6:13 am on Nov 7, 2017 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



If it walks like a duck...
(no offense to ducks)

keyplyr

7:37 pm on Jun 9, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Also coming from...

Host: servers.com
142.0.192.0 - 142.0.207.255
142.0.192.0/20