Forum Moderators: open

Message Too Old, No Replies

Donuts Content Explorer

My website was visited by the Donuts Content Explorer.

         

w3bmastine

1:42 pm on Feb 1, 2016 (gmt 0)

10+ Year Member



Found a Donuts Crawler in my logs on a website with a newTLD. It is not whitelisted in my robots.txt file (403 by UA Go-http-client to enforce robots.txt), yet it keeps sniffing around:

UA: Donuts Content Explorer (www.donuts.domains)
Protocol: HTTP/1.1
Robots.txt: No
Host: AWS
IP: 52.34.255.46

52.34.255.46 - - [01/Feb/2016:12:00:04 +0100] "GET / HTTP/1.1" 301 224 "-" "Donuts Content Explorer (www.donuts.domains)" "www.example.com"
52.34.255.46 - - [01/Feb/2016:12:00:04 +0100] "GET / HTTP/1.1" 403 2934 "http://www.example.com/" "Go-http-client/1.1" "example.com"
52.34.255.46 - - [01/Feb/2016:12:00:05 +0100] "GET /robots.txt HTTP/1.1" 301 234 "-" "Donuts Content Explorer (www.donuts.domains)" "www.example.com"
52.34.255.46 - - [01/Feb/2016:12:00:05 +0100] "GET /robots.txt HTTP/1.1" 200 334 "http://www.example.com/robots.txt" "Go-http-client/1.1" "example.com"
52.34.255.46 - - [01/Feb/2016:12:00:06 +0100] "GET /sitemap.xml HTTP/1.1" 301 235 "-" "Donuts Content Explorer (www.donuts.domains)" "www.example.com"
52.34.255.46 - - [01/Feb/2016:12:00:06 +0100] "GET /sitemap.xml HTTP/1.1" 403 1007 "http://www.example.com/sitemap.xml" "Go-http-client/1.1" "example.com"

keyplyr

10:51 am on Mar 7, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I have yet to see this UA. A "crawler" is also a type of "donut" so there goes any relevant SERP :(

lucy24

4:25 pm on Mar 7, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Where do you draw the line between legitimate UA-string information and, er, User-Agent spam?

That was a rhetorical question.

The first good thing about 52.aa.bb.cc is that it is not 54.aa.bb.cc. The second good thing is ... uhm ... wait, I'm thinking ...

keyplyr

1:29 am on Mar 8, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Where do you draw the line between legitimate UA-string information and, er, User-Agent spam?
I'm convinced some bots are created just to drag the propriety UA string as a means to SPAM.