Screaming Frog SEO Spider

[keyplyr]

UA: Screaming Frog SEO Spider/7.1
Protocol: HTTP/1.1
Robots.txt: Yes
Host: Various

The SEO Spider is a desktop program you can install locally on PC, Mac or Linux which crawls websites’ links, images, CSS, script and apps to evaluate onsite SEO

screamingfrog.co.uk/seo-spider

Mentioned a few times in other forums, this agent can be used to scrape data from remote locations.

[martinibuster]

The user agent can be changed from Screaming Frog to Donkey Site Banger Gold, which is what I usually do.

[keyplyr]

UA: Donkey Site Banger Gold
Protocol: HTTP/1.1
Robots.txt: No
Host: martinibuster

[martinibuster]

hehe

[lucy24]

Donkey Site Banger Gold, which is what I usually do

Do you find that this increases the robot's chances of getting in? You surprise me.

[martinibuster]

I think it makes a nice easter egg for publishers who look at their logs. Kind of like those origami the cop in Blade Runner dropped to mark his passage.

[lucy24]

I think it makes a nice easter egg for publishers who look at their logs.

I gotta concede it's nicer than the robot from May 2014* whose UA was

RAPE RAPE RAPE RAPE

* I was going to say "last year", but log check reveals that it was further back. Understandably it remains vivid in the memory, what with the 12000-plus requests.

[keyplyr]

I remember that guy. Was he ever caught?