Forum Moderators: open

Message Too Old, No Replies

Screaming Frog SEO Spider

         

keyplyr

8:20 pm on Dec 21, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



UA: Screaming Frog SEO Spider/7.1
Protocol: HTTP/1.1
Robots.txt: Yes
Host: Various
The SEO Spider is a desktop program you can install locally on PC, Mac or Linux which crawls websites’ links, images, CSS, script and apps to evaluate onsite SEO
screamingfrog.co.uk/seo-spider

Mentioned a few times in other forums, this agent can be used to scrape data from remote locations.

martinibuster

11:23 pm on Dec 21, 2016 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



The user agent can be changed from Screaming Frog to Donkey Site Banger Gold, which is what I usually do.

keyplyr

11:25 pm on Dec 21, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month




UA: Donkey Site Banger Gold
Protocol: HTTP/1.1
Robots.txt: No
Host: martinibuster

martinibuster

11:27 pm on Dec 21, 2016 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



hehe

lucy24

4:56 am on Dec 22, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Donkey Site Banger Gold, which is what I usually do

Do you find that this increases the robot's chances of getting in? You surprise me.

martinibuster

2:38 pm on Dec 22, 2016 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



I think it makes a nice easter egg for publishers who look at their logs. Kind of like those origami the cop in Blade Runner dropped to mark his passage.

lucy24

6:36 pm on Dec 22, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I think it makes a nice easter egg for publishers who look at their logs.

I gotta concede it's nicer than the robot from May 2014* whose UA was
RAPE RAPE RAPE RAPE


* I was going to say "last year", but log check reveals that it was further back. Understandably it remains vivid in the memory, what with the 12000-plus requests.

keyplyr

9:50 pm on Dec 22, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I remember that guy. Was he ever caught?