1. Home
  2. Forums Index
  3. General SEO Issues / Crawler, Spider, and User Agent ID 3:20 pm May 4, 2026

Forum Moderators: open

Message Too Old, No Replies

The Donald web spider user agent

it had to happen.

         

lucy24

11:39 pm on Feb 7, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I don't make this stuff up, y'know... 
69.163.200.125 - - [07/Feb/2016:01:45:57 -0800] "GET / HTTP/1.1" 403 3450 "https://www.donaldjtrump.com/" "The Donald" 
69.163.200.125 - - [07/Feb/2016:01:46:26 -0800] "GET /robots.txt HTTP/1.1" 200 704 "https://www.donaldjtrump.com/" "The Donald" 
69.163.200.125 - - [07/Feb/2016:01:46:30 -0800] "GET /robots.txt HTTP/1.1" 200 704 "https://www.donaldjtrump.com/" "The Donald" 
69.163.200.125 - - [07/Feb/2016:01:46:35 -0800] "GET /robots.txt HTTP/1.1" 200 704 "https://www.donaldjtrump.com/" "The Donald" 
69.163.200.125 - - [07/Feb/2016:01:46:48 -0800] "GET / HTTP/1.1" 403 3450 "https://www.donaldjtrump.com/" "The Donald" 
69.163.200.125 - - [07/Feb/2016:01:47:16 -0800] "GET /robots.txt HTTP/1.1" 200 704 "https://www.donaldjtrump.com/" "The Donald" 
69.163.200.125 - - [07/Feb/2016:01:47:18 -0800] "GET /robots.txt HTTP/1.1" 200 704 "https://www.donaldjtrump.com/" "The Donald" 
69.163.200.125 - - [07/Feb/2016:01:47:22 -0800] "GET /robots.txt HTTP/1.1" 200 704 "https://www.donaldjtrump.com/" "The Donald" 
69.163.200.125 - - [07/Feb/2016:01:47:32 -0800] "GET /index.php HTTP/1.1" 403 3450 "https://www.donaldjtrump.com/" "The Donald" 
69.163.200.125 - - [07/Feb/2016:01:47:45 -0800] "GET /index.html HTTP/1.1" 403 3450 "https://www.donaldjtrump.com/" "The Donald" 
69.163.200.125 - - [07/Feb/2016:01:47:52 -0800] "GET /robots.txt HTTP/1.1" 200 704 "https://www.donaldjtrump.com/" "The Donald" 
69.163.200.125 - - [07/Feb/2016:01:47:58 -0800] "GET /robots.txt HTTP/1.1" 200 704 "https://www.donaldjtrump.com/" "The Donald" 
69.163.200.125 - - [07/Feb/2016:01:48:05 -0800] "GET /robots.txt HTTP/1.1" 200 704 "https://www.donaldjtrump.com/" "The Donald" 
69.163.200.125 - - [07/Feb/2016:01:48:38 -0800] "GET /fun/ HTTP/1.1" 403 3450 "https://www.donaldjtrump.com/" "The Donald" 
69.163.200.125 - - [07/Feb/2016:01:48:44 -0800] "GET /robots.txt HTTP/1.1" 200 704 "https://www.donaldjtrump.com/" "The Donald" 
69.163.200.125 - - [07/Feb/2016:01:48:47 -0800] "GET /robots.txt HTTP/1.1" 200 704 "https://www.donaldjtrump.com/" "The Donald" 
69.163.200.125 - - [07/Feb/2016:01:48:49 -0800] "GET /robots.txt HTTP/1.1" 200 704 "https://www.donaldjtrump.com/" "The Donald" 
69.163.200.125 - - [07/Feb/2016:01:49:01 -0800] "GET /no-fun/ HTTP/1.1" 403 3450 "https://www.donaldjtrump.com/" "The Donald" 
69.163.200.125 - - [07/Feb/2016:01:49:09 -0800] "GET /robots.txt HTTP/1.1" 200 704 "https://www.donaldjtrump.com/" "The Donald" 
69.163.200.125 - - [07/Feb/2016:01:49:11 -0800] "GET /robots.txt HTTP/1.1" 200 704 "https://www.donaldjtrump.com/" "The Donald"
I pasted in the full visit to show the remarkable appetite for robots.txt. Or possibly they were just eager for something-- anything-- that would get them a safe 200.

I do not have any pages called index.php; in fact I only noticed the request because my log-wrangling assumed it was part of the "index.php" botnet and sorted it accordingly. /fun/ is a real directory; /no-fun/ isn't. Bit of a headscratcher, that. The IP, incidentally, is one that I've only just blocked. Whew!

keyplyr

10:59 am on Feb 8, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Likely just log spam where a block is just as good as an OK, probably better since it made its way to a public forum now; wouldn't surprise me with that guy.

Andy Langton

11:03 am on Feb 8, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Same IP as where am I [webmasterworld.com]?

lucy24

4:30 pm on Feb 8, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Yup, that's why I'd only just blocked the IP ;)

It would obviously have been a lot less funny if I'd followed standard procedure and exemplified the referer spam.

aristotle

12:56 am on Feb 10, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



maybe this is part of his election campaign. a way to get free publicity and exposure. Would be even better if he put his campaign slogan in the user agent.

keyplyr

1:29 am on Feb 10, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Would be even better if he put his campaign slogan in the user agent.
eh... no it wouldn't.

tangor

3:19 am on Feb 10, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



This thread might inspire other spider folks to name their bots in other "humorous" ways "political". (Note: I have in place, and have had for some time, denies in htaccess re: political names as a proactive, includes entertainment names as well.)

aristotle

3:05 pm on Feb 10, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



keyplyr -- eh... no it wouldn't.

I just meant that it would make it an even better campaign tactic, in the sense that the full possibilities of the use of the user agent aren't being taken advantage of in the instance that Lucy saw in the logs. In other words, even more promotion could have been done.

keyplyr

9:21 am on Feb 11, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Well in the big scheme of things... how many of USA's 357,178,284 inhabitants (279,834,232 internet users) would actually see this?
statistic: U.S. Census Bureau, November 30, 2015

aristotle

12:34 pm on Feb 11, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Well in the big scheme of things... how many of USA's 357,178,284 inhabitants (279,834,232 internet users) would actually see this?

Obviously only a very small percentage would see it. But since it doesn't cost anything, or very little, it's essentially free advertising.

keyplyr

4:28 am on Feb 13, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



But since it doesn't cost anything...

The cost is yet to be seen.

tangor

4:37 am on Feb 13, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Got to ask... how many crazy webmasters are there that allow their logs to be viewed in the first place? Log spam does NOT work if it can't be seen.

And my proactive denies are to feed 403s instead of content. Less bandwidth!

lucy24

4:46 am on Feb 13, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I just broke down and looked up the domain in the UA. (I didn't actually GO there, fer hevvins sakes.) Free lookup says:
THE {guess what surname} ORGANIZATION is associated with ~3,348 other domains

Holy cow. Three thousand, three hundred and forty-eight? Why?! More fascinatingly, they claim the name was created way back on 1997-01-20.

The IP for the website, or rather six websites, will only admit to "CloudFlare". Haven't there been a couple of threads about them just recently?

keyplyr

5:49 am on Feb 13, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Cloudfare is a colo that routes your site through a dozen servers around the world for efficient redundancy. They also boast an effective firewall. IMO however, they are neither the universal end-all solution nor an evil doer.

tangor

6:12 am on Feb 13, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



But they are (Cloudflare) yet another venue for baddies to party on the side, inside, outside, all sides. Just not quite as popular for that gang as AWS.

keyplyr

6:20 am on Feb 13, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Well you can say that about any server farm since most use an unaccountable cloud platform now. However on the positive side, many of those former bad neighborhoods now use cloud for mobile apps & connectivity because of that efficient dynamic redundancy.

lucy24

7:38 pm on Feb 13, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



<tangent>
efficient dynamic redundancy

Doesn't that sound like something someone in middle management and/or human resources would say? :)
</tangent>

keyplyr

10:38 pm on Feb 13, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



The nearest I've gotten to middle management is watching Mad Men.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

  1. Home
  2. Forums Index
  3. General SEO Issues / Crawler, Spider, and User Agent ID 3:20 pm May 4, 2026