Forum Moderators: open

Message Too Old, No Replies

another plainclothes bingbot?

         

lucy24

7:47 pm on Sep 15, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Showed up abruptly on 12 September:

40.77.169.100 - - [12/Sep/2016:17:12:27 -0700] "GET /ebooks/bourquin/ HTTP/1.1" 200 8391 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36" 
40.77.169.101 - - [12/Sep/2016:19:47:31 -0700] "GET /ebooks/kleinschmidt HTTP/1.1" 301 564 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36"
40.77.169.101 - - [12/Sep/2016:19:47:31 -0700] "GET /ebooks/kleinschmidt/ HTTP/1.1" 200 5297 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36"

Identical pattern--even the same times, give or take ten minutes--on the 13th and 14th.

I've got a long-standing lockout on the plainclothes bingbot, but (a) it doesn't cover the 40.77 range, and (b) it's for the MSIE user-agent.

A possibly-bing-related entity calling itself Chrome is almost as funny as the mobile bingbot calling itself iPhone.

Edit: Eeeuw. I'm not used to viewing the Forums in Firefox. What language does the color-coder think the "code" is in?

not2easy

8:22 pm on Sep 15, 2016 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



I see the identical UA as a humanoid visit from a Verizon IP this month. I have not completed all logs yet. From a nearby but different 40.77.167.80 I see a Bingbot UA blocked itself by ignoring robots.txt (after requesting it 3 times in a row.)

I don't have anything (yet) with that UA in that range, but as mentioned, I'm not all up to date yet.

keyplyr

8:25 pm on Sep 15, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



While the range is registered to Microsoft Corp, it is not a designated crawl range so the visits to your web site could really be unrelated to Bingbot. I assume "plainclothes bingbot" is just a moniker you give to anything unexplained from M$. As you've noted, M$ has always done this stuff.

lucy24

4:17 am on Sep 16, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I assume "plainclothes bingbot" is just a moniker you give to anything unexplained from M$.

Come to think of it, these visits have shown pure robotic behavior: html alone, nothing else. Other bingoid entities (such as the notorious Drake Holdings) also get styles and scripts, though not images.

Simon_H

11:58 am on Sep 28, 2016 (gmt 0)

10+ Year Member Top Contributors Of The Month



Was about to start a thread on this and came across this one!

Seeing something similar - the IPs are Bingbot, but the UA is wrong, so they're messing up the custom tracking we have as our tracking uses the UA to determine bot exclusions. They include IP ranges 131.253.xxx.xxx and UA is Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/5.0; Trident/5.0).

They're hitting lots of legacy product pages on the site, so this feels like normal bot activity, just with the wrong UA. Or maybe I'm missing something very obvious?