Welcome to WebmasterWorld Guest from 54.81.220.239

Forum Moderators: Ocean10000

Getting hammered by Bingbot

     
4:25 pm on May 17, 2019 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
posts:9884
votes: 967


Since May 2nd I have been getting hammered (over 68000 hits) for filenames NOT on my server. They are in the form of

/xxxxxxxxxx where x is any number 0-9, always 10 digits.

More strange, while majority are 404, a significant number are showing as 301!

What is going on?

Meanwhile, I want to ensure that any 301 (other files are affected as well) is never processed as I HAVE NO REDIRECTS IN FORCE or is returned as a 404-410. Point me in the direction to learn enough regex to make any request for /xxxxxxxxxx (any integer) die a horrible 404 death. What I know would fill a gnat's thimble, and that's a lot smaller than a regular thimble.

Kicked into overdrive last night in the 301s began to take in existing 200 files on the system!

All IP addresses checked so far resolve to MS, though I did locate at least one that was G.

.htaccess is clean ... has nothing I did not put in it.

Site moved from host where it lived for 15 years April 13 with no problems. Logs were normal until recently.

Asking here since the villain involved is:

Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

What is most perplexing is the site getting hammered only has 762 pages, including css!
4:40 pm on May 17, 2019 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Aug 30, 2002
posts: 2658
votes: 103


Bingbot ranges or MSFT's Azure cloud? Microsoft tends to have some rather dimwitted and completely amateurish approach to dealing with sitemaps and will hammer away at site. The /xxxxx thing is a bit unusual. Did you do a search on Google or Bing to see if there are such links from other sites? Also, do these IPs have reverse-dns set up so that they resolve to .search.msn.com ranges?

Regards...jmcc
5:28 pm on May 17, 2019 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
posts:9884
votes: 967


Combination of both (bing and Azure) ... as for the ips tested, just pasted into reverse lookups and all resolve to search-msm or at least one azure. Checking ips past xxx.xxx might reveal other things?

What is crazy for me is that up until last night these /xxxxxxxxxx requests were getting 404s and NOW have changed to 301!
5:52 pm on May 17, 2019 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15698
votes: 810


a significant number are showing as 301
Nothing unexpected there: It's asking for the wrong www and/or wrong https; the server doesn’t go looking for the (nonexistent) file until the request is otherwise acceptable.

Pattern:
^\d{10}$
6:06 pm on May 17, 2019 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
posts:9884
votes: 967


That might be an answer ... this site lay fallow for a zillion years ... had to move it. In process of that did some updates. Just recently installed:

#Force non-www:
RewriteEngine on
RewriteCond %{HTTP_HOST} ^www\.example\.com [NC]
RewriteRule ^(.*)$ http://example.com/$1 [L,R=301]

I suppose I should just wait this out?

As for the regex on the pestiferous /xxxxxxxxxx, THANKS!
6:21 am on May 18, 2019 (gmt 0)

Junior Member

10+ Year Member

joined:Sept 18, 2004
posts:45
votes: 3


Getting hammered by Bingbot


Microsoft should rebrand it to Bangbot?
12:50 pm on May 18, 2019 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
posts:9884
votes: 967


@Montresor ... that deserves a trip to the woodshed for ... cake and ice cream! :)