Odd Bingbot activity.

Bingbot spidering pages that don't exist

[bj61251]

Ok, I've had about enough of this, so I'm trying to see if anyone has any insight into this odd issue.

For some time now (6 months at least), I've been witnessing in my error logs a significant number of requests by Bingbot ("Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" more specifically), where the url requested looks like it belongs to another domain. I've checked the IPs of these requests, and they seem to all trace to MSN.

A few examples:

• http://www.[mydomain].com:80/forum.php?mod=redirect&goto=findpost&ptid=10165&pid=172148
  
• http://www.[mydomain].com:80/jobsearch/searchjobs.php?x_y=1dc529d45e957bc00e5912787ad1a752&err=no&x_n509cbd198e7c172e27549a075da512d5
  
• http://www.[mydomain].com:80/news/society/2010/12/28/215099.shtml
  
• http://www.[mydomain].com:80/promoG/LandingPage.asp?lpid=542&pid=564&storeID=D9339NK282S92NV700AKHLBD3LHF7NV2
  
• http://www.[mydomain].com:80/invar/services/auth.php?women_add=85945&referer_referer=CLOSE_WINDOW
  
• http://www.[mydomain].com:80/hangan-WEISEI.TTF-
  

None of these are anywhere close to urls that exist on my site (which is purely .aspx). Also, none of the requests have a referrer url, keeping it consistent with a spider.

Has anyone seen this before? I've seen requests get garbled on occasion (missing characters or two urls jammed together for whatever reason), but I chalk most of those up to a browser issue or a hiccup in the communication process, but these look like they could be legitimate pages on other domains. I know that, unfortunately, there probably isn't much I can do about it, but it would comfort me a bit to know that I'm not the only person seeing this activity, especially if there is a reason why it is happening.

[lucy24]

Frivolous though: Gee, Bing really is trying to become more like g### every day isn't it :)

Has anyone else ever got the impression that Bing uses MSN to do its dirty work?

[tangor]

Are you returning proper 404s? All the SEs out there test that from time to time. Not saying the above is that specifically.

Operating a forum? If so, are others linking to it? Properly? A blank referer (sic) alone does not equal "bot" activity.

[bj61251]

Yep, they all return a 404. I thought about the chance that it was just Bing testing for 404's, but that wouldn't explain the frequency, or the seemingly random URLs that it is looking for. No forum, either (we are a retail site, if that matters in any way).

Could it be a DNS issue of some kind? On much rarer occasions, I've seen requests for a completely different domain (from regular browser user agents), and I've assumed that these were likely because some DNS somewhere spit out the wrong IP address for the domain that someone was looking for. I just figured that wasn't the case here, because the requests for the current problem are clearly directed towards my domain.