homepage Welcome to WebmasterWorld Guest from 184.72.69.79
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

This 131 message thread spans 5 pages: < < 131 ( 1 [2] 3 4 5 > >     
MSN fakes referrers
SEOPTI




msg:3875365
 6:14 pm on Mar 20, 2009 (gmt 0)

This has been discussed in 2007:
[webmasterworld.com...]

They do it again, I see hundreds of fake visitors from MSN IPs across all of my domains.

Are there any news what they try to accomplish by doing this?

 

blend27




msg:3902256
 9:35 am on Apr 28, 2009 (gmt 0)

I am not going to be any less sarcastic than I usually am, but that looks like "Fill Out a Form" link.

There is still no response to why this is happening. hmmm, Friday word game anyone?

-- Thank you once again! MSN fakes referrers

wilderness




msg:3902497
 4:13 pm on Apr 28, 2009 (gmt 0)

Went through 24-pages of results and no links to my sites appeared on this [url=http:// search.live.com/results.aspx?q=gauman]search phrase[/url], however a visitor entered one of my pages on this same search.

[edited by: wilderness at 4:16 pm (utc) on April 28, 2009]

msndude




msg:3902498
 4:13 pm on Apr 28, 2009 (gmt 0)

Hi,

It may seem different, but I really do enjoy helping people overcome issues. We are still working to correct this issue.

Out of curiosity, have any of you tried setting crawl-delays in your robots.txt or set conditional gets to slow the requests?

Brett

SEOPTI




msg:3902573
 5:50 pm on Apr 28, 2009 (gmt 0)

Yes, crawl-delays have been set, but of course a crawl delay won't sort out this issue, it simply delays the fake requests.

incrediBILL




msg:3902581
 6:09 pm on Apr 28, 2009 (gmt 0)

Yes, I've been using crawl-delays for years now and every now and then msnbot just goes off the deep end and brought my server down 2 or 3 times last year until I temporarily blocked it altogether until it decided to behave itself.

msndude




msg:3902624
 7:15 pm on Apr 28, 2009 (gmt 0)

I just spoke with the team. They said they are working on a fix for this, but the feature causing the problem is turned off currently, so you shouldn't be seeing any fake referrals from us.

Samizdata




msg:3902630
 7:17 pm on Apr 28, 2009 (gmt 0)

Thanks msndude, we really appreciate the feedback.

...

sem4u




msg:3903873
 8:22 am on Apr 30, 2009 (gmt 0)

I still see them coming through here.

JS_Harris




msg:3903953
 11:44 am on Apr 30, 2009 (gmt 0)

EPN affiliates apparently noticed this again on April 19th, Interestingly enough it appears that Google Analytics didn't count the massive temporary increase in traffic but EPN reported it and 3rd party tracking software/sources confirmed the "carpet bombing".

The nofollow tag was ignored of course.

blend27




msg:3904043
 2:26 pm on Apr 30, 2009 (gmt 0)

65.55.110.86 at 2009-04-30 00:58:51.793
65.55.109.162 at 2009-04-29 16:19:49.923
65.55.232.49 at 2009-04-29 15:43:22.970
65.55.232.49 at 2009-04-29 14:53:50.553
65.55.107.239 at 2009-04-29 06:11:25.913
65.55.109.19 at 2009-04-28 23:52:58.977
65.55.109.12 at 2009-04-28 22:43:18.100
65.55.110.107 at 2009-04-28 19:29:28.467
65.55.109.234 at 2009-04-28 19:24:47.077
65.55.110.122 at 2009-04-28 19:15:08.140
65.55.109.196 at 2009-04-28 17:04:53.120
65.55.109.118 at 2009-04-28 14:02:28.837
65.55.110.192 at 2009-04-28 12:30:52.497
65.55.110.237 at 2009-04-28 10:34:44.017
65.55.109.210 at 2009-04-28 09:23:40.790
65.55.109.211 at 2009-04-28 08:29:07.243
65.55.107.233 at 2009-04-28 06:11:22.050

NetRange: 65.52.0.0 - 65.55.255.255
CIDR: 65.52.0.0/14
NetName: MICROSOFT-1BLK
NetHandle: NET-65-52-0-0-1
NetType: Direct Assignment

They are still comming, so

nealrodriguez




msg:3904254
 6:19 pm on Apr 30, 2009 (gmt 0)

Is there really a reason to hit a site with fake search requests for "phone" or "address" hundreds of times a day?

chronic amnesia?

antoineFR




msg:3904277
 6:50 pm on Apr 30, 2009 (gmt 0)

same with french web site. Fake referer from Microsoft IP.

But also fake agent, and even javascript simulation to get cookies !

Ex today on a lot of sites : IP 65.55.110.242

Agent : Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR 1.1.4322)
Referer : [search.live.com...]

If i was not sure at 100% that microsoft is an honnest company i would say that it is done to fake world stats about Live use...

Antoine.

confuscius




msg:3904279
 6:54 pm on Apr 30, 2009 (gmt 0)

As the original poster back in 2007, I am pleased to see that action now seems to be forthcoming on this issue. I took the decision to ban MSN from all of my domains except one that I left open - this has been happening continuously during the period and once MSN finds you then there is no escape.

Given past responses from MSN then I am not getting my hopes up that this will be fixed but look forward to being surprised.

Is there a 'Guiness World Record for the slowest technical fix of all time'? - if so, then it is about time MSN received the accolade they deserve.

particleman




msg:3904515
 11:12 pm on Apr 30, 2009 (gmt 0)

Glad this has come up, I've noticed this for a long time in our stats. Keywords seem to be centered around the subject of the site also. Can someone actually explain what this is about?

koan




msg:3905277
 11:16 pm on May 1, 2009 (gmt 0)

This was a serious pet peeves of mine with Live, glad to hear it's finally getting some attention. None too soon!

msndude




msg:3905345
 4:06 am on May 2, 2009 (gmt 0)

Thanks for keeping me updated. I'll take this back to the team. I'll update this thread as soon as I have more info.

If any of you have recent screen caps that you wouldn't mind sharing, could you please send them to me at lswmc@microsoft.com with your domain name and this thread's title in the subject line?

Thanks,

Brett

msndude




msg:3916081
 10:20 pm on May 18, 2009 (gmt 0)

Writing a new blog post regarding these issues. Should be up on our webmaster blog in a couple days/ week max.

Brett

Pfui




msg:3918905
 5:35 am on May 23, 2009 (gmt 0)

While we await Brett's blog post (thank you), I thought I'd mention that MSN's generic-word, fake-referer scheme/program is baaaaack, or never left.

Hitting html, js, and css files, the following are from two disparate sites today, and earlier this month. Examples chosen because of the IP/Host mix.

65.55.110.208
[search.live.com...]

65.55.110.42
[search.live.com...]

msnbot-65-55-109-241.search.msn.com
[search.live.com...]

The UA is always: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR 1.1.4322)

Pfui




msg:3927440
 7:30 pm on Jun 5, 2009 (gmt 0)

FWIW, MSN is still faking it. From a quick skim of my logs (vs grepping for all instances):

06-03: 65.55.109.64
06-04: msnbot-65-55-110-207.search.msn.com
06-05: 65.55.110.110

REF x3: [search.live.com...]
UA x3: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR 1.1.4322)
URIs: Three different html files.

I deny MSN's usual fakes (e.g., SetEnvIfNoCase Referer "q=secrets" scram) but doing so doesn't deter.

Pfui




msg:3927492
 8:56 pm on Jun 5, 2009 (gmt 0)

And one more, just in, because this fake q= is nothing but a waste of everyone's resources:

msnbot-65-55-109-147.search.msn.com
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR 1.1.4322)
[search.live.com...]

(Gimme a break.)

TravelMan




msg:3927756
 9:25 am on Jun 6, 2009 (gmt 0)

I'm getting this too - I'm tired of these jokers, msndude, you bing guys are doing yourself noooo favours. Can someone just turn this nonsense off!

incrediBILL




msg:3928020
 10:29 pm on Jun 6, 2009 (gmt 0)

Writing a new blog post regarding these issues. Should be up on our webmaster blog in a couple days/ week max.

Thanks MSNDUDE, we'll be looking forward to it.

C'mon guys, give MSNDUDE some respect, he's here trying to help and explain what's going on with this.

BillyS




msg:3928040
 11:33 pm on Jun 6, 2009 (gmt 0)

>>FWIW, MSN is still faking it

Of course they are, this is one of the ways they detect spam. They're not going to fix this anytime soon.

I've already started blocking them...

blend27




msg:3928235
 2:57 pm on Jun 7, 2009 (gmt 0)

I think any half baked blackhat out there knows that THIS BOT comes from CIDR: 65.52.0.0/14, how easy is it to cloak the JS/CSS content to that if they wanted to?, think about it.

.classI{text-indent: <cfif not request.bot>-999</cfif>4px;} :)

JQuery UI Lib is 300 kb minified, it was pull down by this BOT more than several times a day, NO MORE - 403 with no changes in rankings so far.

Patiently waiting for that blog post by MSNDUDE...

Blend27

rytis




msg:3934264
 11:54 am on Jun 16, 2009 (gmt 0)

MS just searched my site for "windows", I am willing to help, but how?

blend27




msg:3934493
 5:53 pm on Jun 16, 2009 (gmt 0)

and my site for "company", musta been a lonely Bot looking for something to Bing. Too bad, I am already hitched... :)

BillyS




msg:3934804
 1:54 am on Jun 17, 2009 (gmt 0)

I just saw another fake one from Live... it came from 65.55.110.186.

[edited by: BillyS at 1:54 am (utc) on June 17, 2009]

Pfui




msg:3937399
 5:23 pm on Jun 20, 2009 (gmt 0)

MSN's fake-rate is higher than they ever, with multiples and even dupes, too. All ridiculously broad and resource-wasting:

06/18 13:12:59 [search.live.com...]
06/18 13:13:52 [search.live.com...]
06/18 13:13:53 [search.live.com...]
06/18 14:10:04 [search.live.com...]

All of those came from this real Host:

msnbot-65-55-104-nn.search.msn.com
(nn = multiple numbers)

All Hosts used this cloaked UA --

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4325; .NET CLR 2.0.40607; .NET CLR 3.0.04506.648)

All 403'd. C'est la vie.

dstiles




msg:3937474
 8:25 pm on Jun 20, 2009 (gmt 0)

Has anyone seen a real msn referral querystring that's as simple as these? Would ^q=[a-z]$ ever trap a real "live" (or bing) referral?

I'm in the process of setting up a new stats analysis system for my clients and have had to warn them that MS referrals are artificially high and are probably nearer 0.3% than the 3.3% that seems typical in the logs.

GaryK




msg:3937921
 11:34 pm on Jun 21, 2009 (gmt 0)

Now that it's happening to me I have a better handle on what this thread is all about.

06/14/2009:
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR 1.1.4325; .NET CLR 2.0.40607; .NET CLR 3.0.04506.648)
65.55.104.67
msnbot-65-55-104-67.search.msn.com
[search.live.com...]

06/16/2009:
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648)
65.55.104.65
msnbot-65-55-104-65.search.msn.com
[search.live.com...]

06/17/2009:
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SLCC1; .NET CLR 1.1.4325; .NET CLR 2.0.40607; .NET CLR 3.0.30729; .NET CLR 3.5.30707; MS-RTC LM 8)
65.55.104.61
msnbot-65-55-104-61.search.msn.com
[search.live.com...]

06/18/2009:
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SLCC1; .NET CLR 1.1.4322; .NET CLR 2.0.40607; .NET CLR 3.0.04506.648)
65.55.104.62
msnbot-65-55-104-62.search.msn.com
[search.live.com...]

06/20/2009:
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SLCC1; .NET CLR 1.1.4325; .NET CLR 2.0.40607; .NET CLR 3.0.30729; .NET CLR 3.5.30707; InfoPath.2)
65.55.104.72
msnbot-65-55-104-72.search.msn.com
[search.live.com...]

GaryK




msg:3941996
 5:45 pm on Jun 28, 2009 (gmt 0)

Another week and yet more fake referrers from MSN. Personally I've had enough. I just don't get enough traffic from MSN to make this worthwhile so as of now they're banned from all my sites.

6/24/2009
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SLCC1; .NET CLR 1.1.4325; .NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30707; MS-RTC LM 8)
65.55.104.64
msnbot-65-55-104-64.search.msn.com
[search.live.com...]

6/27/2009
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SLCC1; .NET CLR 1.1.4325; .NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729; MS-RTC LM 8)
65.55.104.63
msnbot-65-55-104-63.search.msn.com
[search.live.com...]

This 131 message thread spans 5 pages: < < 131 ( 1 [2] 3 4 5 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved