homepage Welcome to WebmasterWorld Guest from 54.237.38.30
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

This 131 message thread spans 5 pages: 131 ( [1] 2 3 4 5 > >     
MSN fakes referrers
SEOPTI

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3875363 posted 6:14 pm on Mar 20, 2009 (gmt 0)

This has been discussed in 2007:
[webmasterworld.com...]

They do it again, I see hundreds of fake visitors from MSN IPs across all of my domains.

Are there any news what they try to accomplish by doing this?

 

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 3875363 posted 11:05 pm on Mar 20, 2009 (gmt 0)

SEOPTI,
There's been multiple discussions of the "search.live" refers in numerous MSN threads.

I don't recall a valid explanation being presented, however I may be mistaken.

In October of last year, I had partial key word phrases, which resulted in multiple page request in succession.
EX:
Valid phrase= "widget house".
Search1= widget
Search2=house

Both secondary searches were/are invalid on my sites.

I grew weary of the daily requests and denied ALL search.live refers for a time.
Eventually these request ceased and I removed the denial.

Sorry I'm unable to be more help.

Don

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 3875363 posted 11:40 pm on Mar 20, 2009 (gmt 0)

Wilderness - did denying them affect MSN searches / results in any way? I have to admit their junk hits are annoying.

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 3875363 posted 12:00 am on Mar 21, 2009 (gmt 0)

did denying them affect MSN searches / results in any way?

dstiles,
No!

However in all honesty, there are more "search.live" refers these days than "msn" refers.

The majority of the "search.live" refers to my sites are on topic.

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 3875363 posted 1:42 am on Mar 21, 2009 (gmt 0)

Thanks. I may consider blocking them for a while in that case. MSN has never been a good deliverer anyway, despite constantly crawling the sites.

blend27

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3875363 posted 3:18 am on Mar 21, 2009 (gmt 0)

-- did denying them affect MSN searches --

NO, not the Arin/Ripe Traffic at least.

SEOPTI

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3875363 posted 4:56 pm on Mar 25, 2009 (gmt 0)

I really don't know what to do with all those fake requests from MSN, they are bombing my sites with requests like "phone" or "city" .. this really screws up my statistics.

Maybe they are trying to fool people so it looks like MSN gains market share? What a cool strategy!

[edited by: SEOPTI at 4:57 pm (utc) on Mar. 25, 2009]

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 3875363 posted 5:04 pm on Mar 25, 2009 (gmt 0)

really don't know what to do with all those fake requests from MSN, they are bombing my sites with requests like "phone" or "city" .. this really screws up my statistics.

In the event the refer terms are the same?
An option of denial based on same is possible, although not 100% effective.

Had some visitors that were bombing my sites regularly with:

"contact mail email phone fax tel"

SEOPTI

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3875363 posted 2:24 pm on Apr 18, 2009 (gmt 0)

400 fake requests with the word "phone" today and counting for one of my sites from MSN, what a dumb machine!

SEOPTI

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3875363 posted 1:50 am on Apr 20, 2009 (gmt 0)

I think I have a solution, maybe MSN is trying to spot mars people on my sites? ;)

Ocean10000

WebmasterWorld Administrator 10+ Year Member



 
Msg#: 3875363 posted 3:18 pm on Apr 20, 2009 (gmt 0)

Best that I can tell, they are trying to use these automated queries, which are all on the MSN Bot ranges, to improve their search engine results. My guess is that these queries do not go though as many filters as the production Search does. And are designed to show some indirect terms, which might not be on the page, but on pages referring to your page, or common in your nitch. They may also be designed to detect if you alter your page based on those search terms entered.

I know myself I just filter these out of my resulting Search terms and Referrer stats based on ip range. I ignore all search terms and referrers from known search engine bot ranges, so they do not skew my stats.

MS_Jason

5+ Year Member



 
Msg#: 3875363 posted 7:30 pm on Apr 20, 2009 (gmt 0)

SEOPTI,

If you keep having the problem, please contact Betsy Aoki. You can get hold of her via the contact form on her blog: [blogs.msdn.com...]

She can make sure your issue gets to the right person on the search team.

Cheers,
Jason

incrediBILL

WebmasterWorld Administrator incredibill us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



 
Msg#: 3875363 posted 2:50 pm on Apr 21, 2009 (gmt 0)

Jason, Welcome to WebmasterWorld, and thanks for the contact because many of us have similar bizarre issues with MSNBOT and other strange things crawling out of MS and any help to stop this or explain it is always greatly appreciated.

Umbra

10+ Year Member



 
Msg#: 3875363 posted 3:23 pm on Apr 21, 2009 (gmt 0)

I didn't understand MS_Jason's reply. So many of us have seen this strange activity -- are we all supposed to separately contact Betsy, thus requiring her to individually respond to all of us? Isn't it more practical if MS_Jason took advantage of this public forum to explain what was going on?

Or is he implying that the strange activity should now have ceased and we should only contact Betsy if we're still seeing it?

[edited by: Umbra at 3:26 pm (utc) on April 21, 2009]

Frank_Rizzo

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3875363 posted 4:31 pm on Apr 21, 2009 (gmt 0)

I had a similar issue with the msnbot - crawling non existent pages.

Contacted microsoft live support who said they would escalate it. That was over a week ago. Nothing heard since.

Betsy is going to be busy.

incrediBILL

WebmasterWorld Administrator incredibill us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



 
Msg#: 3875363 posted 4:57 pm on Apr 21, 2009 (gmt 0)

One thing I've noticed are 3rd party sites deliberately linking to pages that don't exist in what it looks like they are doing is trying to make MSNBOT (and others) unwittingly hack sites.

For instance, one time I found msnbot shoving some 7K long URIs into my server for some reason which caused the server to hiccup, hopefully that's all it did.

All of these requested URIs had a huge binary string embedded that typically starts with "%C3%83%C6%92%C3%86%E2%80%99%C3%83%E2%80" and continues.

The requests look like this:

example.com/my-page-%C3%83%C6%92%C3%86%E2%80%99%C3%83%E2%80....-here.html

I did a search in Live and sure enough, there appear to be sites out there indexed that are embedding these strings in URIs either for some purpose and some of those sites are linked to malware, so draw your own conclusion.

msndude

10+ Year Member



 
Msg#: 3875363 posted 6:27 pm on Apr 21, 2009 (gmt 0)

Hi,

This is Brett Yount from Live Search's Webmaster Center. I would like to apologize for the inconvenience this is causing. We are working to correct this issue ASAP.

blend27

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3875363 posted 7:35 pm on Apr 21, 2009 (gmt 0)

Brett,

Thanks for stopping by.

*ttp://search.live.com/results.aspx?q=string

Yes, the actual keyword is a word "STRING". Why are your bots wasting my bandwidth, server resources and most precious - My Time?

We constantly SCORE about useless Scrapers and Spammers for wasting our time.

[webmasterworld.com...]

Why are your bots contributing to wasted energy?

GaryK

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3875363 posted 8:47 pm on Apr 21, 2009 (gmt 0)

Might I respectfully suggest we give Brett a chance to get this issue resolved? Then if they are still wasting our bandwidth we can gripe some more. We've griped. Gotten the attention we want. Now it's time to be patient for just a bit. :)

SEOPTI

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3875363 posted 3:08 am on Apr 22, 2009 (gmt 0)

Thank you MS_Jason and Brett for looking into it.

sem4u

WebmasterWorld Senior Member sem4u us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3875363 posted 8:54 am on Apr 22, 2009 (gmt 0)

This has been affecting many of my sites. It is good to see the issue is being looked into.

Rosalind

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3875363 posted 5:00 pm on Apr 22, 2009 (gmt 0)

I'm also seeing an increase in fake referrer activity from MSN. An explanation would be welcome. It seems to me this has been going on for a long time, and so far there's been nothing but an embarrassed silence.

sem4u

WebmasterWorld Senior Member sem4u us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3875363 posted 3:42 pm on Apr 23, 2009 (gmt 0)

From looking at my stats it looks like it started happening on 16th/17th March.

msndude

10+ Year Member



 
Msg#: 3875363 posted 7:24 pm on Apr 23, 2009 (gmt 0)

Thank you for your patience. Our team is continuing the investigation. We have made some changes. Please let me know if you notice any changes (positive or negative) over the next few days.

Thanks,

Brett

SEOPTI

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3875363 posted 1:48 pm on Apr 25, 2009 (gmt 0)

No changes at all, still heavy fake requests.

sem4u

WebmasterWorld Senior Member sem4u us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3875363 posted 11:07 am on Apr 27, 2009 (gmt 0)

It is still happening on my sites.

Rosalind

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3875363 posted 2:03 pm on Apr 27, 2009 (gmt 0)

No change here, I'm still seeing it.

blend27

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3875363 posted 6:31 pm on Apr 27, 2009 (gmt 0)

1 small site - less than 300 active pages.

6136 page downloads in the past 4 month by this thingy, last hit: 2009-04-26 18:04:23.680 EST

Average page weight 80KB

That is 490880 KB = HaFa GIG

I wonder how much time it will take an average "Pedro, The Realtor" to visit and read 6136 pages?

SEOPTI

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3875363 posted 10:23 pm on Apr 27, 2009 (gmt 0)

Is there really a reason to hit a site with fake search requests for "phone" or "address" hundreds of times a day?

[edited by: SEOPTI at 10:24 pm (utc) on April 27, 2009]

MS_Jason

5+ Year Member



 
Msg#: 3875363 posted 11:13 pm on Apr 27, 2009 (gmt 0)

Hi All,
Thanks for updating us on what you're seeing. FWIW, we've got a post up on the webmaster blog on how to escalate these issues in the future: [blogs.msdn.com...]

Cheers,
Jason

This 131 message thread spans 5 pages: 131 ( [1] 2 3 4 5 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved