homepage Welcome to WebmasterWorld Guest from 54.225.57.156
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

This 131 message thread spans 5 pages: 131 ( [1] 2 3 4 5 > >     
MSN fakes referrers
SEOPTI




msg:3875365
 6:14 pm on Mar 20, 2009 (gmt 0)

This has been discussed in 2007:
[webmasterworld.com...]

They do it again, I see hundreds of fake visitors from MSN IPs across all of my domains.

Are there any news what they try to accomplish by doing this?

 

wilderness




msg:3875572
 11:05 pm on Mar 20, 2009 (gmt 0)

SEOPTI,
There's been multiple discussions of the "search.live" refers in numerous MSN threads.

I don't recall a valid explanation being presented, however I may be mistaken.

In October of last year, I had partial key word phrases, which resulted in multiple page request in succession.
EX:
Valid phrase= "widget house".
Search1= widget
Search2=house

Both secondary searches were/are invalid on my sites.

I grew weary of the daily requests and denied ALL search.live refers for a time.
Eventually these request ceased and I removed the denial.

Sorry I'm unable to be more help.

Don

dstiles




msg:3875587
 11:40 pm on Mar 20, 2009 (gmt 0)

Wilderness - did denying them affect MSN searches / results in any way? I have to admit their junk hits are annoying.

wilderness




msg:3875601
 12:00 am on Mar 21, 2009 (gmt 0)

did denying them affect MSN searches / results in any way?

dstiles,
No!

However in all honesty, there are more "search.live" refers these days than "msn" refers.

The majority of the "search.live" refers to my sites are on topic.

dstiles




msg:3875635
 1:42 am on Mar 21, 2009 (gmt 0)

Thanks. I may consider blocking them for a while in that case. MSN has never been a good deliverer anyway, despite constantly crawling the sites.

blend27




msg:3875688
 3:18 am on Mar 21, 2009 (gmt 0)

-- did denying them affect MSN searches --

NO, not the Arin/Ripe Traffic at least.

SEOPTI




msg:3878587
 4:56 pm on Mar 25, 2009 (gmt 0)

I really don't know what to do with all those fake requests from MSN, they are bombing my sites with requests like "phone" or "city" .. this really screws up my statistics.

Maybe they are trying to fool people so it looks like MSN gains market share? What a cool strategy!

[edited by: SEOPTI at 4:57 pm (utc) on Mar. 25, 2009]

wilderness




msg:3878595
 5:04 pm on Mar 25, 2009 (gmt 0)

really don't know what to do with all those fake requests from MSN, they are bombing my sites with requests like "phone" or "city" .. this really screws up my statistics.

In the event the refer terms are the same?
An option of denial based on same is possible, although not 100% effective.

Had some visitors that were bombing my sites regularly with:

"contact mail email phone fax tel"

SEOPTI




msg:3895173
 2:24 pm on Apr 18, 2009 (gmt 0)

400 fake requests with the word "phone" today and counting for one of my sites from MSN, what a dumb machine!

SEOPTI




msg:3895826
 1:50 am on Apr 20, 2009 (gmt 0)

I think I have a solution, maybe MSN is trying to spot mars people on my sites? ;)

Ocean10000




msg:3896183
 3:18 pm on Apr 20, 2009 (gmt 0)

Best that I can tell, they are trying to use these automated queries, which are all on the MSN Bot ranges, to improve their search engine results. My guess is that these queries do not go though as many filters as the production Search does. And are designed to show some indirect terms, which might not be on the page, but on pages referring to your page, or common in your nitch. They may also be designed to detect if you alter your page based on those search terms entered.

I know myself I just filter these out of my resulting Search terms and Referrer stats based on ip range. I ignore all search terms and referrers from known search engine bot ranges, so they do not skew my stats.

MS_Jason




msg:3896390
 7:30 pm on Apr 20, 2009 (gmt 0)

SEOPTI,

If you keep having the problem, please contact Betsy Aoki. You can get hold of her via the contact form on her blog: [blogs.msdn.com...]

She can make sure your issue gets to the right person on the search team.

Cheers,
Jason

incrediBILL




msg:3897044
 2:50 pm on Apr 21, 2009 (gmt 0)

Jason, Welcome to WebmasterWorld, and thanks for the contact because many of us have similar bizarre issues with MSNBOT and other strange things crawling out of MS and any help to stop this or explain it is always greatly appreciated.

Umbra




msg:3897064
 3:23 pm on Apr 21, 2009 (gmt 0)

I didn't understand MS_Jason's reply. So many of us have seen this strange activity -- are we all supposed to separately contact Betsy, thus requiring her to individually respond to all of us? Isn't it more practical if MS_Jason took advantage of this public forum to explain what was going on?

Or is he implying that the strange activity should now have ceased and we should only contact Betsy if we're still seeing it?

[edited by: Umbra at 3:26 pm (utc) on April 21, 2009]

Frank_Rizzo




msg:3897115
 4:31 pm on Apr 21, 2009 (gmt 0)

I had a similar issue with the msnbot - crawling non existent pages.

Contacted microsoft live support who said they would escalate it. That was over a week ago. Nothing heard since.

Betsy is going to be busy.

incrediBILL




msg:3897140
 4:57 pm on Apr 21, 2009 (gmt 0)

One thing I've noticed are 3rd party sites deliberately linking to pages that don't exist in what it looks like they are doing is trying to make MSNBOT (and others) unwittingly hack sites.

For instance, one time I found msnbot shoving some 7K long URIs into my server for some reason which caused the server to hiccup, hopefully that's all it did.

All of these requested URIs had a huge binary string embedded that typically starts with "%C3%83%C6%92%C3%86%E2%80%99%C3%83%E2%80" and continues.

The requests look like this:

example.com/my-page-%C3%83%C6%92%C3%86%E2%80%99%C3%83%E2%80....-here.html

I did a search in Live and sure enough, there appear to be sites out there indexed that are embedding these strings in URIs either for some purpose and some of those sites are linked to malware, so draw your own conclusion.

msndude




msg:3897211
 6:27 pm on Apr 21, 2009 (gmt 0)

Hi,

This is Brett Yount from Live Search's Webmaster Center. I would like to apologize for the inconvenience this is causing. We are working to correct this issue ASAP.

blend27




msg:3897247
 7:35 pm on Apr 21, 2009 (gmt 0)

Brett,

Thanks for stopping by.

*ttp://search.live.com/results.aspx?q=string

Yes, the actual keyword is a word "STRING". Why are your bots wasting my bandwidth, server resources and most precious - My Time?

We constantly SCORE about useless Scrapers and Spammers for wasting our time.

[webmasterworld.com...]

Why are your bots contributing to wasted energy?

GaryK




msg:3897299
 8:47 pm on Apr 21, 2009 (gmt 0)

Might I respectfully suggest we give Brett a chance to get this issue resolved? Then if they are still wasting our bandwidth we can gripe some more. We've griped. Gotten the attention we want. Now it's time to be patient for just a bit. :)

SEOPTI




msg:3897484
 3:08 am on Apr 22, 2009 (gmt 0)

Thank you MS_Jason and Brett for looking into it.

sem4u




msg:3897592
 8:54 am on Apr 22, 2009 (gmt 0)

This has been affecting many of my sites. It is good to see the issue is being looked into.

Rosalind




msg:3897894
 5:00 pm on Apr 22, 2009 (gmt 0)

I'm also seeing an increase in fake referrer activity from MSN. An explanation would be welcome. It seems to me this has been going on for a long time, and so far there's been nothing but an embarrassed silence.

sem4u




msg:3898708
 3:42 pm on Apr 23, 2009 (gmt 0)

From looking at my stats it looks like it started happening on 16th/17th March.

msndude




msg:3898924
 7:24 pm on Apr 23, 2009 (gmt 0)

Thank you for your patience. Our team is continuing the investigation. We have made some changes. Please let me know if you notice any changes (positive or negative) over the next few days.

Thanks,

Brett

SEOPTI




msg:3900375
 1:48 pm on Apr 25, 2009 (gmt 0)

No changes at all, still heavy fake requests.

sem4u




msg:3901438
 11:07 am on Apr 27, 2009 (gmt 0)

It is still happening on my sites.

Rosalind




msg:3901526
 2:03 pm on Apr 27, 2009 (gmt 0)

No change here, I'm still seeing it.

blend27




msg:3901735
 6:31 pm on Apr 27, 2009 (gmt 0)

1 small site - less than 300 active pages.

6136 page downloads in the past 4 month by this thingy, last hit: 2009-04-26 18:04:23.680 EST

Average page weight 80KB

That is 490880 KB = HaFa GIG

I wonder how much time it will take an average "Pedro, The Realtor" to visit and read 6136 pages?

SEOPTI




msg:3901921
 10:23 pm on Apr 27, 2009 (gmt 0)

Is there really a reason to hit a site with fake search requests for "phone" or "address" hundreds of times a day?

[edited by: SEOPTI at 10:24 pm (utc) on April 27, 2009]

MS_Jason




msg:3901978
 11:13 pm on Apr 27, 2009 (gmt 0)

Hi All,
Thanks for updating us on what you're seeing. FWIW, we've got a post up on the webmaster blog on how to escalate these issues in the future: [blogs.msdn.com...]

Cheers,
Jason

This 131 message thread spans 5 pages: 131 ( [1] 2 3 4 5 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved