homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Blocking Windows Live robot/spider.

10+ Year Member

Msg#: 3442381 posted 9:14 pm on Sep 5, 2007 (gmt 0)

I would rather lose all windows live traffic rather than explain why sites I host get a simmulated or live MS sex traffic search visitor for a '***** enlargement' and so on.

I guess its the robots file I need to change but what is the spider - can anyone help?

I have had MS live's search engine visits for other unrelated searches on cars and drugs from MS live before but I draw the line here and wish to block them all now.


Receptional Andy

Msg#: 3442381 posted 10:56 am on Sep 7, 2007 (gmt 0)

Hi Tidal2,

It's probably worth looking at the thread strange referrer activity [webmasterworld.com] which includes a reply from an MSN rep.

Unfortunately, the spider resulting in the adult terms you see is not the Live search engine spider (indeed, it does not identify itself as a spider at all). Your only solution if you genuinely don't care about MSN referrals is to block the IP addresses used by this 'test'.

The suggestion in the thread I mentioned is that blocking this activity could result in you dropping out of live results entirely, but you probably have a better chance of staying in the results and still blocking the unwanted activity by banning the IPs, which I imagine would be the best of both worlds.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved