Welcome to WebmasterWorld Guest from 18.104.22.168
Forum Moderators: goodroi
I guess its the robots file I need to change but what is the spider - can anyone help?
I have had MS live's search engine visits for other unrelated searches on cars and drugs from MS live before but I draw the line here and wish to block them all now.
It's probably worth looking at the thread strange referrer activity [webmasterworld.com] which includes a reply from an MSN rep.
Unfortunately, the spider resulting in the adult terms you see is not the Live search engine spider (indeed, it does not identify itself as a spider at all). Your only solution if you genuinely don't care about MSN referrals is to block the IP addresses used by this 'test'.
The suggestion in the thread I mentioned is that blocking this activity could result in you dropping out of live results entirely, but you probably have a better chance of staying in the results and still blocking the unwanted activity by banning the IPs, which I imagine would be the best of both worlds.