Forum Moderators: coopster

Message Too Old, No Replies

Serving different HTTP response to different spiders

Want to send a 410 to Google and 404 to Bing

         

Karma

7:36 pm on Sep 8, 2009 (gmt 0)

10+ Year Member



I'm trying to remove a bunch of URLs from the Bing index that have been there for ages.

I need to keep the current HTTP 410 response for all spiders/bots except Bing, which I need to serve a 404.

Can (should) I do this using PHP, and if so...how?

Karma

8:44 pm on Sep 8, 2009 (gmt 0)

10+ Year Member



Ok, after a little investigation it appears that the only way proper way is to retrieve the user agent string. From another post on WebmasterWorld it looks like this user-agent can be any of the following:

msnbot/1.0 (+http://search.msn.com/msnbot.htm)
msnbot/1.1 (+http://search.msn.com/msnbot.htm)
msnbot/2.0b
msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)
msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
MSNBOT_Mobile MSMOBOT Mozilla/2.0 (compatible; MSIE 4.02; Windows CE; Default)
msnbot-Products
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; livebot-searchsense/0.1; +http://search.msn.com/msnbot.htm)

All of which contain the string 'msnbot', so I'm guessing that I'll just scan the string for 'msnbot' and if it's found serve a 404 instead.