homepage Welcome to WebmasterWorld Guest from 54.227.160.102
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
New MSN bot?
adidxbot
dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 3931327 posted 3:32 pm on Jun 11, 2009 (gmt 0)

Just trapped this as an unknown bot. IP resolves to msnbot.

IP: 65.55.214.nnn
UA: adidxbot/1.1 (+http://search.msn.com/msnbot.htm)
Robots: No idea, haven't checked.

The link goes to help.live.com which has been restyled as bing. Not much direct help but I didn't delve through lack of time. If they can't direct me to an exact page for the bot, it stays trapped.

Is it ever going to end?

 

Samizdata

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3931327 posted 4:48 pm on Jun 11, 2009 (gmt 0)

Is it ever going to end?

A two-letter word beginning with "N" and ending with "O" springs to mind.

...

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 3931327 posted 9:09 pm on Jun 11, 2009 (gmt 0)

It's very tempting to use those two letters to msn/live/bing/choose_next_name. I'm in the process of setting up a new logs anaylser and am about to add a rule to eliminate all the q=rubbish spam. I could do without that time-waster!

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 3931327 posted 3:00 am on Jun 12, 2009 (gmt 0)

And now a hit with no UA at all. Completely blank header. Guess what it got?

IP: 65.55.115.nnn

Pfui

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3931327 posted 5:48 am on Jun 12, 2009 (gmt 0)

msnbot-65-55-214-150.search.msn.com
adidxbot/1.1 (+http://search.msn.com/msnbot.htm)

Robots.txt? Yes-BUT

Despite getting this robots.txt --

User-agent: *
Disallow: /

-- "adidxbot" went straight for a subdir content page. Whereupon it was 403'd because only "msnbot" bots are allowed (and only in certain dirs, and maybe not any for very much longer).

enigma1

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3931327 posted 12:42 pm on Jun 12, 2009 (gmt 0)

Yes I saw that too (UA adidxbot), but its rdns resolves to msnbot so I do not block it or anything. As of the robots.txt I don't rely on. You can send spiders right into traps with this, so for me is not a good way to do verification.

GaryK

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3931327 posted 5:02 am on Jun 14, 2009 (gmt 0)

"adidxbot" went straight for a subdir content page.

Same here this past week. It read robots.txt three times in a row and then took whatever it wanted. It was a small site though so it was gone before my bot blockers could kick in.

Different IP Address though:

adidxbot/1.1 ( [search.msn.com...]
65.55.214.182
msnbot-65-55-214-182.search.msn.com

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 3931327 posted 10:13 pm on Jun 15, 2009 (gmt 0)

Just got three hits from three MSN IPs with no rDNS and a UA of msnbot-webmaster.

All IPs were in the range: 65.55.248.150 - 65.55.248.170
UA: msnbot-webmaster/1.0 (+http://search.msn.com/msnbot.htm)
Robots: No idea, haven't checked.
Headers: none

It's possible this was in response to the site owner, who told me she was trying to get her site into bing local (probably on a loser: we're in the UK). Still doesn't mean they can come in like that.

GaryK

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3931327 posted 12:30 am on Jun 16, 2009 (gmt 0)

UA: msnbot-webmaster/1.0 (+http://search.msn.com/msnbot.htm)

It's my understanding this is their webmaster tools bot. It's been around since at least August 2008.

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 3931327 posted 10:37 pm on Jun 16, 2009 (gmt 0)

Then they could at least sort out the headers and get a proper rDNS for it.

In fact it was the site owner trying to get her site verified. It said the site was not valid. Actually, I KNOW the site is valid; whether msnbot can SEE it is different problem. :)

shopvince

5+ Year Member



 
Msg#: 3931327 posted 11:40 pm on Jun 19, 2009 (gmt 0)

The adidxbot/1.1 (+http://search.msn.com/msnbot.htm) bot drove us crazy for the last week or so. It was crawling every SEM keyword URL we had configured in our MSN account. Sort of a "Welcome to Bing" I guess.

It was crawling at 25 pages/second so we bumped the crawl rate down to 1 but it took them about 8 hours to change their rate. What's really odd is that the adidxbot never actually read our robots.txt file! So it must get the info from another bot (MSNBot?). All the IPs were unique (i.e. they weren't the same as any other MSNBot). We still aren't sure what Microsoft is doing with this Bot. Anyone know?

msndude

10+ Year Member



 
Msg#: 3931327 posted 6:54 pm on Jul 6, 2009 (gmt 0)

Hi,

This is our bot. We are currently working to fix the over-crawl issues you are experiencing. I'll update this thread as soon as I have more information.

~B

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved