Wanted: Crawler Quality Assurance Engineer

For MSNbot/2.0

[jdMorgan]

This would be funny if it weren't so sad...

65.55.3.209 Fri Mar 19 01:22:32 2010 "GET / HTTP/1.1" 403 666 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.ht[b]m)._"[/b]
Connection="Close" From="msnbo[b]tf([/b]at)microsoft.com"
Accept="*/*" Accept-Encoding="gzip, deflate"
Reason denied="Unknown-invalid-unwelcome-or-spoofed-UA"

Not sure what the trailing period and underscore are needed for on the UA string, and msnbotf doesn't ring a bell, either... Anyway, two failures to match expected values, and kicked to the curb...

IP resolves to msnbot-65-55-3-209.search.msn.com

Jim

[Staffa]

At least at msn they seem to have an apprentice typist something which is lacking at Y! because it came twice today without a UA :)

The IPs were from Y! but no referrer, no UA, no robots.txt so they were sent waltzing.

[jdMorgan]

I should note that the MSNbot QA engineer may have reported to work now -- I saw this reported UA only once, and it has not recurred.

> The IPs were from Y! but no referrer, no UA, no robots.txt so they were sent waltzing.

Does the rDNS of your Yahoo IP address resolve to one of the mysterious and unidentified "ycarN.mobile.spN.yahoo.com" hostnames (where both Ns are digits) by any chance?

I *suspect* this may be a Yahoo! mobile gateway/transcoder/something, but have not been able to verify -- still an open question here at WebmasterWorld, AFAIK. The only HTTP request header it sends is Accept="*/*"

Jim

[Staffa]

It seems to be that mysterious wanderer

log file entry =
2010-03-19 02:23:25 GET /default.asp - 80 - 209.131.38.43 HTTP/1.1 - - - www.example.com 302 0 0 426 55 343

rDNS = ycar2.mobile.sp1.yahoo.com

log file entry =
2010-03-19 02:43:30 GET /default.asp - 80 - 69.147.115.59 HTTP/1.1 - - - www.example.com 302 0 0 426 55 234

rDNS = ycar2.mobile.re3.yahoo.com

[jdMorgan]

That doesn't surprise me... I wish Yahoo! would identify this thing -- either in the UA, or in some easily-findable description of what it is and what it's for.

Jim

[Staffa]

I agree with you and that's why it got a 302, being an unknown entity, but it didn't hang around long enough to find out where that was leading to.

[jdMorgan]

It's back:

65.55.3.209 Thu Apr 8 10:18:25 2010 "GET /" "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._"
Connection="Close" From="msnbotf(at)microsoft.com"
Accept="*/*" Accept-Encoding="gzip, deflate"
Reason denied="Invalid-or-spoofed-UA"

Jim

[Pfui]

LOL. It only took me (mumbles) weeks to realize the UA you detail in this thread is the same one I noted on April 5th here:

MSN's many cloaked bots.
[webmasterworld.com...]

(slaps head:)