Forum Moderators: mack
Just saw this guy, fell into a spider trap:
131.107.137.47 - - [11/Apr/2003:01:31:08 -0600] "GET /a/deep/link.html HTTP/1.1" 200 12589 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705)"
No referer, came in on a deep link (like from a SE), and d/l pages but no images. After about 5 hits, he tried to grab a trap, and got banned. Grabbed a page every 5 secs or so...
IP resolves to Redmond.... did Bill just get himself banned?
dave
131.107.163.49 - MicrosoftPrototypeCrawler (please report obnoxious behavior to newbiecrawler@hotmail.com
and
131.107.65.225 - Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.2;+.NET+CLR+1.1.4322)
Although I am at a loss as to what exactly it is doing there, doesn't seem to follow any particular pattern, it will be interested to see what pendanticist learns.
LisaB
Well I hope so, but being the cynic we are, it would be my perception that the only time MS called me back was after I gave them a CC# and they charged me $175.00 to prove to them they had a bug in one of their compilers. (wondering how MS got so rich)
<quote>we'll all speculate our weekends</quote>
Actually the biggest concern I have right now is, are any of the IP’s in 131.107. used by MSN. I already know I need to block their corporate HQ, but I don’t really want to block their SE bot or MSN users. Although, that would probably be fine with AOL and google. God knows I’ve shot myself in the foot enough for one lifetime.
The alleged ‘MicrosoftPrototypeCrawler’ hasn’t been back to see us since 25/Apr/2003:19:25:59 –0500 and they read robots.txt. 2 days before that they tried to snag our policy page and got 403’ed, but it may have already been in a cache somewhere, so it looks like they may be leaving us alone.
If google traffic was their #1 goal, they would probably already have it. They have so much money that they can throw at their top goals, (I don’t have a clue though how they got so much though), that it isn’t even funny. Look at what happened to Netscape. And the Apple law suite was so costly for Apple, I’ll bet Apple wishes they would have spent that money on R&D for their OS in hindsight. They even figured out how to get the upper hand with IBM and WARP.
Someone may find this funny. Even MS uses google. (GRIN)
131.107.3.86 - - [08/Apr/2003:11:34:35 -0500] "GET / HTTP/1.0" 200 31279 "http://www.google.com/search?q=xxx+xxxx&hl=en&lr=&ie=UTF-8&oe=UTF-8&start=10&sa=N" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Q312461; .NET CLR 1.1.4322)"
If they call, see if you can find out what IP’s their SE bot will use. Tell them it so that 90% of the professional webmasters won’t block their SE bot by mistake. (GRIN) I’ve had to contact several SE’s to get the IP’s they use because I also have an AXS log and I like to filter the SE bots out of so that I have stats on just the eyeballs that see the page, and of course the ‘evil doers’ that I can spot right away and ban. I wish all SE bits would publish what IP’s their bots used.
And may your beer not be warmer or yellowier.
Lisa-
131.107.65.225 - Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.2;+.NET+CLR+1.1.4322)
Do you have time and date this happened?
Jim, yes... here's the gist:
Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.2;+.NET+CLR+1.1.4322
showed requests around:
17/Apr/2003:14:09:55
18/Apr/2003:11:58:09
19/Apr/2003:19:58:44
22/Apr/2003:17:24:31
25/Apr/2003:01:03:13
26/Apr/2003:17:41:09
This was obviously a coding error by someone whom no doubt fixed it right away
Or maybe they didn't?
So, has anyone found out what the deal is yet? My site has nothing to do with MS - it's a fansite board that it's crawling right now.
So, has anyone found out what the deal is yet?
Yes and no. Being somewhat impatient, I called them just a few minutes ago (5:00 EST). The receptionist 'did' remember me and stated that whomever is in charge (I assume publicity/legal) underwent some form of surgery Thursday.
As she said to me today, 'when we spoke Friday she had no idea this particular individual was the one who 'clears' any forthcoming information' therefore she couldn't tell me he/she was out.
I believe the 'in charge' speaks to no one in particular, just that this individual must be in the loop with respect to any public discussions/admissions whatever. So, for the moment, let's call him/her Public Relations.
I did re-stress our concerns as unilaterally as I could, saying that "Webmasters from around the World are more than concerned as to the authenticity of the relationship w/MS based solely on the moniker 'MicrosoftPrototypeCrawler as we've seen in our log files."
I wish I had more definitiveinformation, but I do not.
Of course, if Brett wants to make that phone call too.....
Pendanticist.
This is indeed a Microsoft sanctioned crawler!
...it is something Microsoft created and soon instead of having the newbiecrawler@hotmail.com contact for question, it will have a microsoft email address to avoid confusion.
There it is, folks.
Take it and ruuuuunnnnnnn.
Pendanticist.
Thanks for chasing this down...
You might suggest to them - if you haven't already - that putting up a web page with their crawler particulars on it, and including that URL in their UA string would be a good idea. That way, webmasters don't have to wait for an answer, and they won't have thousands of e-mails to answer every day.
Thanks again,
Jim
See also It's Official!
MicrosoftPrototypeCrawler is legitimate! [webmasterworld.com]
<added>
The response I got (after the phone call as noted above) was the snippet posted which came to me as e-mail. It is by no means comprehensive.
Jim, I'll answer your question thru the above thread, as best I can.
As far as I'm concerned this thread is dead.
</added>
Pendanticist.
