weird robot

216.231.54.187 66.14.117.87 & C

[berno72]

This ip spidered since days are spidering my sites.

Any idea about them?

Mozilla/4.05 [en] (X11; I; Linux 2.0.36 i686)
IP Address: 216.231.54.187

Mozilla/4.05 [en] (X11; I; Linux 2.0.36 i686) visited using IP Address: 207.46.170.126

Mozilla/4.05 [en] (X11; I; Linux 2.0.36 i686) visited using IP Address: 66.14.88.168

Mozilla/4.05 [en] (X11; I; Linux 2.0.36 i686) visited using IP Address: 66.14.117.87

One of them resolves to Microsoft

NetRange: 207.46.0.0 - 207.46.255.255
CIDR: 207.46.0.0/16
NetName: MICROSOFT-GLOBAL-NET

Microsoft using Linux? Does it sound weird!

[wilderness]

berno
Welcome to Webmaster World.

Since their were no replies I'll attempt to assist.

The UA fairly common although I couldn't tell you what OS it is. Some of the others may?

Were the four visitors you defined spaced close together in their visits (same day, within a few minutes) or were they from different days?

[jim_w]

You think that is weird, check this out...

I have banned 131.107.

131.107.3.86 - - [28/May/2003:10:57:09 -0500] "GET / HTTP/1.0" 403 274 "-" "FavOrg"
207.46.228.98 - - [28/May/2003:10:57:10 -0500] "GET /favicon.ico HTTP/1.0" 200 318 "-" "FavOrg"

Why would they want my icon? I have also banned 207.46. now.

Note how quick the IP changed.

[wilderness]

jim
It's not unusual to have another IP and even a blank UA for a favicon request.

As I mentioned in another thread, I've been working on putting my accumualted deny/allow/records into a database and have seen some wierd results all part of that MS promotion of their business net.
I've given the 131 a free roam of my pages. They really aren't overdoing and are only grabbing a few pages daily.
I'm still not positive were they are headed. It's something to keep an eye on.

Especially in light of my new leaf ;)

[jim_w]

Yea, but my icon is real crappy! :-))

I have to ban them, they may be competing with me.

[Key_Master]

FavOrg scans the Web sites in your Favorites list for favicons. Not something I would ban.

[pcmag.com...]

[jim_w]

Yea, I didn't ban FavOrg, just the MS Ip's. They must have had a problem with 131.107. being banned to set it up to switch to another IP.

I also don't recall that 131.107.xxx.xxx ever visiting my site before. Just the other 2 that was talked about in another thread. It's strange. Maybe has something to do with beta longhorn or something. And I never had 207.46.xxx.xxx show up, even when the other IP's got 403's till now, so maybe this is something new they just started.

[balam]

Why would they want my icon? I have also banned 207.46. now.

They want your icon because they have bookmarked your site, and you have banned them because they're interested in your site.

I have to ban them, they may be competing with me.

If Microsoft is competing with you, I'll be sure to drop flowers on your Net-enabled, MS-branded coffin... ;)

It's not unusual to have another IP and even a blank UA for a favicon request.

For example, Apple's Safari browser (annoyingly) fetches icons with no referrers or UAs, assumedly for tabbed windows a la Mozilla & Opera.

They must have had a problem with 131.107. being banned to set it up to switch to another IP.

Ok, think logically for a moment... If MS knows they're banned on 131.107. because they're having trouble (or being troublesome), then why switch to 207.46. just to fetch an icon and not (attempt to) refetch "/"?

Note how quick the IP changed.

Are you familiar with marketscore.com? They "Increase Internet Speed and Defend Against Email Viruses All at No Cost to You!" for the right to monitor your net habits (anonymously, of course! ;).

A single visitor employing their service visits your site to fetch HTML files from IP ranges...

66.119.32.0 - 66.119.47.255
170.224.224.(37) - 170.224.224.(135) (Known lower & upper limits)
216.148.244.32 - 216.148.244.63
216.148.246.128 - 216.148.246.159

(And there could be others...)

...as well as their own IP to fetch graphics. (And if the visitor is an AOL customer, it means another fistful of IP addresses just for your one page!)

Just two examples of visitors coming in on multiple IPs in one session, and not a sign of something wicked this way comes...

I also don't recall that 131.107.xxx.xxx ever visiting my site before.

Perhaps no one at MS has been interested in your site before now? I've never had a visitor from the Central African Republic, but I'm not about to view the first visitor suspiciously. (In fact, there was a time when my site hadn't had any visitors. Goes to show there's a first time for everything.)

Ok, to keep me topically:

Microsoft using Linux? Does it sound weird!

To paraphrase many a military commander, 'To defeat the enemy, you must become the enemy.'

balam

[wilderness]

I've never had a visitor from the Central African Republic, but I'm not about to view the first visitor suspiciously.

I would :(
They likley be associated with all the Nigeria of Zaire nonsense ;)

[balam]

I would :(
They likley be associated with all the Nigeria of Zaire nonsense ;)

Hah! Granted...

Ok, the CAR is perhaps a bad example, given the money spam from Nigeria, so howzabout hold-outs like Bhutan, Norfolk Island, and, uh, Iraq? ;)

Actually, I've had a fair number of visitors from Nigeria and only one seemed to be fishing for email addresses. From looking at the logs, it would seem that some genuine research (by students?) was being conducted...

balam

[jim_w]

balam

Yea, but my icon is real crappy! :-))

They want your icon because they have bookmarked your site, and you have banned them because they're interested in your site

It was an attempt at humor. Perhaps a poor one, but never the less an attempt.

If Microsoft is competing with you, I'll be sure to drop flowers on your Net-enabled, MS-branded coffin... ;)

The good news is they aren’t yet. But they could modify a new product of theirs to do what our product does. And yes, this scares the heck out of me. So I will not give them any information. Training per employee in the area is about $50,000.00 per. I know what they can do. I have been following Bill Gates and his business tactics since around 1983.

Ok, think logically for a moment... If MS knows they're banned on 131.107. because they're having trouble (or being troublesome), then why switch to 207.46. just to fetch an icon and not (attempt to) refetch "/"?

Exactly my point. Unless FavOrg has something goofy in it that makes it to that. But how could an IP that has never been to my site need the icon?

Are you familiar with marketscore.com?…

See Above.

Perhaps no one at MS has been interested in your site before now?

You need to read the previous threads.

…but I'm not about to view the first visitor suspiciously…

Perhaps if it was just a company’s IP and not the IP of a country, in addition to, the fact that the said company could be a competitor, would change your values.

To paraphrase many a military commander, 'To defeat the enemy, you must become the enemy

Humm, what commander said that?

[edit]

Microsoft using Linux?

Maybe MS is coming out with their own Linux version of Windows;-)[/edit]

[balam]

Humm, what commander said that?

To paraphrase many a military commander...

It's snide to ask, but you do know what "paraphrase" means, yes?

Regardless:

"18. Hence the saying: If you know the enemy and know yourself, you need not fear the result of a hundred battles. If you know yourself but not the enemy, for every victory gained you will also suffer a defeat. If you know neither the enemy nor yourself, you will succumb in every battle." - Sun Tzu, The Art of War, Chapter 3 [ca. 512 BC]

"But to exercise the intellect the prince should read histories, and study there the actions of illustrious men, to see how they have borne themselves in war, to examine the causes of their victories and defeat[...]" - Nicolo Machiavelli, The Prince, Chapter 14 [1513 AD]

(If Microsoft comes calling...)

The good news is they aren’t yet.

"Without preparedness superiority is not real superiority and there can be no initiative either. Having grasped this point, a force which is inferior but prepared can often defeat a superior enemy by surprise attack." - Chairman Mao Tsetung, "On Protracted War", Selected Works, Vol. II [May 1938]

"I applaud the leaders and employees at the FBI and CIA for beginning essential reforms. They must continue to think and act differently to defeat the enemy." - President George W. Bush, Address to the Nation, [June 6, 2002]

'To defeat the enemy, you must become the enemy.' It's a flexible phrase - substitute "opposition" or "competition" for "enemy"; "listen", "learn from", "watch" for "become" - and a most effective strategem employed for thousands of years by the military, by business, by sports coaches and... webmasters.

Webmasters?! Re-read the Sun Tzu quote but think about keywords & competing websites...

I have been following Bill Gates and his business tactics since around 1983.

I read somewhere that Bill has a copy of The Prince on his bookshelf, and I've never doubted it.

But how could an IP that has never been to my site need the icon?

The specific IP address may have never shown up in your logs before, but that doesn't mean the visitor hasn't been to your site before. That's why I bring up AOL & Marketscore; two examples of a single visitor showing up via multiple IPs (during a single given session).

Perhaps if it was just a company’s IP and not the IP of a country, in addition to, the fact that the said company could be a competitor, would change your values.

I doubt there's a whole country that comes through via a single IP, but perhaps you're smarter than I am and know better. I'm a firm believer that any banning throws out some babies with the bathwater and I don't mind competitors visiting, because I "[k]eep [my] friends close, but [my] enemies closer," ("Michael Coreleone" in The Godfather, amongst others).

Since I'm so full of... ;) ...quotes & proverbs, a final warning proverb to all webmasters looking for #1 in the SERPs...

"If you want a place in the sun, you've got to expect a few blisters." [Anon]

balam

[jim_w]

It's snide to ask, but you do know what "paraphrase" means, yes?

Opps, my fault, speed reading, or sped reding. Sorry. No wonder I couldn’t place it :-)

The specific IP address may have never shown up in your logs

Did you read the other threads about MS bots, et. al.? Once when they came to my site, they signed up for my newsletter. When they were on the page, 207.xxx.xxx.xxx did not fetch the graphic logo on my site. As a matter of fact, 207.xxx.xxx.xxx has never shown up on my site till then. You are right, there are a lot of ways for the IP’s to be misleading, I don’t think this is the case. I may be wrong, but it is my opinion anyway. And in the overall scheme of things, it doesn’t really matter for maybe it wasn’t MS at all but someone spoofing their IP’s.

I doubt there's a whole country that comes through via a single IP

I never said single IP.

I don't mind competitors visiting

Great, that is indeed your choice, and I will not fault you for your way of doing business, do you think you should fault mine?

Since I'm so full of... ;) ...quotes & proverbs

BTW, I was impressed for what that is worth.

Maybe MS is coming out with their own Windows version of Linux ;-) It works both way!

[balam]

Did you read the other threads about MS bots, et. al.?

Oh, I've been acutely aware of MS's new bot since...

131.107.65.225 - - [20/Mar/2003:06:10:26 -0800] "GET /file.shtml HTTP/1.0" 200 16831 "-" "-"

As you can see, this is before they even bothered using an UA string...

I will not fault you for your way of doing business, do you think you should fault mine?

It's my turn to apologize. I assume (which is a can of worms itself, ;) that you're referring to my comments on visitors being banned. Truth be told, I have no clue about 'your way of doing business,' and if it involves banning visitors, well... then it involves banning visitors, simple as that. (And your business model must be working to some extent if you can afford to drop $50K to train an employee. Whatever it is you do, do ya have an opening for me? ;)

The point I was trying to make was that when IP ranges are banned, there are some innocent people who pay for the sins of the guilty. I've thought of it as sort of "digital racism"... Case in point, earlier this week I (and many others) were banned from wilderness's web site, all because of the actions of some script kiddie. Now it's certainly wilderness's right to ban folks (and we know he really exercises it! ;) but I just think that some of the brushes we use to paint with are too broad. (In wilderness's case, I think the World Wide Web is more of an "Americans Only Web, thank you very much." ;)

Anyway, I've been waiting for the hammer to drop on the two of us for going on so long while being off-topic, so I'm going to do the unheard of - shut up! :)

I'd be happy to continue the chat via Sticky Mail, if you want...

balam

[jim_w]

131.107.65.225 - - [20/Mar/2003:06:10:26 -0800] "GET /file.shtml HTTP/1.0" 200 16831 "-" "-"

Actually I would have never seen that. I have had a UA of '-' banned for sometime. After 3 years of monitoring and doing statistics, and statistics is our business, you start to see trends in the people that actually purchase your product vs the ones that are looking for free information vs the people that save you entire site off to their local to view later, vs etc, etc, etc.

One may make business decisions based on these trends. For me personally, right now I have more US IP’s banned than all the other countries combined. Why, because I can trace the IP’s banned to actual businesses or individual DSL IP’s that I don’t like due to the fact that they come to my site and graze on the content. A lot I put up with, like from the cookie people and universities, some I don’t. I have had numerous competitors spying on me and I want to make it as hard as possible for them to gain any knowledge from me. Of course all they have to do is go home and log in with an ISP and they are in. You would be surprise though on how many aren’t that smart.

I have a #temp area in my .htaccess for IP’s I’m not sure of. I put them in there until I see them in the error_log with a 403, then I remove the ban and watch to see if they keep the same activity up that I didn’t like. If I do, I put them back into the penalty area till I see them get a 403 twice, than I remove them again. If after 3 times they keep doing it, I ban them permanently. To this day I have a someone with a US DSL IP that even though for 3 months they have gotten a 403, they have some kind of bot disguised as a normal UA getting 403’s. This person doesn’t even know or doesn’t care they are getting 403’s. I try not to ban IP’s because I don’t want to cut off my, etc. For every ban I have in the .htaccess, I have noted what company, products, and/or what they were doing to annoy me. Adds a lot to the .htaccess file, but before I started doing this it was too confusing to keep track of all the people.

Lets face it, there are a lot of unethical people on the net. From every country and region. I put up a free stats calculator to try to get backlinks, within 24 hours of putting it online, I had as many people trying to steal the perl script as I did run it. I know because I rigged the script to pop a 500 if someone tried to run it from off site. The 500 made it easy to see who was trying to steal it. One person even tried to FTP in with the browser to get it.

[jim_w]

I just found this in my log.

207.46.225.252 - - [29/May/2003:13:43:36 -0500] "GET /file.html HTTP/1.0" 403 288 "http://www.google.com/search?hl=en&ie=UTF-8&oe=UTF-8&q=kw+kw+steps" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"
131.107.3.71 - - [29/May/2003:13:43:46 -0500] "GET /images/0bg.gif HTTP/1.0" 403 288 "http://216.239.33.100/search?q=cache:QasrH6bF6WIJ:www.mydomain.com/file.html+kw+kw+steps&hl=en&ie=UTF-8" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"
tide78.microsoft.com - - [29/May/2003:13:43:46 -0500] "GET /css/style.css HTTP/1.0" 403 287 "http://216.239.33.100/search?q=cache:QasrH6bF6WIJ:www.mydomain.com/file.html+kw+kw+steps&hl=en&ie=UTF-8" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"
207.46.225.252 - - [29/May/2003:13:43:47 -0500] "GET /file.js HTTP/1.0" 403 282 "http://216.239.33.100/search?q=cache:QasrH6bF6WIJ:www.mydomain.com/file.html+kw+kw+steps&hl=en&ie=UTF-8" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"
207.46.225.252 - - [29/May/2003:13:43:47 -0500] "GET /images/file_s.jpg HTTP/1.0" 403 296 "http://216.239.33.100/search?q=cache:QasrH6bF6WIJ:www.mydomain.com/file.html+kw+kw+steps&hl=en&ie=UTF-8" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"
131.107.3.71 - - [29/May/2003:13:43:47 -0500] "GET /ba/ncmrban.gif HTTP/1.0" 403 288 "http://216.239.33.100/search?q=cache:QasrH6bF6WIJ:www.mydomain.com/file.html+kw+kw+steps&hl=en&ie=UTF-8" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"
207.46.228.14 - - [29/May/2003:13:43:47 -0500] "GET /images/search1.gif HTTP/1.0" 403 292 "http://216.239.33.100/search?q=cache:QasrH6bF6WIJ:www.mydomain.com/file.html+kw+kw+steps&hl=en&ie=UTF-8" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"
131.107.3.74 - - [29/May/2003:13:43:48 -0500] "GET /images/file.gif HTTP/1.0" 403 305 "http://216.239.33.100/search?q=cache:QasrH6bF6WIJ:www.mydomain.com/file.html+kw+kw+steps&hl=en&ie=UTF-8" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"
207.46.228.31 - - [29/May/2003:13:43:48 -0500] "GET /images/k_factor.gif HTTP/1.0" 403 293 "http://216.239.33.100/search?q=cache:QasrH6bF6WIJ:www.mydomain.com/file.html+kw+kw+steps&hl=en&ie=UTF-8" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"

[rbs10025]

It's someone viewing the Google cache of your page, and the images are being pulled from your site since Google doesn't cache those.

The user's IP numbers translate back to Microsoft, perhaps that's a proxy for the MSN service?

[wkitty42]

key_master,

FavOrg scans the Web sites in your Favorites list for favicons. Not something I would ban.

hummm... i guess something like this...

RedirectMatch (.*)\.ico$ [microsoft.com$1\.ico...]

would really twist them up or something?

OB-)

[Key_Master]

Nope. You can't redirect favicon requests.

Microsoft is a huge corporation with lots of employees, many of which use the Internet while on the job. I've been recording every single request from Microsoft registered IP ranges to all of my sites since 08/03/02 and have yet to record a single questionable log entry. The vast majority of the hits are from employees coming through Google looking for information on a subject or simply doing some shopping.

Just because 207.46.0.0/16 is registered by Microsoft doesn't mean it's used exclusively by the Microsoft Corporation (example, reporters for msnbc.com use 207.46.169.60).

[jim_w]

Key_Master

how did you figure that 207.46.169.60 is used my MSNBC? I need to be able to do that also. Or do you have the blocks broken down?

[wkitty42]

key_master,

the only thing i can say is that since incorporating that
filter, i haven't had any 404s on favicon.ico... i can block
them from my site (as shown) and i can eliminate them for
m$' schit and lack of talking with others about what they
are implementing...

i don't have a problem with favicon.ico, actually... what i
do have a problem with is others /ass/uming that others know
and are willing participants...

FWIW: i've actually thought about creating a favicon.ico for
my site and removing that particular block...

[Key_Master]

jim_w,

I recommend SamSpade. It's an excellent tool for tracking down info on IPs. A nslookup on 207.46.169.60 resolves to proxy1.msnbc.com

[samspade.org...]

wkitty42,

The server will try to redirect the browser to the favicon on the Microsoft site but browsers will not follow a redirection header for a favicon request. No 404 is logged because a file request hasn't been made on your server. As for the rest of your post (the /ass/uming part), I'm lost. I don't understand what you mean. Is it a conspiracy thing? ;)

[wkitty42]

key_master,

the /ass/uming part was about m$ or any other entity thinking that others would be understanding of their setups and creating pages or graphics to fulfill the requests...

as for the favicon.ico requests not being fulfilled, i don't know that i can answer that either... all i know is that after i instituted that blocking code, my logs do not show any requests for favicon.ico that do not get redirected...

[Key_Master]

I'm still lost. Isn't that what being a webmaster is all about- understanding the setups of our visitors and the companies we do business with and creating pages or graphics to fulfill those requests? Or did I misunderstand the statement.

[wkitty42]

key_master,

i don't know if you misunderstood the statement or not... however, i have never done business with m$ (directly!) or agreed to their tactics or stratagies... i definitely never talked or read anything from them about any implementation about grabbing favicon.ico when someone bookmarked some page on my site...

[jim_w]

nslookup. Silly me, I have an old freeware version from 1996 local, your right of course, that is what it resolves to. I always rely on arin way too much.