Bot hitting alot

[Rivethed]

w018.z065104122.lax-ca.dsl.cnc.net
Hits 8,916
Errors 606
Visits 32
KBytes 13,333K
Time frame October 01 - October 06

Anybody know this bot? And what it is for?

[wilderness]

w018.z065104122.lax-ca.dsl.cnc.net (65.104.122.18):

[webmasterworld.com...]

See this thread and the last two lines of my reply

[Rivethed]

OK, so banning them should be a good thing. OK, this is my first week playing with logs. How do I ban them? I do not have a .htaccess file that I have seen mentioned, and the server is dedicated, but offsite. I have control of our web pages, and access (finally) to d/l logs, but that is it.

Anyone?

Also, why would an airline company harass like that?

[wilderness]

What makes you believe it's an airline?

65.104.122.18
is part of the backbone provider range
XO Communications
NetRange: 65.104.0.0 - 65.107.255.255
and could be anybody.

A simple beginning
[webmasterworld.com...]

[Rivethed]

Sorry for the assumption, LIving where I live, you see "lax" and you automatically think airport, airline, airplane.

[BlueSky]

wilderness, I see you ban those whose UA is empty or a dash. A few have shown up in the middle of my site with just dashes for both referrer and the UA. They haven't done anything bad yet, but I was wondering if I banned these types of UAs is there any scenario that would catch innocent humans as well. Everyone else seems to always have a UA.

LAX. Los Angeles, CA How's Arnie? lolol

[Rivethed]

IS there anywher eto get a current list of "bad bot" IP's. I am getting alot of weird hits from 'unknown DNS or no reverse lookup not set-up'. accounts for about 5% of my hits right now, adn that is an average of 800 a week. I try to use 'network tools' dot com, but it starts tracing and then the hops start to error out (usually areound hop 12 or 13). If they are just user IP's, I do not wnat to ban them, but is they are spiders belonging to undesirables, I want them gone.

ONe more thing, without programming experience, how do I set up a "spider-trap"?

"Anyone, anyone..." {think ben stein}

<off topic> "Join Ahhhnold." 'He vants to fondle Sa-cramen-Do. HE vants to govern for da people.'
Hey,better him than Bustmonte, and no more Davis. Maybe I will be able to afford to stay here afterall. :) </off topic>

[wilderness]

wilderness, I see you ban those whose UA

BlueSky,
I don't recall when I began using it. It's been more than a year and in all that time I've pretty much been in agreement with the majority in advising that it should only be done in rare instances (deny blank UA, actually not only blank, includes "-" as well).)

Initially the only real SE bot it affected was Lycos and I didn't get much traffic from them anyway.
Until recently, the instances of an average visitor using a blank UA have been minimum. In most instances, the use of a blank UA is used by what might be termed non-desired visitors, at least as related to MY WEBSITES.

Recently, I've had a rash of HEAD (I've been considering for some time denying head requests) requests from AOL users (AOL users are around 30% of my traffic and subscribers) which contain both a DASH refer and dash UA.
Some other IP's as well.
It might be part of a software the visitors are using, which makes them aware of the UA option or the software may do it automatically. In any event, it's not appropiate for my sites.
I may have to change my outlook on this at some later date.

A while back I trialed denial of "Digital Ext."
Later relizing that somebody who had previously marked their browser to store pages for offline viewing and instituted automatic updates could have done so without being aware of the long range capability of what they were doing. In addition, I realize that a visitor would have to go through their accumulated Favorites/Bookmarks and change each one back to the default setting to remove the "Digital Ext."
I removed the deny.

Don

CA How's Arnie?

Arnie Almahurst? ;)

[BlueSky]

wilderness: I don't get very many AOLers probably because my site covers a technical subject. It's interesting you're seeing a bunch of HEAD requests with dashes in the referrer and UA from them. So, am I. About half the AOLers I'm getting right now are this way.

I have DigExt on ban. I've seen it block a few innocent folks, but I'm leaving it that way until I finish my throttle script to put a short leash on those who don't trip the trap. I'm convinced one attacker is using multiple IPs. After tripping the trap twice, he finally figured out what caused it and then sent in another bot from a different IP which bypassed the trap when sucking down pages. I haven't seen him in a couple days after repeatedly eating 403's. I've banned the entire nets of the ISPs he was using. When he comes back though, he will have a nice surprise.

Rivethed: I don't think there's a good list of bad bot IPs. There is one for UAs which is a work in progress, but it's pretty good already. You can see it here: [webmasterworld.com...] There's a link to a spider-trap script in that thread written in Perl. You can do a search on this site to see if others have encountered nasty guys from the same IPs that are bothering you. There's an awful lot of bots running about. So, it's possible you may be getting some that others haven't seen.

[Rivethed]

OK, I gave up on tracking programs and am using excel now, And I can see the UA's:

Questionables in my logs, if you care to comment (replace my url wiht example, spaced the ip:
________________
65. 165. 198. 19--[01/Oct/2003:07:22:55+0800]GET /pdf /n03_body-components. pdf HTTP/1.020657577-contype
65. 165. 198. 19--[01/Oct/2003:07:22:55+0800]GET /pdf /n03_body-components. pdf HTTP/1.0200162513http://www .EXAMPLE .com/Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt)
________________
200. 47. 155. 132--[01/Oct/2003:08:20:01+0800]GET / HTTP/1.120018706-IBSBand

________________

(found flaw in excel, will only open up about 3 hours of logs, I have a l;og file that is almost a gig, covering 8 days, any 'freeware' suggestions?)

[Rivethed]

Well, I deep searched the net, and it led me back to WebmasterWorld, and I found the "contype" here

[webmasterworld.com...]

but I still would like to know what the third 'hit' is.

[wilderness]

contype

Rivethed,
I have plenty of PDF's online.
Currently and only for a few more days, nearly two hundred as part of two annual horse sales.

contype, comes up more than not after the initial non-html content load. Other times it doesn't come up at all.
The PDF plug-in to browsers along with slow internet connection speeds can create some unusual problems for plug-ins.

I've had some browsers load a PDF as many as 15-20 successive times.
My response to this reload and even though the the visitor may not be aware of, or even planned this excess intentionally, is to deny their IP range. They are responsible in any event.
This reaction is extremely overbearing for a webmaster. However as I've noted on many occassions, my websites are an exception from the norm in their very narrow market-desired visitor traffic.

[BlueSky]

Today, I signed up for an accelerator service with my ISP, and they happen to use a proxy server for it. Afterwards I worked a bit on my site. When I looked in my logs later on, I saw a whole bunch of HEAD requests with dashes in both the referrer and UA fields that the proxy had submitted on its own. Having seen this, I now think the same is happening with the AOLers. So, double dashes can indeed come from just regular users who aren't doing anything malicious.