Welcome to WebmasterWorld Guest from 126.96.36.199
Forum Moderators: bakedjake
# uname -a
Linux mysite.com 2.6.9-22.0.2.ELsmp #1 SMP Tue Jan 17 07:10:04 CST 2006 i686 i686 i386 GNU/Linux
Firewall built using /etc/init.d/iptables-live by my (very well experienced, very helpful) host. The very first lines in the
LOG all -- 0.0.0.0/0 0.0.0.0/0 state INVALID LOG flags 0 level 4 prefix `Invalid packet: '
DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID
fgrep -c 'Invalid packet' /var/log/messages*
fgrep '188.8.131.52' /var/log/messages
Jan 23 17:14:26 olivia kernel: Invalid packet: IN=eth0 OUT= MAC=00:20:ed:82:c6:f5:00:30:f2:10:b0:00:08:00 SRC=184.108.40.206 DST=220.127.116.11 LEN=40 TOS=0x00 PREC=0x00 TTL=231 ID=13832 DF PROTO=TCP SPT=1109 DPT=80 WINDOW=0 RES=0x00 ACK RST URGP=0
Jan 23 17:14:28 olivia kernel: Invalid packet: IN=eth0 OUT= MAC=00:20:ed:82:c6:f5:00:30:f2:10:b0:00:08:00 SRC=18.104.22.168 DST=22.214.171.124 LEN=40 TOS=0x00 PREC=0x00 TTL=231 ID=13833 DF PROTO=TCP SPT=1110 DPT=80 WINDOW=0 RES=0x00 ACK RST URGP=0
It's pretty common for an attack to take the form of a 'packet injection' whereby the attacker tries to gain control of a system by guessing the packet stream's numbering sequence, and then to take control of the packet stream by injecting a forged packet with an address within the guessed range. A successful injection could cause a valid request to become corrupted and possibly result in the compromise of the machine. (Obviously more details are available in lots of different places on the 'net ...)
Sounds like it would be a good idea for you to keep dropping those packets. Your real users probably won't be sending invalid packets at a rate that would cause their valid requests to fail.
BTW, your sample IP came from Indonesia ... according to APNIC [apnic.net], which is fairly popular territory from which this type of abuse flows ...
It is the sheer scale of the Invalid packets that is defeating my mind, and causes me to wonder whether I have hit some sort of bug in the Firewall or the kernel. Consider the following:
# egrep -c "Jan 24(.*)Invalid packet(.*)PROTO=TCP(.*)DPT=80" /var/log/messages
# fgrep -c "24/Jan" /var/log/httpd/co*-access_log
Now, if you are saying that more than 1 in 10 of all visitors are either trying to hack the site, or have such badly mis-configured TCP stacks that their page requests are utterly Invalid, and that that is just how life is today, then, ... well, I shall be flabbergasted, but have to accept it.
But surely not...
98,203 hits on the website from 5,258 humans also produced 5,339 bad packets with 603 different IP-addresses
Why not? Thinking aloud, if you *are* being attacked by someone interested in executing a packet injection hack, you might end up with all of the 'real' requests being satisfied with your 'real' visitors none the wiser. For each 'real' visitor and their accompanying connection, the attacker (who may be port monitoring your system) makes at least one attempt to hijack the packet stream ... and fails ... resulting in at least one invalid packet per 'real' visitor.
The nature of a packet injection attack is such that an existing packet stream and a valid connection must be established, then comes the port monitoring to view that packet stream's configuration, then comes the guessing at the next packet number in the sequence (while valid packets are flowing freely), then comes the attempt to inject the forged packet and finally the file buffer is overflowed or whatever 'crowbar' is being applied to compromise the system after a successful injection. An incorrect guess at the packet's sequence number will cause the forged packet to simply bounce off (invalid packet because it doesn't fit into the request sequence), as the original request/exchange is handled normally.
I must confess that my own servers show none of this activity (0 count in all messages logs), and we get bombarded with various kinds of attacks every day. Maybe nobody is trying a packet injection on us, or maybe PortSentry is keeping anyone from monitoring our ports successfully, or maybe I'm off base.
Hopefully, conjecture may lead to truth ... ;)
As an aside, the key to packet injection is guessing the next
assignment correctly, or at least guessing within a range that has not yet been qualified and is still within range of the last packet of the sequence.
In your example, note the two sequential IDs. Are those the only two entries for that IP?
Lastly, perhaps, the 'Invalid' connection state your rule is keying on refers to packets that are 'not part of any connection', according to my iptables, so ... hmmm ... still feeling pretty good about attempted packet injection attacks from zombie servers somewhere in South Asia.
I took 3 hours out to look closely at 31 IPs from the total of 504 from Tue Jan 24 (one day in the life of a web-server). Only 2 IPs did not appear in the Apache logs. Of the other 29 IPs, 22 supplied Invalid packets well after the last hit, and only 7 were blocked during mid-page load (they all got the html page; it was .png files, etc that suffered).
What is very interesting in this is that half of the IPs made only 1 successful page-request, then a second request which totally died (because it was blocked due to Invalid packets). The single-page bounce-rate stands at 53% on my site. That has worried me immensely, and it is some (bitter) comfort to realise that it is so high due to mis-configured machines.
KeepAlive is active on my Apache webserver. Does any one know if there are any problems with this?
Tue Jan 24 - one day in the life of a web-server:
sample of 31 of 504 TCP:80 IPs blocked due to Invalid Packets:
(603 total when UDP packets included)
2 have no entry in the access_log
15 make 1 page request
8 make 2 page requests
1 make 3 page requests
3 make 4 page requests
1 make 5 page requests
1 make 8 page requests
(ignoring the 2 no-access-log, and considering the 29 others):
No of Invalid packets AFTER successful page request: 22
No of Invalid packets DURING successful page request: 7
MSIE 6.0; Windows NT 5.1 20
MSIE 6.0; Windows NT 5.0 5
MSIE 5.0; Windows 98 1
MSIE 6.0; Windows 98 2
Firefox/1.5 Win98 1
Windows XP 73.8%
Windows 2k 10.7%
Windows 98 8.3%
IP-Address - No of Invalid packets - Page-access Time - Browser + OS - Invalid Packet Time (first-last)
126.96.36.199 - 2 packets to tcp(80) 12:06:03 MSIE 6.0; Windows NT 5.1 - 12:09:57
188.8.131.52 - 12 packets to tcp(80) 23:50:35 MSIE 5.0; Windows 98 - 23:53:19-23:54:34
184.108.40.206 - 1 packet to tcp(80) 12:48:55 MSIE 6.0; Windows NT 5.0 - 12:56:17
220.127.116.11 - 10 packets to tcp(80) 21:20:03 MSIE 6.0; Windows NT 5.1 - 21:22:15-21:27:06
18.104.22.168 - 2 packets to tcp(80) - Jan 22 22:25:01
Jan 22 23:30:29
Jan 24 12:56:43
Jan 24 13:26:07
22.214.171.124 - 9 packets to tcp(80,80) 18:03:53 Firefox/1.5 Win98 - 18:07:53-18:17:53
126.96.36.199 - 68 packets to tcp(80,80) 11:38:43 MSIE 6.0; Windows NT 5.0 - 11:40:09-11:53:42
188.8.131.52 - 23 packets to tcp(80,80) 17:17:19 MSIE 6.0; Windows NT 5.1 - 17:19:36-17:31:51
184.108.40.206 - 63 packets to tcp(80,80) 13:33:32 MSIE 6.0; Windows NT 5.1 - 13:35:47-13:49:28
220.127.116.11 - 12 packets to tcp(80) 19:27:40 MSIE 6.0; Windows NT 5.1 - 19:30:41-19:38:52
18.104.22.168 - 2 packets to tcp(80) 23:07:40 MSIE 6.0; Windows NT 5.1 - 23:19:03
22.214.171.124 - 2 packets to tcp(80) 20:00:33 MSIE 6.0; Windows NT 5.0 - 20:07:31
126.96.36.199 - 2 packets to tcp(80) 20:47:48 MSIE 6.0; Windows NT 5.1 - 20:50:00
188.8.131.52 - 42 packets to tcp(80) 17:21:21 MSIE 6.0; Windows 98 - 17:21:41-17:34:38
184.108.40.206 - 1 packet to tcp(80) 12:59:41 MSIE 6.0; Windows NT 5.0 - 13:09:53
220.127.116.11 - 2 packets to tcp(80) 09:42:01 MSIE 6.0; Windows NT 5.1 - 09:47:24
18.104.22.168 - 2 packets to tcp(80) 16:12:36 MSIE 6.0; Windows NT 5.0 - 16:20:36
22.214.171.124 - 9 packets to tcp(80) 07:41:08 MSIE 6.0; Windows NT 5.1 - 07:50:48-08:01:06
126.96.36.199 - 6 packets to tcp(80) 13:57:12 MSIE 6.0; Windows NT 5.1 - 14:03:33-14:11:17
188.8.131.52 - 1 packet to tcp(80) 10:45:35 MSIE 6.0; Windows NT 5.1 - 10:47:52
184.108.40.206 - 11 packets to tcp(80) 13:22:32 MSIE 6.0; Windows NT 5.1 - 13:29:23-13:29:59
220.127.116.11 - 4 packets to tcp(80) 07:58:23 MSIE 6.0; Windows NT 5.1 - 08:01:23-08:06:37
18.104.22.168 - 19 packets to tcp(80) 07:36:04-07:37:00
22.214.171.124 - 1 packet to tcp(80) 12:00:04 MSIE 6.0; Windows NT 5.1 - 12:04:25
126.96.36.199 - 12 packets to tcp(80) 05:18:36 MSIE 6.0; Windows 98 - 05:21:08-05:22:31
188.8.131.52 - 1 packet to tcp(80) 02:34:12 MSIE 6.0; Windows NT 5.1 - 02:43:35
184.108.40.206 - 10 packets to tcp(80) 09:30:49 MSIE 6.0; Windows NT 5.1 - 09:40:44-09:41:23
220.127.116.11 - 13 packets to tcp(80) 19:30:04 MSIE 6.0; Windows NT 5.1 - 19:33:30-19:34:10
18.104.22.168 - 2 packets to tcp(80) 03:16:39 MSIE 6.0; Windows NT 5.1 - 03:26:10
22.214.171.124 - 2 packets to tcp(80) 08:21:20 MSIE 6.0; Windows NT 5.1 - 08:28:27
126.96.36.199 - 1 packet to tcp(80) 18:01:26 MSIE 6.0; Windows NT 5.1 - 18:02:46
I find it difficult to believe that (more than) 1 in 10 are doing that
Aahh ... but more than likely NONE of your 'real' visitors are doing that. Let me explain ...
Your visitors are simply ... visiting. They are not the problem. They provide the conduits.
If my theory holds, the hacker(s) is lurking ... hanging out around your ports, sniffing the 'real' visitors' packets for info. With each 'real' visitor, the un-related hacker tries to inject a forged packet by piggy-backing in on the 'real' visitors connection. They are not requesting anything, they are trying to wedge themselves into the valid packet streams being provided by your 'real' visitors.
If you logged ALL packets (yikes!), you'd see a valid stream something like:
where the OK packets are the 'real' stream and the BAD packet is the injection attempt. It should (in an ideal hack) look nearly exactly like the 'real' packets. The failed attempt has no impact on the 'real' visitor's experience, as all of the valid request's packets make it through just fine.
A forged packet is pretty simple to make out of an 8-bit file, and its transmission profile can contain any kind of information the hacker wants to send. Better hackers make their packets look like 'real' traffic ... with a user agent and everything ... preferably identical (except for the guessed ID= attribute) to 'real' packets being observed in any particular stream. When a packet is suspect ... all of the data it provides is suspect, too. If a packet looks unusual, it draws attention to itself. Much better for it to look like a companion packet in a stream of valid packets that had a little trouble getting validated.
Your list contains invalid packets from Chile, The Netherlands, Indonesia, London, Tehran (more than once), and several other notoriously-porous blocks. Sure, there are a few from the U.S. or from other 'trusted' blocks, but ... hey ... there are loose installations all over the world.
Why would a 'real' packet stream contain invalid packets?
1) Mis-configuration of the protocol stack(s) ... if not on the client end then on the server end.
2) Timeout issues that may close a connection erroneously before all packets have been received.
3) Random packets that really aren't part of any valid connection ... maybe they've been drifting around in cyberspace for awhile and finally make it to the destination after the connection has been closed.
4) Maybe the invalid packets are forged.
Of these (and I'm wide open to others), #3 is kind of possible, but not likely at the rates you noted, and #4 is the most likely, given the patterns you describe.
It still smells like an attack or two (or many).
I'm adding your logging function to my logs, to see what's going on with our otherwise cool servers, and to compare with your experience so I may understand it better.
The failed attempt has no impact on the 'real' visitor's experience
The most common symptom (in the Apache logs) was just the html page, style sheet + one graphic file loaded (there are 7-10 graphics files required). Other symptoms were multiple requests for the same file (the style sheet comes to mind, but it varied), or a duplicate 200 load of graphic files for a second page, then 304s on the 3rd page. Then there are all the folks with Invalid packets 4-10 minutes after the first page, and nothing whatsoever in the apache logs. There isn't the slightest doubt that these folks were having problems getting a full-page load (html + all helper files). The big question, of course, is why?
The big question for me is whether I can trust the Firewall+kernel diagnosing an invalid packet? Could it be a bug in either, or in Apache?
I am fully willing to keep the block if I cannot find a bug elsewhere. The one change that perhaps need making is not to BLOCK but rather to REJECT the packet. The latter sends a message which will allow the client to recover promptly. The former (current behaviour) does not inform, and the client will then have to rely on the TCP timeout, which I understand to be 30 secs. The current behaviour will lead to a very poor perception of my website amongst more than 1 in 10 visitors.
Another interesting feature is how few Firewall scripts out there include this specific coding (block on Invalid), and that also begs the question why?
Is this server running through a proxy?
Is this an all-day-every-day situation? Your inital post seemed to include numbers that started low, peaked and then dropped off. Maybe that's the pattern of your traffic? Are the valid:invalid ratios consistent over time?
The data you provided (if we can trust it) indicates just a few dropped packets per request. 2 bad packets shouldn't keep a visitor from enjoying all of your page elements.
I dunno ... but I still can't get off of the attack scenario. Since I modified my iptables, this morning, I haven't seen any bad packet activity at all. There have been many attack attempts amidst many thousands of requests, and my PortSentry [sourceforge.net] has been deflecting attacks all day, but no invalid packets. Of course, if PortSentry is being completely successful (as usual) then any attempt to lurk and port monitor is being thwarted, anyway, so there is no opportunity to guess at the packet sequence and/or launch an injection attack. Without the lurking there can be no guessing.
Maybe the attacker (if they exist) is trying to create their own valid connection and ride their own wave into your server? The concept is the same, even if the disguise and execution is pretty weak.
I'm not trying to be dense or too paranoid, but all of your evidence still seems to support the attack scenario, IMHO. It's a tough one, all right.
Although it is a far out idea ... maybe there's an issue with the NIC or its interface at your server or with some other piece of hardware having intermittent problems? Not pretty, but I'm at a bit of a loss.