Welcome to WebmasterWorld Guest from 54.82.10.219

Forum Moderators: bakedjake

Message Too Old, No Replies

IPTables: Upto 30,000 Invalid packets logged per week

Host-installed Firewall; the number of invalid packets seems outrageous

     
3:26 am on Jan 25, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 7, 2004
posts:660
votes: 0


# uname -a 
Linux mysite.com 2.6.9-22.0.2.ELsmp #1 SMP Tue Jan 17 07:10:04 CST 2006 i686 i686 i386 GNU/Linux

Firewall built using /etc/init.d/iptables-live by my (very well experienced, very helpful) host. The very first lines in the

bad_packets
chain is:
LOG   all -- 0.0.0.0/0   0.0.0.0/0   state INVALID LOG flags 0 level 4 prefix `Invalid packet: ' 
DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID

and this is producing tens of thousands of dropped packets per week:

fgrep -c 'Invalid packet' /var/log/messages*
/var/log/messages:13579
/var/log/messages.1:28311
/var/log/messages.2:29110
/var/log/messages.3:27177
/var/log/messages.4:20807

Here is just one recent IP that had it's packets dropped:
fgrep '222.124.226.200' /var/log/messages
Jan 23 17:14:26 olivia kernel: Invalid packet: IN=eth0 OUT= MAC=00:20:ed:82:c6:f5:00:30:f2:10:b0:00:08:00 SRC=222.124.226.200 DST=80.168.53.204 LEN=40 TOS=0x00 PREC=0x00 TTL=231 ID=13832 DF PROTO=TCP SPT=1109 DPT=80 WINDOW=0 RES=0x00 ACK RST URGP=0
Jan 23 17:14:28 olivia kernel: Invalid packet: IN=eth0 OUT= MAC=00:20:ed:82:c6:f5:00:30:f2:10:b0:00:08:00 SRC=222.124.226.200 DST=80.168.53.204 LEN=40 TOS=0x00 PREC=0x00 TTL=231 ID=13833 DF PROTO=TCP SPT=1110 DPT=80 WINDOW=0 RES=0x00 ACK RST URGP=0

Anyone else have any experience of this? Any similar stories? Will I cause any problems for my users if I remove the DROP code for Invalid packets? All help greatfully received.
6:23 pm on Jan 25, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 20, 2004
posts:1477
votes: 0


There are a few reasons why a packet might be invalid. Likely either the packet is badly formed by some element in the transmission process, or (most likely) it is not synching up with an existing request.

It's pretty common for an attack to take the form of a 'packet injection' whereby the attacker tries to gain control of a system by guessing the packet stream's numbering sequence, and then to take control of the packet stream by injecting a forged packet with an address within the guessed range. A successful injection could cause a valid request to become corrupted and possibly result in the compromise of the machine. (Obviously more details are available in lots of different places on the 'net ...)

Sounds like it would be a good idea for you to keep dropping those packets. Your real users probably won't be sending invalid packets at a rate that would cause their valid requests to fail.

BTW, your sample IP came from Indonesia ... according to APNIC [apnic.net], which is fairly popular territory from which this type of abuse flows ...

9:55 pm on Jan 25, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 7, 2004
posts:660
votes: 0


Thank you, StupidScript, that input is certainly helpful.

It is the sheer scale of the Invalid packets that is defeating my mind, and causes me to wonder whether I have hit some sort of bug in the Firewall or the kernel. Consider the following:

# egrep -c "Jan 24(.*)Invalid packet(.*)PROTO=TCP(.*)DPT=80" /var/log/messages
5339
# fgrep -c "24/Jan" /var/log/httpd/co*-access_log
/var/log/httpd/com-access_log:66110
/var/log/httpd/couk-access_log:32093

So, 98,203 hits on the website from 5,258 humans also produced 5,339 bad packets with 603 different IP-addresses. That's 5.4% of hits and 11.4% of people! That is surely a lot!

Now, if you are saying that more than 1 in 10 of all visitors are either trying to hack the site, or have such badly mis-configured TCP stacks that their page requests are utterly Invalid, and that that is just how life is today, then, ... well, I shall be flabbergasted, but have to accept it.

But surely not...

10:55 pm on Jan 25, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 20, 2004
posts:1477
votes: 0


98,203 hits on the website from 5,258 humans also produced 5,339 bad packets with 603 different IP-addresses

Why not? Thinking aloud, if you *are* being attacked by someone interested in executing a packet injection hack, you might end up with all of the 'real' requests being satisfied with your 'real' visitors none the wiser. For each 'real' visitor and their accompanying connection, the attacker (who may be port monitoring your system) makes at least one attempt to hijack the packet stream ... and fails ... resulting in at least one invalid packet per 'real' visitor.

The nature of a packet injection attack is such that an existing packet stream and a valid connection must be established, then comes the port monitoring to view that packet stream's configuration, then comes the guessing at the next packet number in the sequence (while valid packets are flowing freely), then comes the attempt to inject the forged packet and finally the file buffer is overflowed or whatever 'crowbar' is being applied to compromise the system after a successful injection. An incorrect guess at the packet's sequence number will cause the forged packet to simply bounce off (invalid packet because it doesn't fit into the request sequence), as the original request/exchange is handled normally.

I must confess that my own servers show none of this activity (0 count in all messages logs), and we get bombarded with various kinds of attacks every day. Maybe nobody is trying a packet injection on us, or maybe PortSentry is keeping anyone from monitoring our ports successfully, or maybe I'm off base.

Hopefully, conjecture may lead to truth ... ;)

11:22 pm on Jan 25, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 20, 2004
posts:1477
votes: 0


Oops! Of course there are no entries in my messages logs, as I don't have the logging function set up like you do, so who knows how many invalid packets my servers may be receiving? <doh>

As an aside, the key to packet injection is guessing the next

ID=
assignment correctly, or at least guessing within a range that has not yet been qualified and is still within range of the last packet of the sequence.

In your example, note the two sequential IDs. Are those the only two entries for that IP?

Lastly, perhaps, the 'Invalid' connection state your rule is keying on refers to packets that are 'not part of any connection', according to my iptables, so ... hmmm ... still feeling pretty good about attempted packet injection attacks from zombie servers somewhere in South Asia.

2:36 pm on Jan 26, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 7, 2004
posts:660
votes: 0


Whilst I am sure that some web visitors are attempting packet injection attacks, I find it difficult to believe that (more than) 1 in 10 are doing that. My web-host was not in the least surprised at the 11% figure, but hypothesised a combo of compromised machines + installed (otherwise-legal) malware causing corruption. That sounds more likely to me (of the 8 sites that probed the server on Jan 24, three made an Invalid-packet webpage request [none are listed below]).

I took 3 hours out to look closely at 31 IPs from the total of 504 from Tue Jan 24 (one day in the life of a web-server). Only 2 IPs did not appear in the Apache logs. Of the other 29 IPs, 22 supplied Invalid packets well after the last hit, and only 7 were blocked during mid-page load (they all got the html page; it was .png files, etc that suffered).

What is very interesting in this is that half of the IPs made only 1 successful page-request, then a second request which totally died (because it was blocked due to Invalid packets). The single-page bounce-rate stands at 53% on my site. That has worried me immensely, and it is some (bitter) comfort to realise that it is so high due to mis-configured machines.

KeepAlive is active on my Apache webserver. Does any one know if there are any problems with this?

2:45 pm on Jan 26, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 7, 2004
posts:660
votes: 0


For propeller-heads like me, below is the excruciating detail which is summarised in the posting above:

Tue Jan 24 - one day in the life of a web-server:

sample of 31 of 504 TCP:80 IPs blocked due to Invalid Packets:
(603 total when UDP packets included)
2 have no entry in the access_log
15 make 1 page request
8 make 2 page requests
1 make 3 page requests
3 make 4 page requests
1 make 5 page requests
1 make 8 page requests
.
(ignoring the 2 no-access-log, and considering the 29 others):
No of Invalid packets AFTER successful page request: 22
No of Invalid packets DURING successful page request: 7
.
MSIE 6.0; Windows NT 5.1 20
MSIE 6.0; Windows NT 5.0 5
MSIE 5.0; Windows 98 1
MSIE 6.0; Windows 98 2
Firefox/1.5 Win98 1
.
(AWStats)
Windows XP 73.8%
Windows 2k 10.7%
Windows 98 8.3%
MSIE 82.7%
Firefox 12.9%

And some (more raw) data from which the above summary is drawn:

IP-Address - No of Invalid packets - Page-access Time - Browser + OS - Invalid Packet Time (first-last) 
213.46.14.221 - 2 packets to tcp(80) 12:06:03 MSIE 6.0; Windows NT 5.1 - 12:09:57
12:06:17
213.119.29.151 - 12 packets to tcp(80) 23:50:35 MSIE 5.0; Windows 98 - 23:53:19-23:54:34
213.120.71.162 - 1 packet to tcp(80) 12:48:55 MSIE 6.0; Windows NT 5.0 - 12:56:17
213.137.125.151 - 10 packets to tcp(80) 21:20:03 MSIE 6.0; Windows NT 5.1 - 21:22:15-21:27:06
21:22:15
213.192.203.21 - 2 packets to tcp(80) - Jan 22 22:25:01
Jan 22 23:30:29
Jan 24 12:56:43
Jan 24 13:26:07
213.217.35.41 - 9 packets to tcp(80,80) 18:03:53 Firefox/1.5 Win98 - 18:07:53-18:17:53
18:05:38
213.217.46.50 - 68 packets to tcp(80,80) 11:38:43 MSIE 6.0; Windows NT 5.0 - 11:40:09-11:53:42
11:40:09
213.217.51.7 - 23 packets to tcp(80,80) 17:17:19 MSIE 6.0; Windows NT 5.1 - 17:19:36-17:31:51
213.217.54.66 - 63 packets to tcp(80,80) 13:33:32 MSIE 6.0; Windows NT 5.1 - 13:35:47-13:49:28
213.244.215.54 - 12 packets to tcp(80) 19:27:40 MSIE 6.0; Windows NT 5.1 - 19:30:41-19:38:52
19:30:42
19:30:53
19:31:28
213.255.248.188 - 2 packets to tcp(80) 23:07:40 MSIE 6.0; Windows NT 5.1 - 23:19:03
216.0.1.90 - 2 packets to tcp(80) 20:00:33 MSIE 6.0; Windows NT 5.0 - 20:07:31
216.155.87.72 - 2 packets to tcp(80) 20:47:48 MSIE 6.0; Windows NT 5.1 - 20:50:00
217.41.0.248 - 42 packets to tcp(80) 17:21:21 MSIE 6.0; Windows 98 - 17:21:41-17:34:38
17:23:41
17:31:01
17:31:32
17:31:50
217.41.245.90 - 1 packet to tcp(80) 12:59:41 MSIE 6.0; Windows NT 5.0 - 13:09:53
13:00:21
13:01:38
13:01:44
217.46.153.185 - 2 packets to tcp(80) 09:42:01 MSIE 6.0; Windows NT 5.1 - 09:47:24
217.58.40.220 - 2 packets to tcp(80) 16:12:36 MSIE 6.0; Windows NT 5.0 - 16:20:36
217.68.80.37 - 9 packets to tcp(80) 07:41:08 MSIE 6.0; Windows NT 5.1 - 07:50:48-08:01:06
07:42:11
07:42:42
07:43:10
217.115.254.100 - 6 packets to tcp(80) 13:57:12 MSIE 6.0; Windows NT 5.1 - 14:03:33-14:11:17
13:57:55
13:58:22
13:58:38
14:06:26
14:05:14
14:06:03
14:06:19
217.136.16.88 - 1 packet to tcp(80) 10:45:35 MSIE 6.0; Windows NT 5.1 - 10:47:52
217.140.43.124 - 11 packets to tcp(80) 13:22:32 MSIE 6.0; Windows NT 5.1 - 13:29:23-13:29:59
217.166.54.67 - 4 packets to tcp(80) 07:58:23 MSIE 6.0; Windows NT 5.1 - 08:01:23-08:06:37
08:01:23
217.201.64.195 - 19 packets to tcp(80) 07:36:04-07:37:00
217.220.71.153 - 1 packet to tcp(80) 12:00:04 MSIE 6.0; Windows NT 5.1 - 12:04:25
12:00:41
218.91.111.17 - 12 packets to tcp(80) 05:18:36 MSIE 6.0; Windows 98 - 05:21:08-05:22:31
05:21:50
219.89.206.174 - 1 packet to tcp(80) 02:34:12 MSIE 6.0; Windows NT 5.1 - 02:43:35
220.130.158.245 - 10 packets to tcp(80) 09:30:49 MSIE 6.0; Windows NT 5.1 - 09:40:44-09:41:23
09:31:18
220.226.53.249 - 13 packets to tcp(80) 19:30:04 MSIE 6.0; Windows NT 5.1 - 19:33:30-19:34:10
19:33:31
19:26:55
220.233.5.136 - 2 packets to tcp(80) 03:16:39 MSIE 6.0; Windows NT 5.1 - 03:26:10
222.126.35.114 - 2 packets to tcp(80) 08:21:20 MSIE 6.0; Windows NT 5.1 - 08:28:27
222.166.160.248 - 1 packet to tcp(80) 18:01:26 MSIE 6.0; Windows NT 5.1 - 18:02:46
6:09 pm on Jan 26, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 20, 2004
posts:1477
votes: 0


I find it difficult to believe that (more than) 1 in 10 are doing that

Aahh ... but more than likely NONE of your 'real' visitors are doing that. Let me explain ...

Your visitors are simply ... visiting. They are not the problem. They provide the conduits.

If my theory holds, the hacker(s) is lurking ... hanging out around your ports, sniffing the 'real' visitors' packets for info. With each 'real' visitor, the un-related hacker tries to inject a forged packet by piggy-backing in on the 'real' visitors connection. They are not requesting anything, they are trying to wedge themselves into the valid packet streams being provided by your 'real' visitors.

If you logged ALL packets (yikes!), you'd see a valid stream something like:

...OK-OK-OK-BAD-OK-OK-OK-OK-OK-OK-OK...

where the OK packets are the 'real' stream and the BAD packet is the injection attempt. It should (in an ideal hack) look nearly exactly like the 'real' packets. The failed attempt has no impact on the 'real' visitor's experience, as all of the valid request's packets make it through just fine.

A forged packet is pretty simple to make out of an 8-bit file, and its transmission profile can contain any kind of information the hacker wants to send. Better hackers make their packets look like 'real' traffic ... with a user agent and everything ... preferably identical (except for the guessed ID= attribute) to 'real' packets being observed in any particular stream. When a packet is suspect ... all of the data it provides is suspect, too. If a packet looks unusual, it draws attention to itself. Much better for it to look like a companion packet in a stream of valid packets that had a little trouble getting validated.

Your list contains invalid packets from Chile, The Netherlands, Indonesia, London, Tehran (more than once), and several other notoriously-porous blocks. Sure, there are a few from the U.S. or from other 'trusted' blocks, but ... hey ... there are loose installations all over the world.

Why would a 'real' packet stream contain invalid packets?

1) Mis-configuration of the protocol stack(s) ... if not on the client end then on the server end.

2) Timeout issues that may close a connection erroneously before all packets have been received.

3) Random packets that really aren't part of any valid connection ... maybe they've been drifting around in cyberspace for awhile and finally make it to the destination after the connection has been closed.

4) Maybe the invalid packets are forged.

Of these (and I'm wide open to others), #3 is kind of possible, but not likely at the rates you noted, and #4 is the most likely, given the patterns you describe.

It still smells like an attack or two (or many).

I'm adding your logging function to my logs, to see what's going on with our otherwise cool servers, and to compare with your experience so I may understand it better.

8:27 pm on Jan 26, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 7, 2004
posts:660
votes: 0


StupidScript:
The failed attempt has no impact on the 'real' visitor's experience

Not the case for (most of) the IPs as above.

The most common symptom (in the Apache logs) was just the html page, style sheet + one graphic file loaded (there are 7-10 graphics files required). Other symptoms were multiple requests for the same file (the style sheet comes to mind, but it varied), or a duplicate 200 load of graphic files for a second page, then 304s on the 3rd page. Then there are all the folks with Invalid packets 4-10 minutes after the first page, and nothing whatsoever in the apache logs. There isn't the slightest doubt that these folks were having problems getting a full-page load (html + all helper files). The big question, of course, is why?

The big question for me is whether I can trust the Firewall+kernel diagnosing an invalid packet? Could it be a bug in either, or in Apache?

I am fully willing to keep the block if I cannot find a bug elsewhere. The one change that perhaps need making is not to BLOCK but rather to REJECT the packet. The latter sends a message which will allow the client to recover promptly. The former (current behaviour) does not inform, and the client will then have to rely on the TCP timeout, which I understand to be 30 secs. The current behaviour will lead to a very poor perception of my website amongst more than 1 in 10 visitors.

Another interesting feature is how few Firewall scripts out there include this specific coding (block on Invalid), and that also begs the question why?

8:43 pm on Jan 26, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 20, 2004
posts:1477
votes: 0


Are your visitors actually receiving only partial pages/page elements? Have you had trouble yourself?

Is this server running through a proxy?

Is this an all-day-every-day situation? Your inital post seemed to include numbers that started low, peaked and then dropped off. Maybe that's the pattern of your traffic? Are the valid:invalid ratios consistent over time?

The data you provided (if we can trust it) indicates just a few dropped packets per request. 2 bad packets shouldn't keep a visitor from enjoying all of your page elements.

I dunno ... but I still can't get off of the attack scenario. Since I modified my iptables, this morning, I haven't seen any bad packet activity at all. There have been many attack attempts amidst many thousands of requests, and my PortSentry [sourceforge.net] has been deflecting attacks all day, but no invalid packets. Of course, if PortSentry is being completely successful (as usual) then any attempt to lurk and port monitor is being thwarted, anyway, so there is no opportunity to guess at the packet sequence and/or launch an injection attack. Without the lurking there can be no guessing.

Maybe the attacker (if they exist) is trying to create their own valid connection and ride their own wave into your server? The concept is the same, even if the disguise and execution is pretty weak.

I'm not trying to be dense or too paranoid, but all of your evidence still seems to support the attack scenario, IMHO. It's a tough one, all right.

Although it is a far out idea ... maybe there's an issue with the NIC or its interface at your server or with some other piece of hardware having intermittent problems? Not pretty, but I'm at a bit of a loss.

This 24 message thread spans 3 pages: 24
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members