homepage Welcome to WebmasterWorld Guest from 50.19.169.37
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Hardware and OS Related Technologies / Linux, Unix, and *nix like Operating Systems
Forum Library, Charter, Moderators: bakedjake

Linux, Unix, and *nix like Operating Systems Forum

    
IPTables: Upto 30,000 Invalid packets logged per week
Host-installed Firewall; the number of invalid packets seems outrageous
AlexK




msg:910936
 3:26 am on Jan 25, 2006 (gmt 0)

# uname -a
Linux mysite.com 2.6.9-22.0.2.ELsmp #1 SMP Tue Jan 17 07:10:04 CST 2006 i686 i686 i386 GNU/Linux

Firewall built using /etc/init.d/iptables-live by my (very well experienced, very helpful) host. The very first lines in the bad_packets chain is:

LOG   all -- 0.0.0.0/0   0.0.0.0/0   state INVALID LOG flags 0 level 4 prefix `Invalid packet: ' 
DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID

and this is producing tens of thousands of dropped packets per week:

fgrep -c 'Invalid packet' /var/log/messages*
/var/log/messages:13579
/var/log/messages.1:28311
/var/log/messages.2:29110
/var/log/messages.3:27177
/var/log/messages.4:20807

Here is just one recent IP that had it's packets dropped:
fgrep '222.124.226.200' /var/log/messages
Jan 23 17:14:26 olivia kernel: Invalid packet: IN=eth0 OUT= MAC=00:20:ed:82:c6:f5:00:30:f2:10:b0:00:08:00 SRC=222.124.226.200 DST=80.168.53.204 LEN=40 TOS=0x00 PREC=0x00 TTL=231 ID=13832 DF PROTO=TCP SPT=1109 DPT=80 WINDOW=0 RES=0x00 ACK RST URGP=0
Jan 23 17:14:28 olivia kernel: Invalid packet: IN=eth0 OUT= MAC=00:20:ed:82:c6:f5:00:30:f2:10:b0:00:08:00 SRC=222.124.226.200 DST=80.168.53.204 LEN=40 TOS=0x00 PREC=0x00 TTL=231 ID=13833 DF PROTO=TCP SPT=1110 DPT=80 WINDOW=0 RES=0x00 ACK RST URGP=0

Anyone else have any experience of this? Any similar stories? Will I cause any problems for my users if I remove the DROP code for Invalid packets? All help greatfully received.

 

StupidScript




msg:910937
 6:23 pm on Jan 25, 2006 (gmt 0)

There are a few reasons why a packet might be invalid. Likely either the packet is badly formed by some element in the transmission process, or (most likely) it is not synching up with an existing request.

It's pretty common for an attack to take the form of a 'packet injection' whereby the attacker tries to gain control of a system by guessing the packet stream's numbering sequence, and then to take control of the packet stream by injecting a forged packet with an address within the guessed range. A successful injection could cause a valid request to become corrupted and possibly result in the compromise of the machine. (Obviously more details are available in lots of different places on the 'net ...)

Sounds like it would be a good idea for you to keep dropping those packets. Your real users probably won't be sending invalid packets at a rate that would cause their valid requests to fail.

BTW, your sample IP came from Indonesia ... according to APNIC [apnic.net], which is fairly popular territory from which this type of abuse flows ...

AlexK




msg:910938
 9:55 pm on Jan 25, 2006 (gmt 0)

Thank you, StupidScript, that input is certainly helpful.

It is the sheer scale of the Invalid packets that is defeating my mind, and causes me to wonder whether I have hit some sort of bug in the Firewall or the kernel. Consider the following:

# egrep -c "Jan 24(.*)Invalid packet(.*)PROTO=TCP(.*)DPT=80" /var/log/messages
5339
# fgrep -c "24/Jan" /var/log/httpd/co*-access_log
/var/log/httpd/com-access_log:66110
/var/log/httpd/couk-access_log:32093

So, 98,203 hits on the website from 5,258 humans also produced 5,339 bad packets with 603 different IP-addresses. That's 5.4% of hits and 11.4% of people! That is surely a lot!

Now, if you are saying that more than 1 in 10 of all visitors are either trying to hack the site, or have such badly mis-configured TCP stacks that their page requests are utterly Invalid, and that that is just how life is today, then, ... well, I shall be flabbergasted, but have to accept it.

But surely not...

StupidScript




msg:910939
 10:55 pm on Jan 25, 2006 (gmt 0)

98,203 hits on the website from 5,258 humans also produced 5,339 bad packets with 603 different IP-addresses

Why not? Thinking aloud, if you *are* being attacked by someone interested in executing a packet injection hack, you might end up with all of the 'real' requests being satisfied with your 'real' visitors none the wiser. For each 'real' visitor and their accompanying connection, the attacker (who may be port monitoring your system) makes at least one attempt to hijack the packet stream ... and fails ... resulting in at least one invalid packet per 'real' visitor.

The nature of a packet injection attack is such that an existing packet stream and a valid connection must be established, then comes the port monitoring to view that packet stream's configuration, then comes the guessing at the next packet number in the sequence (while valid packets are flowing freely), then comes the attempt to inject the forged packet and finally the file buffer is overflowed or whatever 'crowbar' is being applied to compromise the system after a successful injection. An incorrect guess at the packet's sequence number will cause the forged packet to simply bounce off (invalid packet because it doesn't fit into the request sequence), as the original request/exchange is handled normally.

I must confess that my own servers show none of this activity (0 count in all messages logs), and we get bombarded with various kinds of attacks every day. Maybe nobody is trying a packet injection on us, or maybe PortSentry is keeping anyone from monitoring our ports successfully, or maybe I'm off base.

Hopefully, conjecture may lead to truth ... ;)

StupidScript




msg:910940
 11:22 pm on Jan 25, 2006 (gmt 0)

Oops! Of course there are no entries in my messages logs, as I don't have the logging function set up like you do, so who knows how many invalid packets my servers may be receiving? <doh>

As an aside, the key to packet injection is guessing the next ID= assignment correctly, or at least guessing within a range that has not yet been qualified and is still within range of the last packet of the sequence.

In your example, note the two sequential IDs. Are those the only two entries for that IP?

Lastly, perhaps, the 'Invalid' connection state your rule is keying on refers to packets that are 'not part of any connection', according to my iptables, so ... hmmm ... still feeling pretty good about attempted packet injection attacks from zombie servers somewhere in South Asia.

AlexK




msg:910941
 2:36 pm on Jan 26, 2006 (gmt 0)

Whilst I am sure that some web visitors are attempting packet injection attacks, I find it difficult to believe that (more than) 1 in 10 are doing that. My web-host was not in the least surprised at the 11% figure, but hypothesised a combo of compromised machines + installed (otherwise-legal) malware causing corruption. That sounds more likely to me (of the 8 sites that probed the server on Jan 24, three made an Invalid-packet webpage request [none are listed below]).

I took 3 hours out to look closely at 31 IPs from the total of 504 from Tue Jan 24 (one day in the life of a web-server). Only 2 IPs did not appear in the Apache logs. Of the other 29 IPs, 22 supplied Invalid packets well after the last hit, and only 7 were blocked during mid-page load (they all got the html page; it was .png files, etc that suffered).

What is very interesting in this is that half of the IPs made only 1 successful page-request, then a second request which totally died (because it was blocked due to Invalid packets). The single-page bounce-rate stands at 53% on my site. That has worried me immensely, and it is some (bitter) comfort to realise that it is so high due to mis-configured machines.

KeepAlive is active on my Apache webserver. Does any one know if there are any problems with this?

AlexK




msg:910942
 2:45 pm on Jan 26, 2006 (gmt 0)

For propeller-heads like me, below is the excruciating detail which is summarised in the posting above:

Tue Jan 24 - one day in the life of a web-server:
sample of 31 of 504 TCP:80 IPs blocked due to Invalid Packets:
(603 total when UDP packets included)
2 have no entry in the access_log
15 make 1 page request
8 make 2 page requests
1 make 3 page requests
3 make 4 page requests
1 make 5 page requests
1 make 8 page requests
.
(ignoring the 2 no-access-log, and considering the 29 others):
No of Invalid packets AFTER successful page request: 22
No of Invalid packets DURING successful page request: 7
.
MSIE 6.0; Windows NT 5.1 20
MSIE 6.0; Windows NT 5.0 5
MSIE 5.0; Windows 98 1
MSIE 6.0; Windows 98 2
Firefox/1.5 Win98 1
.
(AWStats)
Windows XP 73.8%
Windows 2k 10.7%
Windows 98 8.3%
MSIE 82.7%
Firefox 12.9%

And some (more raw) data from which the above summary is drawn:

IP-Address - No of Invalid packets - Page-access Time - Browser + OS - Invalid Packet Time (first-last) 
213.46.14.221 - 2 packets to tcp(80) 12:06:03 MSIE 6.0; Windows NT 5.1 - 12:09:57
12:06:17
213.119.29.151 - 12 packets to tcp(80) 23:50:35 MSIE 5.0; Windows 98 - 23:53:19-23:54:34
213.120.71.162 - 1 packet to tcp(80) 12:48:55 MSIE 6.0; Windows NT 5.0 - 12:56:17
213.137.125.151 - 10 packets to tcp(80) 21:20:03 MSIE 6.0; Windows NT 5.1 - 21:22:15-21:27:06
21:22:15
213.192.203.21 - 2 packets to tcp(80) - Jan 22 22:25:01
Jan 22 23:30:29
Jan 24 12:56:43
Jan 24 13:26:07
213.217.35.41 - 9 packets to tcp(80,80) 18:03:53 Firefox/1.5 Win98 - 18:07:53-18:17:53
18:05:38
213.217.46.50 - 68 packets to tcp(80,80) 11:38:43 MSIE 6.0; Windows NT 5.0 - 11:40:09-11:53:42
11:40:09
213.217.51.7 - 23 packets to tcp(80,80) 17:17:19 MSIE 6.0; Windows NT 5.1 - 17:19:36-17:31:51
213.217.54.66 - 63 packets to tcp(80,80) 13:33:32 MSIE 6.0; Windows NT 5.1 - 13:35:47-13:49:28
213.244.215.54 - 12 packets to tcp(80) 19:27:40 MSIE 6.0; Windows NT 5.1 - 19:30:41-19:38:52
19:30:42
19:30:53
19:31:28
213.255.248.188 - 2 packets to tcp(80) 23:07:40 MSIE 6.0; Windows NT 5.1 - 23:19:03
216.0.1.90 - 2 packets to tcp(80) 20:00:33 MSIE 6.0; Windows NT 5.0 - 20:07:31
216.155.87.72 - 2 packets to tcp(80) 20:47:48 MSIE 6.0; Windows NT 5.1 - 20:50:00
217.41.0.248 - 42 packets to tcp(80) 17:21:21 MSIE 6.0; Windows 98 - 17:21:41-17:34:38
17:23:41
17:31:01
17:31:32
17:31:50
217.41.245.90 - 1 packet to tcp(80) 12:59:41 MSIE 6.0; Windows NT 5.0 - 13:09:53
13:00:21
13:01:38
13:01:44
217.46.153.185 - 2 packets to tcp(80) 09:42:01 MSIE 6.0; Windows NT 5.1 - 09:47:24
217.58.40.220 - 2 packets to tcp(80) 16:12:36 MSIE 6.0; Windows NT 5.0 - 16:20:36
217.68.80.37 - 9 packets to tcp(80) 07:41:08 MSIE 6.0; Windows NT 5.1 - 07:50:48-08:01:06
07:42:11
07:42:42
07:43:10
217.115.254.100 - 6 packets to tcp(80) 13:57:12 MSIE 6.0; Windows NT 5.1 - 14:03:33-14:11:17
13:57:55
13:58:22
13:58:38
14:06:26
14:05:14
14:06:03
14:06:19
217.136.16.88 - 1 packet to tcp(80) 10:45:35 MSIE 6.0; Windows NT 5.1 - 10:47:52
217.140.43.124 - 11 packets to tcp(80) 13:22:32 MSIE 6.0; Windows NT 5.1 - 13:29:23-13:29:59
217.166.54.67 - 4 packets to tcp(80) 07:58:23 MSIE 6.0; Windows NT 5.1 - 08:01:23-08:06:37
08:01:23
217.201.64.195 - 19 packets to tcp(80) 07:36:04-07:37:00
217.220.71.153 - 1 packet to tcp(80) 12:00:04 MSIE 6.0; Windows NT 5.1 - 12:04:25
12:00:41
218.91.111.17 - 12 packets to tcp(80) 05:18:36 MSIE 6.0; Windows 98 - 05:21:08-05:22:31
05:21:50
219.89.206.174 - 1 packet to tcp(80) 02:34:12 MSIE 6.0; Windows NT 5.1 - 02:43:35
220.130.158.245 - 10 packets to tcp(80) 09:30:49 MSIE 6.0; Windows NT 5.1 - 09:40:44-09:41:23
09:31:18
220.226.53.249 - 13 packets to tcp(80) 19:30:04 MSIE 6.0; Windows NT 5.1 - 19:33:30-19:34:10
19:33:31
19:26:55
220.233.5.136 - 2 packets to tcp(80) 03:16:39 MSIE 6.0; Windows NT 5.1 - 03:26:10
222.126.35.114 - 2 packets to tcp(80) 08:21:20 MSIE 6.0; Windows NT 5.1 - 08:28:27
222.166.160.248 - 1 packet to tcp(80) 18:01:26 MSIE 6.0; Windows NT 5.1 - 18:02:46

StupidScript




msg:910943
 6:09 pm on Jan 26, 2006 (gmt 0)

I find it difficult to believe that (more than) 1 in 10 are doing that

Aahh ... but more than likely NONE of your 'real' visitors are doing that. Let me explain ...

Your visitors are simply ... visiting. They are not the problem. They provide the conduits.

If my theory holds, the hacker(s) is lurking ... hanging out around your ports, sniffing the 'real' visitors' packets for info. With each 'real' visitor, the un-related hacker tries to inject a forged packet by piggy-backing in on the 'real' visitors connection. They are not requesting anything, they are trying to wedge themselves into the valid packet streams being provided by your 'real' visitors.

If you logged ALL packets (yikes!), you'd see a valid stream something like:

...OK-OK-OK-BAD-OK-OK-OK-OK-OK-OK-OK...

where the OK packets are the 'real' stream and the BAD packet is the injection attempt. It should (in an ideal hack) look nearly exactly like the 'real' packets. The failed attempt has no impact on the 'real' visitor's experience, as all of the valid request's packets make it through just fine.

A forged packet is pretty simple to make out of an 8-bit file, and its transmission profile can contain any kind of information the hacker wants to send. Better hackers make their packets look like 'real' traffic ... with a user agent and everything ... preferably identical (except for the guessed ID= attribute) to 'real' packets being observed in any particular stream. When a packet is suspect ... all of the data it provides is suspect, too. If a packet looks unusual, it draws attention to itself. Much better for it to look like a companion packet in a stream of valid packets that had a little trouble getting validated.

Your list contains invalid packets from Chile, The Netherlands, Indonesia, London, Tehran (more than once), and several other notoriously-porous blocks. Sure, there are a few from the U.S. or from other 'trusted' blocks, but ... hey ... there are loose installations all over the world.

Why would a 'real' packet stream contain invalid packets?

1) Mis-configuration of the protocol stack(s) ... if not on the client end then on the server end.

2) Timeout issues that may close a connection erroneously before all packets have been received.

3) Random packets that really aren't part of any valid connection ... maybe they've been drifting around in cyberspace for awhile and finally make it to the destination after the connection has been closed.

4) Maybe the invalid packets are forged.

Of these (and I'm wide open to others), #3 is kind of possible, but not likely at the rates you noted, and #4 is the most likely, given the patterns you describe.

It still smells like an attack or two (or many).

I'm adding your logging function to my logs, to see what's going on with our otherwise cool servers, and to compare with your experience so I may understand it better.

AlexK




msg:910944
 8:27 pm on Jan 26, 2006 (gmt 0)

StupidScript:
The failed attempt has no impact on the 'real' visitor's experience

Not the case for (most of) the IPs as above.

The most common symptom (in the Apache logs) was just the html page, style sheet + one graphic file loaded (there are 7-10 graphics files required). Other symptoms were multiple requests for the same file (the style sheet comes to mind, but it varied), or a duplicate 200 load of graphic files for a second page, then 304s on the 3rd page. Then there are all the folks with Invalid packets 4-10 minutes after the first page, and nothing whatsoever in the apache logs. There isn't the slightest doubt that these folks were having problems getting a full-page load (html + all helper files). The big question, of course, is why?

The big question for me is whether I can trust the Firewall+kernel diagnosing an invalid packet? Could it be a bug in either, or in Apache?

I am fully willing to keep the block if I cannot find a bug elsewhere. The one change that perhaps need making is not to BLOCK but rather to REJECT the packet. The latter sends a message which will allow the client to recover promptly. The former (current behaviour) does not inform, and the client will then have to rely on the TCP timeout, which I understand to be 30 secs. The current behaviour will lead to a very poor perception of my website amongst more than 1 in 10 visitors.

Another interesting feature is how few Firewall scripts out there include this specific coding (block on Invalid), and that also begs the question why?

StupidScript




msg:910945
 8:43 pm on Jan 26, 2006 (gmt 0)

Are your visitors actually receiving only partial pages/page elements? Have you had trouble yourself?

Is this server running through a proxy?

Is this an all-day-every-day situation? Your inital post seemed to include numbers that started low, peaked and then dropped off. Maybe that's the pattern of your traffic? Are the valid:invalid ratios consistent over time?

The data you provided (if we can trust it) indicates just a few dropped packets per request. 2 bad packets shouldn't keep a visitor from enjoying all of your page elements.

I dunno ... but I still can't get off of the attack scenario. Since I modified my iptables, this morning, I haven't seen any bad packet activity at all. There have been many attack attempts amidst many thousands of requests, and my PortSentry [sourceforge.net] has been deflecting attacks all day, but no invalid packets. Of course, if PortSentry is being completely successful (as usual) then any attempt to lurk and port monitor is being thwarted, anyway, so there is no opportunity to guess at the packet sequence and/or launch an injection attack. Without the lurking there can be no guessing.

Maybe the attacker (if they exist) is trying to create their own valid connection and ride their own wave into your server? The concept is the same, even if the disguise and execution is pretty weak.

I'm not trying to be dense or too paranoid, but all of your evidence still seems to support the attack scenario, IMHO. It's a tough one, all right.

Although it is a far out idea ... maybe there's an issue with the NIC or its interface at your server or with some other piece of hardware having intermittent problems? Not pretty, but I'm at a bit of a loss.

AlexK




msg:910946
 9:57 pm on Jan 26, 2006 (gmt 0)

StupidScript:
Are your visitors actually receiving only partial pages/page elements?
A very large number of the ones that I investigated above.

Have you had trouble yourself?
Never noticed any.

Is this server running through a proxy?
No.

Is this an all-day-every-day situation?
Yes.

Your inital post seemed to include numbers that started low, peaked and then dropped off. Maybe that's the pattern of your traffic?
Isn't that the pattern of *everybody's* traffic?

Are the valid:invalid ratios consistent over time?
That would involve analysis that I am not, at this moment, prepared to do.

2 bad packets shouldn't keep a visitor from enjoying all of your page elements.
Just 1 dropped packet will shaft at least one page element.

StupidScript




msg:910947
 10:50 pm on Jan 26, 2006 (gmt 0)

Hmmm ...

Did this just start one day, or has it been a featured issue the entire time you've had this server?

I mean, if so many visitors are experiencing it (although you haven't, yourself), then it seems likely there's a fundamental issue with your box, either some config thing or some hardware thing.

I've known NICs to munge packet data when they're not working properly, and encryption errors or timing issues could come into play. But I've never had invalid packet issues (or visitors receiving incomplete pages) with a server unless it was being attacked.

Back to the research board, I guess ... sorry to not come up with a simple answer for ya, AlexK. I really appreciate the work you've done with blocking bad bots.

AlexK




msg:910948
 12:19 am on Jan 27, 2006 (gmt 0)

StupidScript:
has it been a featured issue the entire time you've had this server?

Was a (big, surprising) issue from the first day that the Firewall was installed, some months back. I was tired of the number of people trying to SSH into the box every day - often scores of IPs, sometimes thousands of dictionary attempts. It was tiring, so I got my host to Firewall them out.

It is worth pointing out, I think, that I've been concentrating on TCP:80 packets. There are also a very large number of UDP and also some TCP:some-port-other-than-80 packets also being rejected (on 24 Jan, 504 TCP:80 IPs and 99 not-TCP:80 IPs). They are definately attacks (unless you can suggest a good reason why UDP packets should be coming at my box).

The conversation has been very useful - the practical value for me has been to perceive that, assuming no bugs--hardware or software--come to light, the TCP:80 packets should be REJECTed rather than DROPped. That one step should improve the perception of the site enormously. I would rather get to the source of the problem, however.

StupidScript




msg:910949
 1:09 am on Jan 27, 2006 (gmt 0)

from the first day that the Firewall was installed

Then the firewall is the place to start checking. If it's hardware, suspect it might be intermittently bad. If it's software, then there may be a config problem.

But I'm sure you are already moving on that ... :)

AlexK




msg:910950
 2:23 pm on Jan 28, 2006 (gmt 0)

StupidScript:
If it's hardware, ...

Aaarghh! No, no, no (holds up 2 pencils in the shape of a cross).

13 months ago I discovered that a memory module had gone bad in my server [webmasterworld.com], something that my stupid hosts-at-the-time-but-not-for-much-longer never did actually fix (they replaced the bad module, but it was part of a matched-pair, so other, similar, problems began to appear).

...suspect it might be intermittently bad

Disagree with you here.

Yes, hardware can present intermittant problems whilst in the process of breaking down but, the very core of hardware is, once bad always bad. So, hardware presents consistently-bad issues. It is software that manages to give sometimes-good sometimes-bad symptoms.

AlexK




msg:910951
 2:27 pm on Jan 28, 2006 (gmt 0)

StupidScript:
the key to packet injection is guessing the next ID= assignment correctly

I have found good info on this at an excellent--and huge--iptables tutorial [iptables-tutorial.frozentux.net].

AlexK




msg:910952
 3:17 pm on Jan 28, 2006 (gmt 0)

More forensics:

*All* Invalid packets (in the current set of logs) are inbound.

My server is getting a number of Invalid ICMP packets. These are either Type 3 (61%) (Destination Unreachable) or Type 11 (36%) (Time Exceeded) (119 total in one log, with 3 unknown). Here is a small sample from log.1 (for reasons which become obvious in a later post):
Jan 17 12:25:21 olivia kernel: Invalid packet: IN=eth0 OUT= MAC=00:20:ed:82:c6:f5:00:30:f2:10:b0:00:08:00 SRC=202.1.53.3 DST=80.168.53.204 LEN=68 TOS=0x00 PREC=0xC0 TTL=239 ID=3681 PROTO=ICMP TYPE=3 CODE=1 [SRC=80.168.53.204 DST=202.1.53.100 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=3423 DF PROTO=TCP SPT=80 DPT=1331 WINDOW=31592 RES=0x00 ACK FIN URGP=0 ]
Jan 17 16:29:17 olivia kernel: Invalid packet: IN=eth0 OUT= MAC=00:20:ed:82:c6:f5:00:30:f2:10:b0:00:08:00 SRC=64.254.97.10 DST=80.168.53.204 LEN=56 TOS=0x00 PREC=0x00 TTL=101 ID=30404 PROTO=ICMP TYPE=3 CODE=2 [SRC=80.168.53.204 DST=64.254.97.10 LEN=244 TOS=0x00 PREC=0xC0 TTL=118 ID=4920 PROTO=46 ]
Jan 17 16:51:13 olivia kernel: Invalid packet: IN=eth0 OUT= MAC=00:20:ed:82:c6:f5:00:30:f2:10:b0:00:08:00 SRC=200.204.208.81DST=80.168.53.204 LEN=56 TOS=0x00 PREC=0x00 TTL=238 ID=10789 PROTO=ICMP TYPE=3 CODE=13 [SRC=80.168.53.204 DST=200.148.31.46 LEN=44 TOS=0x00 PREC=0x00 TTL=254 ID=1 PROTO=TCP INCOMPLETE [8 bytes] ]
Jan 17 22:35:36 olivia kernel: Invalid packet: IN=eth0 OUT= MAC=00:20:ed:82:c6:f5:00:30:f2:10:b0:00:08:00 SRC=12.41.252.3 DST=80.168.53.204 LEN=56 TOS=0x00 PREC=0x00 TTL=45 ID=7427 PROTO=ICMP TYPE=3 CODE=1 [SRC=80.168.53.204 DST=12.41.252.148 LEN=644 TOS=0x00 PREC=0x00 TTL=48 ID=19937 DF PROTO=TCP INCOMPLETE [8 bytes] ]
Jan 17 22:37:13 olivia kernel: Invalid packet: IN=eth0 OUT= MAC=00:20:ed:82:c6:f5:00:30:f2:10:b0:00:08:00 SRC=12.41.252.3 DST=80.168.53.204 LEN=56 TOS=0x00 PREC=0x00 TTL=45 ID=9987 PROTO=ICMP TYPE=3 CODE=1 [SRC=80.168.53.204 DST=12.41.252.148 LEN=644 TOS=0x00 PREC=0x00 TTL=48 ID=19939 DF PROTO=TCP INCOMPLETE [8 bytes] ]
Jan 18 03:51:48 olivia kernel: Invalid packet: IN=eth0 OUT= MAC=00:20:ed:82:c6:f5:00:30:f2:10:b0:00:08:00 SRC=210.211.64.204DST=80.168.53.204 LEN=56 TOS=0x00 PREC=0x00 TTL=110 ID=1115 PROTO=ICMP TYPE=11 CODE=1 [SRC=80.168.53.204 DST=210.211.64.204 LEN=1468 TOS=0x00 PREC=0x00 TTL=46 ID=62151 MF PROTO=TCP INCOMPLETE [8 bytes] ]

AlexK




msg:910953
 4:09 pm on Jan 28, 2006 (gmt 0)

Not a full resolution, but my response to the issue:

I switched the DROP off for 48+ hours, but kept the LOG, then compared results. There was just one instance found (not exhaustive) where an IP sent Invalid packets *and* was caught by another rule:
Jan 26 03:39:49 olivia kernel: Invalid packet: IN=eth0 OUT= MAC=00:20:ed:82:c6:f5:00:30:f2:10:b0:00:08:00 SRC=203.144.143.7 DST=80.168.53.204 LEN=52 TOS=0x00 PREC=0x00 TTL=41 ID=62431 PROTO=TCP SPT=64852 DPT=80 WINDOW=65535 RES=0x00 ACK FIN URGP=0
Jan 26 03:42:05 olivia kernel: New not syn: IN=eth0 OUT= MAC=00:20:ed:82:c6:f5:00:30:f2:10:b0:00:08:00 SRC=203.144.143.7 DST=80.168.53.204 LEN=40 TOS=0x00 PREC=0x00 TTL=41 ID=4270 PROTO=TCP SPT=25275 DPT=80 WINDOW=65535 RES=0x00 ACK URGP=0
Jan 26 03:42:07 olivia kernel: New not syn: IN=eth0 OUT= MAC=00:20:ed:82:c6:f5:00:30:f2:10:b0:00:08:00 SRC=203.144.143.7 DST=80.168.53.204 LEN=40 TOS=0x00 PREC=0x00 TTL=41 ID=44529 PROTO=TCP SPT=25557 DPT=80 WINDOW=65535 RES=0x00 ACK URGP=0
(both occurred many times with this IP; above is just a sample)

Going through the same exhaustive process as msg #:6+7 also produced little difference: there were perhaps more complete pages loaded, but still the same catalogue of partial page-loads (graphics files missing), multiple requests for the style sheet, etc. etc.. The only thing that I did get for my trouble was an extremely wierd page request [webmasterworld.com].

In the end, it seems that switching the DROP off offers little or no benefit to site visitors, and may clog up the server. So, it has been reinstituted, and the LOG has been dropped instead.

Here is a sight of the relevant part of the iptables script:
# bad_packets chain
#
.
# Drop INVALID packets immediately
# 2006-01-28 logging for INVALID switched off -AK
# log is filling with >500 IPs daily due to foll rule
# $IPT -A bad_packets -p ALL -m state --state INVALID -j LOG \
# --log-prefix "Invalid packet: "
.
$IPT -A bad_packets -p ALL -m state --state INVALID -j DROP
.
# Then check the tcp packets for additional problems
$IPT -A bad_packets -p tcp -j bad_tcp_packets
.
# All good, so return
$IPT -A bad_packets -p ALL -j RETURN
.
# bad_tcp_packets chain
#
# All tcp packets will traverse this chain.
# Every new connection attempt should begin with
# a syn packet.
.
.
# 2006-01-28 added -AK
# Act honourably on behalf of others receiving TCP
# Sequence Number Prediction attacks (these attacks
# rely on the spoofed host DROPping unknown SYN/ACKs
# rather than REJECTing them).
# See [iptables-tutorial.frozentux.net...]
$IPT -A bad_tcp_packets -p tcp --tcp-flags SYN,ACK SYN,ACK \
-m state --state NEW -j REJECT --reject-with tcp-reset
.
# Fix iptables feature of allowing NEW packets with SYN bit unset
# See [iptables-tutorial.frozentux.net...]
$IPT -A bad_tcp_packets -p tcp! --syn -m state --state NEW -j LOG \
--log-prefix "New not syn: "
$IPT -A bad_tcp_packets -p tcp! --syn -m state --state NEW -j DROP

I still want to know where all these wretched INVALID packets come from.

AlexK




msg:910954
 4:15 pm on Jan 28, 2006 (gmt 0)

Finally, and for the last time, here is a duplicate of the mind-numbing detail also offered in msg #:7, but this time for a period whilst the DROP command was not operative; it is a different range of IPs, and thus includes a number of ISPs sharing IPs amongst their customers (not the case with the earlier ones):

Thu Jan 26 - one day in the life of a web-server:
sample of 36 of 408 TCP:80 IPs logged due to Invalid Packets:
(547 total when none-TCP:80 packets included)
(40 visitors due to IP-sharing)
5 have no entry in the access_log
18 make 1 page request
4 make 2 page requests
7 make 3 page requests
2 make 4 page requests
1 make 6 page requests
1 make 7 page requests
1 make 10 page requests
1 make 12 page requests
.
(ignoring the 5 no-access-log, and considering the 35 others):
No of Invalid packets AFTER successful page request: 28
No of Invalid packets DURING successful page request: 7
.
MSIE 6.0; Windows NT 5.1 26
MSIE 6.0; Windows NT 5.0 3
MSIE 5.5; Windows NT 5.0 2
MSIE 6.0; Windows 98 1
Firefox/1.0.4 Windows NT 5.0 1
Firefox/1.5 Windows NT 5.1 3

IP-Address  No of Invalid packets Page-access Time - Browser + OS  Packet Time (first-last) 
4.153.84.204 1 packet to tcp(80) 03:07:10 MSIE 6.0; Windows 98 03:11:00 Invalid
03:08:14
03:13:12
12.19.113.105 2 packets to tcp(80) 17:59:43 MSIE 6.0; Windows NT 5.1 20:25:48 Invalid
12.46.111.83 2 packets to tcp(80) 20:20:40 MSIE 6.0; Windows NT 5.1 20:28:09 Invalid
12.47.172.138 2 packets to tcp(80) 23:44:16 MSIE 6.0; Windows NT 5.1 23:50:07 Invalid
12.129.230.13 2 packets to tcp(80) 03:04:02 MSIE 6.0; Windows NT 5.1 03:13:01 Invalid
12.145.168.100 2 packets to tcp(80) 15:04:50 MSIE 6.0; Windows NT 5.0 15:09:17 Invalid
12.152.251.106 2 packets to tcp(80) 17:53:15 MSIE 6.0; Windows NT 5.1 18:00:21 Invalid
12.161.66.6 1 packet to tcp(80) 17:45:39 Firefox/1.0.4 Windows NT 5.0 17:53:10 Invalid
17:45:54
17:46:12
12.170.50.206 2 packets to tcp(80) 20:20:42 MSIE 6.0; Windows NT 5.1 20:27:26 Invalid
15.203.169.126 54 packets to tcp(80,80)09:07:41 MSIE 6.0; Windows NT 5.1 09:09:58-09:18:43 (New not syn)
09:19:58 (Invalid)
16:33:44 MSIE 6.0; Windows NT 5.0 16:35:59-16:44:48 (New not syn)
16:33:44 16:45:59-16:46:03 (Invalid)
16:33:45
15.219.201.70 16 packets to tcp(80,80)03:44:53 MSIE 6.0; Windows NT 5.1 03:47:09-03:56:00 (New not syn)
03:57:09-03:57:15 (Invalid)
15.235.153.102 2 packets to tcp(80) 20:22:20 MSIE 6.0; Windows NT 5.1 20:29:27 Invalid
20.133.0.13 4 packets to tcp(80,80) 13:20:26 MSIE 5.5; Windows NT 5.0 13:22:53-13:25:23 (New not syn)
13:26:14 (Invalid)
20.133.0.14 9 packets to tcp(80,80) 12:07:06 MSIE 5.5; Windows NT 5.0 13:22:42-13:22:51 (New not syn)
13:20:24 13:23:41-13:23:45 (Invalid)
20.138.246.89 20 packets to tcp(80,80)13:23:36 MSIE 6.0; Windows NT 5.1 13:25:53-21:50:28 (Invalid)
13:25:53 13:28:09-21:49:13 (New not syn)
24.18.141.111 1 packet to tcp(80) 23:12:38 MSIE 6.0; Windows NT 5.1 23:22:48-02:28:21 Invalid
23:22:48
24.75.123.98 2 packets to tcp(80) 14:04:03 MSIE 6.0; Windows NT 5.1 14:13:07-14:13:09 Invalid
24.105.193.23 1 packet to tcp(80) 23:46:38 MSIE 6.0; Windows NT 5.1 00:03:27 Invalid
23:46:39
23:47:02
24.237.168.13 1 packet to tcp(80) 03:00:22 MSIE 6.0; Windows NT 5.1 03:03:02 Invalid
03:00:36
03:00:39
03:20:24 Firefox/1.5 Windows NT 5.1
38.119.107.81 36 packets to tcp(80) 15:42:50 MSIE 6.0; Windows NT 5.1 15:42:59-15:45:05 Invalid
15:44:05
15:44:25
15:44:31
15:44:44
15:44:56
58.65.199.228 5 packets to tcp(80) 07:11:16 Firefox/1.5 Windows NT 5.1 07:14:03-07:19:47 Invalid
07:14:03
58.147.0.42 22 packets to tcp(80,80)09:57:48 MSIE 6.0; Windows NT 5.1 10:01:13-10:04:58 (New not syn)
09:58:36 10:05:58 Invalid
09:58:51 12:01:00-12:04:46 (New not syn)
09:58:58 12:05:18 Invalid
11:56:57 MSIE 6.0; Windows NT 5.1 15:27:25-15:29:55 (New not syn)
11:57:54
11:58:43
15:25:04 MSIE 6.0; Windows NT 5.1 15:30:52-15:31:00 Invalid
58.186.23.199 1 packet to tcp(80) 17:06:40 Invalid
59.113.2.89 4 packets to tcp(80) 07:06:07 MSIE 6.0; Windows NT 5.1 07:08:54-07:31:23 Invalid
07:06:43
07:07:03
07:07:10
07:07:19
07:07:35
07:08:54
07:10:22
07:11:37
07:12:31
07:15:58
07:16:14
61.1.196.111 1 packet to tcp(80) 07:21:11 MSIE 6.0; Windows NT 5.1 07:23:17 Invalid
61.68.11.131 2 packets to tcp(80) 05:02:25 MSIE 6.0; Windows NT 5.1 05:04:15 Invalid
61.173.31.54 2 packets to tcp(80) 01:47:14 MSIE 6.0; Windows NT 5.1 01:51:20 Invalid
01:49:28
01:49:42
61.175.193.132 1 packet to tcp(80) 12:28:03 Invalid
61.218.223.67 1 packet to tcp(80) 01:45:01 Invalid
61.221.196.67 1 packet to tcp(80) 01:45:01 Invalid
61.241.79.3 1 packet to tcp(80) 12:28:03 Invalid
61.246.92.193 4 packets to tcp(80) 12:56:26 MSIE 6.0; Windows NT 5.0 13:05:42-13:09:47 Invalid
12:58:40
13:02:14
13:05:42
13:08:00
13:09:51
13:10:54
62.25.109.195 18 packets to tcp(80,80)08:32:59 MSIE 6.0; Windows NT 5.1 08:35:16-08:44:01 (New not syn)
08:45:16 Invalid
62.38.20.196 1 packet to tcp(80) 08:36:00 Firefox/1.5 Windows NT 5.1 09:19:48 Invalid
08:36:55
08:37:45
08:38:47
08:39:12
08:39:23
08:39:49
08:40:00
08:40:22
08:40:53
09:10:21 MSIE 6.0; Windows NT 5.1
09:10:56
62.38.254.39 2 packets to tcp(80) 16:52:44 MSIE 6.0; Windows NT 5.1 17:05:48 Invalid
16:53:13
16:54:33
16:54:43
62.90.217.106 2 packets to tcp(80) 13:11:59 MSIE 6.0; Windows NT 5.1 13:21:46 Invalid

AlexK




msg:910955
 5:04 pm on Jan 28, 2006 (gmt 0)

Ah, yes - a postscript.

I wanted a send a REJECT for INVALID tcp-packets so that the clients could clean up properly. That would send an ICMP packet Type-3 (Destination Unreachable) with a choice of:
icmp-net-unreachable (Code 0)
icmp-host-unreachable (Code 1)
icmp-port-unreachable (Code 2) (default)
icmp-proto-unreachable (Code 3)
icmp-net-prohibited (Code 9)
icmp-host-prohibited (Code 10)

None of the codes seemed correct. Possibly Code 13 (Communication Administratively Prohibited [RFC1812]) would have been right but, unless I am missing something, none of the codes said "hey, you have sent an INVALID packet" so, I threw up my hands and decided to simply DROP them (which does not send any notification).

SeanW




msg:910956
 8:16 pm on Feb 3, 2006 (gmt 0)

Jan 23 17:14:26 olivia kernel: Invalid packet: IN=eth0 OUT= MAC=00:20:ed:82:c6:f5:00:30:f2:10:b0:00:08:00 SRC=222.124.226.200 DST=80.168.53.204 LEN=40 TOS=0x00 PREC=0x00 TTL=231 ID=13832 DF PROTO=TCP SPT=1109 DPT=80 WINDOW=0 RES=0x00 ACK RST URGP=0

Since the packet has got RESET set, I'm pretty sure the other side wasn't hoping for more data :)

Sean

IPTables Friend




msg:3501408
 10:59 am on Nov 10, 2007 (gmt 0)

As this was one of only a couple of places where this problem was being discussed, and as there was no solution or proposed solution published, I wanted to post my improvements to his information and what I believe was my root cause (testing is still in progress).


The problem starts out in a straight forward way;

- User level observation is an intermittently failing protocol;

  • in mail this may be an SMTP server that doesn't seem to
    allow mail to be sent once or twice every four of five days
    (with a local "unable to send" / "host is down" type message
    reported to the user)

  • in web this may be an HTTP server that serves all content
    fine, except in some cases where an image will halt half
    way through, and leave the page "hanging" (like it's yet
    to receive more data).


But the problem is hard to diagnose;

- System level observation is an INVALID packet match;

  • i.e. where you have a rule in iptables that logs and drops
    packets that match the iptables state INVALID;

    iptables -A INPUT -m state --state INVALID -j LOG
    --log-level info --log-prefix "INVALID (input):"
    --log-ip-options --log-tcp-options

    iptables -A INPUT -m state --state INVALID -j DROP

  • so you can log and view INVALID packets/sessions;

    xx xx xx:xx:xx localhost kernel: INVALID (input):
    IN=eth1 OUT= MAC= SRC=xx.xx.xx.xx DST=xx.xx.xx.xx
    LEN=561 TOS=0x0C PREC=0x20 TTL=56 ID=2945 PROTO=TCP
    SPT=34920 DPT=80 WINDOW=183 RES=0x00 ACK PSH URGP=0


To the point that it took me a week of on-again off-again testing before I got to an indication of the cause;

- Packet level observation is a TCP packet that reportedly
contains an incorrect checksum;

  • i.e. where you run tcpdump on the problem interface of
    the affected host and monitor its verbose output;


    tcpdump -i eth1 -s1500 -vvv port 80 grep -i incorrect

    xx:xx:xx.577564 IP (tos 0x2c, ttl 56, id 2945, offset 0,
    flags [none], proto: TCP (6), length: 561)
    sourcehost.34920 > desthost.http: P,
    cksum 0x39fd (incorrect (-> 0xfc9e),
    1:522(521) ack 1 win 183

Like AlexK, I wasn't prepared to believe that this was typical hacker crap. Unlike AlexK, I found that I could intermittently reproduce the problem using one of my own Internet connections (so I could confirm his and my theory, and create a somewhat slow and painful testing platform -- but slightly faster than "wait-n-see").

So the real piece of missing information for AlexK (and myself originally) was that the INVALID state match doesn't just match on "out of state" packets - it also matches packets with other bad attributes, including an incorrect TCP checksum. This isn't commonly discussed, but is noted around the traps .. ie;
[osdir.com ]

In my case, I saw this problem on all of the hosts on my public network. This is odd as my iptables scripts have evolved across the past 10 years, with this particular install being a slightly improved version again - but with no substantial changes. And this was certainly something that I hadn't seen before.

While testing is still running, in my case, due to the common affect of the problem across multiple hosts and multiple services, and due to the incorrect TCP checksum seen at the host interface, I concluded that;

  1. I either had problem hardware (as all hosts were built from
    identical hardware purchased at the same point in time from
    the same supplier - but were assembled by me)

  2. or I had a dodgy router/firewall upstream from these hosts

The second conclusion was going to be the easiest to test, as I didn't have alternative parts for these boxes, not lying around anyway.


The solution;

Upstream I had a small router-firewall appliance of a sort I'd not used before; I typically deploy iptables boxes on hardened Linux platforms as these seem to be the most robust. This particular router-firewall had a couple of "native" security features which I'd taken advantage of - something the vendor called "Port Scan and DOS Protection". The Port Scan detection was working, so I left them running.

However, given the TCP checksum issue, I concluded that the Denial of Service Protection was probably trying to do its own version of SYN Cookies (or something similar), which was requiring packets to be re-assembled in-line, before reaching the hosts; thus being the most likely cause of my problem.

As it turns out, disabling this service has, so far, proven to resolve my INVALID packet issue.

I hope that this information, along with AlexK's original thread, remain available to help other future surfers who are attempting to resolve this very intricate fault.

AlexK, thanks for posting the detail you did; you helped to shorten my root-cause analysis. If you've been waiting for two years to resolve this problem, then I'm glad I could help (and sorry I wasn't here sooner ;-))

Thanks,

Ian.

[edited by: encyclo at 11:12 am (utc) on Nov. 10, 2007]

AlexK




msg:3509462
 4:08 am on Nov 20, 2007 (gmt 0)

Many thanks, IPTables Friend, for your Sticky, and response to this thread.

If you've been waiting for two years to resolve this problem

One of the problems with no-logging is that no-notice means no-attention; I have absolutely no idea what is happening at this moment with this issue on my current server.

So the real piece of missing information for AlexK (and myself originally) was that the INVALID state match doesn't just match on "out of state" packets - it also matches packets with other bad attributes, including an incorrect TCP checksum.

That is certainly useful info, and helps to explain why so many Invalid packets.

My site is now on a different box with a different host. In my experience with commercial hosts, the chances of them tracking down badly-configured routers is zero. My current host has a badly-configured DHCP server that causes on-the-fly IP-address renewals to fail; every server's message log fills up with "last message repeated 5 times" messages after 20 or so hours, so I have a cron-job on my server to restart the network each day. Asking them to search out a badly-configured router... I think not!

Finally, and for the benefit of anybody not using IPTables, I recently switched the log-analysis to Full, and finally understood the reason for many of the rejected packets (in the following partial snapshot, understand that most ports on this server are set to DROP packets; this is just one day on a low-volume server):
--------------------- iptables firewall Begin ------------------------

Logged 1129 packets on interface eth0
...
From 87.105.27.90 - 4 packets
To 87.106.136.186 - 4 packets
Service: epmap (tcp/135) (INPUT packet died:) - 2 packets
Service: netbios-ssn (tcp/139) (INPUT packet died:) - 2 packets
From 87.105.32.89 - 8 packets
To 87.106.136.186 - 8 packets
Service: epmap (tcp/135) (INPUT packet died:) - 1 packet
Service: netbios-ssn (tcp/139) (INPUT packet died:) - 2 packets
Service: microsoft-ds (tcp/445) (INPUT packet died:) - 3 packets
Service: ms-sql-s (tcp/1433) (INPUT packet died:) - 2 packets
From 87.105.37.151 - 7 packets
To 87.106.136.186 - 7 packets
Service: epmap (tcp/135) (INPUT packet died:) - 2 packets
Service: netbios-ssn (tcp/139) (INPUT packet died:) - 3 packets
Service: ms-sql-s (tcp/1433) (INPUT packet died:) - 2 packets

Thanks again, Ian.

craig1972




msg:3574900
 3:10 pm on Feb 14, 2008 (gmt 0)

This is a very informative thread, but it is still sparse on what one can *do* about the synflood attacks?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Hardware and OS Related Technologies / Linux, Unix, and *nix like Operating Systems
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved