Welcome to WebmasterWorld Guest from 220.127.116.11
Forum Moderators: bakedjake
# uname -a
Linux mysite.com 2.6.9-22.0.2.ELsmp #1 SMP Tue Jan 17 07:10:04 CST 2006 i686 i686 i386 GNU/Linux
Firewall built using /etc/init.d/iptables-live by my (very well experienced, very helpful) host. The very first lines in the
LOG all -- 0.0.0.0/0 0.0.0.0/0 state INVALID LOG flags 0 level 4 prefix `Invalid packet: '
DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID
fgrep -c 'Invalid packet' /var/log/messages*
fgrep '18.104.22.168' /var/log/messages
Jan 23 17:14:26 olivia kernel: Invalid packet: IN=eth0 OUT= MAC=00:20:ed:82:c6:f5:00:30:f2:10:b0:00:08:00 SRC=22.214.171.124 DST=126.96.36.199 LEN=40 TOS=0x00 PREC=0x00 TTL=231 ID=13832 DF PROTO=TCP SPT=1109 DPT=80 WINDOW=0 RES=0x00 ACK RST URGP=0
Jan 23 17:14:28 olivia kernel: Invalid packet: IN=eth0 OUT= MAC=00:20:ed:82:c6:f5:00:30:f2:10:b0:00:08:00 SRC=188.8.131.52 DST=184.108.40.206 LEN=40 TOS=0x00 PREC=0x00 TTL=231 ID=13833 DF PROTO=TCP SPT=1110 DPT=80 WINDOW=0 RES=0x00 ACK RST URGP=0
It's pretty common for an attack to take the form of a 'packet injection' whereby the attacker tries to gain control of a system by guessing the packet stream's numbering sequence, and then to take control of the packet stream by injecting a forged packet with an address within the guessed range. A successful injection could cause a valid request to become corrupted and possibly result in the compromise of the machine. (Obviously more details are available in lots of different places on the 'net ...)
Sounds like it would be a good idea for you to keep dropping those packets. Your real users probably won't be sending invalid packets at a rate that would cause their valid requests to fail.
BTW, your sample IP came from Indonesia ... according to APNIC [apnic.net], which is fairly popular territory from which this type of abuse flows ...
It is the sheer scale of the Invalid packets that is defeating my mind, and causes me to wonder whether I have hit some sort of bug in the Firewall or the kernel. Consider the following:
# egrep -c "Jan 24(.*)Invalid packet(.*)PROTO=TCP(.*)DPT=80" /var/log/messages
# fgrep -c "24/Jan" /var/log/httpd/co*-access_log
Now, if you are saying that more than 1 in 10 of all visitors are either trying to hack the site, or have such badly mis-configured TCP stacks that their page requests are utterly Invalid, and that that is just how life is today, then, ... well, I shall be flabbergasted, but have to accept it.
But surely not...
98,203 hits on the website from 5,258 humans also produced 5,339 bad packets with 603 different IP-addresses
Why not? Thinking aloud, if you *are* being attacked by someone interested in executing a packet injection hack, you might end up with all of the 'real' requests being satisfied with your 'real' visitors none the wiser. For each 'real' visitor and their accompanying connection, the attacker (who may be port monitoring your system) makes at least one attempt to hijack the packet stream ... and fails ... resulting in at least one invalid packet per 'real' visitor.
The nature of a packet injection attack is such that an existing packet stream and a valid connection must be established, then comes the port monitoring to view that packet stream's configuration, then comes the guessing at the next packet number in the sequence (while valid packets are flowing freely), then comes the attempt to inject the forged packet and finally the file buffer is overflowed or whatever 'crowbar' is being applied to compromise the system after a successful injection. An incorrect guess at the packet's sequence number will cause the forged packet to simply bounce off (invalid packet because it doesn't fit into the request sequence), as the original request/exchange is handled normally.
I must confess that my own servers show none of this activity (0 count in all messages logs), and we get bombarded with various kinds of attacks every day. Maybe nobody is trying a packet injection on us, or maybe PortSentry is keeping anyone from monitoring our ports successfully, or maybe I'm off base.
Hopefully, conjecture may lead to truth ... ;)
As an aside, the key to packet injection is guessing the next
assignment correctly, or at least guessing within a range that has not yet been qualified and is still within range of the last packet of the sequence.
In your example, note the two sequential IDs. Are those the only two entries for that IP?
Lastly, perhaps, the 'Invalid' connection state your rule is keying on refers to packets that are 'not part of any connection', according to my iptables, so ... hmmm ... still feeling pretty good about attempted packet injection attacks from zombie servers somewhere in South Asia.
I took 3 hours out to look closely at 31 IPs from the total of 504 from Tue Jan 24 (one day in the life of a web-server). Only 2 IPs did not appear in the Apache logs. Of the other 29 IPs, 22 supplied Invalid packets well after the last hit, and only 7 were blocked during mid-page load (they all got the html page; it was .png files, etc that suffered).
What is very interesting in this is that half of the IPs made only 1 successful page-request, then a second request which totally died (because it was blocked due to Invalid packets). The single-page bounce-rate stands at 53% on my site. That has worried me immensely, and it is some (bitter) comfort to realise that it is so high due to mis-configured machines.
KeepAlive is active on my Apache webserver. Does any one know if there are any problems with this?
Tue Jan 24 - one day in the life of a web-server:
sample of 31 of 504 TCP:80 IPs blocked due to Invalid Packets:
(603 total when UDP packets included)
2 have no entry in the access_log
15 make 1 page request
8 make 2 page requests
1 make 3 page requests
3 make 4 page requests
1 make 5 page requests
1 make 8 page requests
(ignoring the 2 no-access-log, and considering the 29 others):
No of Invalid packets AFTER successful page request: 22
No of Invalid packets DURING successful page request: 7
MSIE 6.0; Windows NT 5.1 20
MSIE 6.0; Windows NT 5.0 5
MSIE 5.0; Windows 98 1
MSIE 6.0; Windows 98 2
Firefox/1.5 Win98 1
Windows XP 73.8%
Windows 2k 10.7%
Windows 98 8.3%
IP-Address - No of Invalid packets - Page-access Time - Browser + OS - Invalid Packet Time (first-last)
220.127.116.11 - 2 packets to tcp(80) 12:06:03 MSIE 6.0; Windows NT 5.1 - 12:09:57
18.104.22.168 - 12 packets to tcp(80) 23:50:35 MSIE 5.0; Windows 98 - 23:53:19-23:54:34
22.214.171.124 - 1 packet to tcp(80) 12:48:55 MSIE 6.0; Windows NT 5.0 - 12:56:17
126.96.36.199 - 10 packets to tcp(80) 21:20:03 MSIE 6.0; Windows NT 5.1 - 21:22:15-21:27:06
188.8.131.52 - 2 packets to tcp(80) - Jan 22 22:25:01
Jan 22 23:30:29
Jan 24 12:56:43
Jan 24 13:26:07
184.108.40.206 - 9 packets to tcp(80,80) 18:03:53 Firefox/1.5 Win98 - 18:07:53-18:17:53
220.127.116.11 - 68 packets to tcp(80,80) 11:38:43 MSIE 6.0; Windows NT 5.0 - 11:40:09-11:53:42
18.104.22.168 - 23 packets to tcp(80,80) 17:17:19 MSIE 6.0; Windows NT 5.1 - 17:19:36-17:31:51
22.214.171.124 - 63 packets to tcp(80,80) 13:33:32 MSIE 6.0; Windows NT 5.1 - 13:35:47-13:49:28
126.96.36.199 - 12 packets to tcp(80) 19:27:40 MSIE 6.0; Windows NT 5.1 - 19:30:41-19:38:52
188.8.131.52 - 2 packets to tcp(80) 23:07:40 MSIE 6.0; Windows NT 5.1 - 23:19:03
184.108.40.206 - 2 packets to tcp(80) 20:00:33 MSIE 6.0; Windows NT 5.0 - 20:07:31
220.127.116.11 - 2 packets to tcp(80) 20:47:48 MSIE 6.0; Windows NT 5.1 - 20:50:00
18.104.22.168 - 42 packets to tcp(80) 17:21:21 MSIE 6.0; Windows 98 - 17:21:41-17:34:38
22.214.171.124 - 1 packet to tcp(80) 12:59:41 MSIE 6.0; Windows NT 5.0 - 13:09:53
126.96.36.199 - 2 packets to tcp(80) 09:42:01 MSIE 6.0; Windows NT 5.1 - 09:47:24
188.8.131.52 - 2 packets to tcp(80) 16:12:36 MSIE 6.0; Windows NT 5.0 - 16:20:36
184.108.40.206 - 9 packets to tcp(80) 07:41:08 MSIE 6.0; Windows NT 5.1 - 07:50:48-08:01:06
220.127.116.11 - 6 packets to tcp(80) 13:57:12 MSIE 6.0; Windows NT 5.1 - 14:03:33-14:11:17
18.104.22.168 - 1 packet to tcp(80) 10:45:35 MSIE 6.0; Windows NT 5.1 - 10:47:52
22.214.171.124 - 11 packets to tcp(80) 13:22:32 MSIE 6.0; Windows NT 5.1 - 13:29:23-13:29:59
126.96.36.199 - 4 packets to tcp(80) 07:58:23 MSIE 6.0; Windows NT 5.1 - 08:01:23-08:06:37
188.8.131.52 - 19 packets to tcp(80) 07:36:04-07:37:00
184.108.40.206 - 1 packet to tcp(80) 12:00:04 MSIE 6.0; Windows NT 5.1 - 12:04:25
220.127.116.11 - 12 packets to tcp(80) 05:18:36 MSIE 6.0; Windows 98 - 05:21:08-05:22:31
18.104.22.168 - 1 packet to tcp(80) 02:34:12 MSIE 6.0; Windows NT 5.1 - 02:43:35
22.214.171.124 - 10 packets to tcp(80) 09:30:49 MSIE 6.0; Windows NT 5.1 - 09:40:44-09:41:23
126.96.36.199 - 13 packets to tcp(80) 19:30:04 MSIE 6.0; Windows NT 5.1 - 19:33:30-19:34:10
188.8.131.52 - 2 packets to tcp(80) 03:16:39 MSIE 6.0; Windows NT 5.1 - 03:26:10
184.108.40.206 - 2 packets to tcp(80) 08:21:20 MSIE 6.0; Windows NT 5.1 - 08:28:27
220.127.116.11 - 1 packet to tcp(80) 18:01:26 MSIE 6.0; Windows NT 5.1 - 18:02:46
I find it difficult to believe that (more than) 1 in 10 are doing that
Aahh ... but more than likely NONE of your 'real' visitors are doing that. Let me explain ...
Your visitors are simply ... visiting. They are not the problem. They provide the conduits.
If my theory holds, the hacker(s) is lurking ... hanging out around your ports, sniffing the 'real' visitors' packets for info. With each 'real' visitor, the un-related hacker tries to inject a forged packet by piggy-backing in on the 'real' visitors connection. They are not requesting anything, they are trying to wedge themselves into the valid packet streams being provided by your 'real' visitors.
If you logged ALL packets (yikes!), you'd see a valid stream something like:
where the OK packets are the 'real' stream and the BAD packet is the injection attempt. It should (in an ideal hack) look nearly exactly like the 'real' packets. The failed attempt has no impact on the 'real' visitor's experience, as all of the valid request's packets make it through just fine.
A forged packet is pretty simple to make out of an 8-bit file, and its transmission profile can contain any kind of information the hacker wants to send. Better hackers make their packets look like 'real' traffic ... with a user agent and everything ... preferably identical (except for the guessed ID= attribute) to 'real' packets being observed in any particular stream. When a packet is suspect ... all of the data it provides is suspect, too. If a packet looks unusual, it draws attention to itself. Much better for it to look like a companion packet in a stream of valid packets that had a little trouble getting validated.
Your list contains invalid packets from Chile, The Netherlands, Indonesia, London, Tehran (more than once), and several other notoriously-porous blocks. Sure, there are a few from the U.S. or from other 'trusted' blocks, but ... hey ... there are loose installations all over the world.
Why would a 'real' packet stream contain invalid packets?
1) Mis-configuration of the protocol stack(s) ... if not on the client end then on the server end.
2) Timeout issues that may close a connection erroneously before all packets have been received.
3) Random packets that really aren't part of any valid connection ... maybe they've been drifting around in cyberspace for awhile and finally make it to the destination after the connection has been closed.
4) Maybe the invalid packets are forged.
Of these (and I'm wide open to others), #3 is kind of possible, but not likely at the rates you noted, and #4 is the most likely, given the patterns you describe.
It still smells like an attack or two (or many).
I'm adding your logging function to my logs, to see what's going on with our otherwise cool servers, and to compare with your experience so I may understand it better.
The failed attempt has no impact on the 'real' visitor's experience
The most common symptom (in the Apache logs) was just the html page, style sheet + one graphic file loaded (there are 7-10 graphics files required). Other symptoms were multiple requests for the same file (the style sheet comes to mind, but it varied), or a duplicate 200 load of graphic files for a second page, then 304s on the 3rd page. Then there are all the folks with Invalid packets 4-10 minutes after the first page, and nothing whatsoever in the apache logs. There isn't the slightest doubt that these folks were having problems getting a full-page load (html + all helper files). The big question, of course, is why?
The big question for me is whether I can trust the Firewall+kernel diagnosing an invalid packet? Could it be a bug in either, or in Apache?
I am fully willing to keep the block if I cannot find a bug elsewhere. The one change that perhaps need making is not to BLOCK but rather to REJECT the packet. The latter sends a message which will allow the client to recover promptly. The former (current behaviour) does not inform, and the client will then have to rely on the TCP timeout, which I understand to be 30 secs. The current behaviour will lead to a very poor perception of my website amongst more than 1 in 10 visitors.
Another interesting feature is how few Firewall scripts out there include this specific coding (block on Invalid), and that also begs the question why?
Is this server running through a proxy?
Is this an all-day-every-day situation? Your inital post seemed to include numbers that started low, peaked and then dropped off. Maybe that's the pattern of your traffic? Are the valid:invalid ratios consistent over time?
The data you provided (if we can trust it) indicates just a few dropped packets per request. 2 bad packets shouldn't keep a visitor from enjoying all of your page elements.
I dunno ... but I still can't get off of the attack scenario. Since I modified my iptables, this morning, I haven't seen any bad packet activity at all. There have been many attack attempts amidst many thousands of requests, and my PortSentry [sourceforge.net] has been deflecting attacks all day, but no invalid packets. Of course, if PortSentry is being completely successful (as usual) then any attempt to lurk and port monitor is being thwarted, anyway, so there is no opportunity to guess at the packet sequence and/or launch an injection attack. Without the lurking there can be no guessing.
Maybe the attacker (if they exist) is trying to create their own valid connection and ride their own wave into your server? The concept is the same, even if the disguise and execution is pretty weak.
I'm not trying to be dense or too paranoid, but all of your evidence still seems to support the attack scenario, IMHO. It's a tough one, all right.
Although it is a far out idea ... maybe there's an issue with the NIC or its interface at your server or with some other piece of hardware having intermittent problems? Not pretty, but I'm at a bit of a loss.
Are your visitors actually receiving only partial pages/page elements?A very large number of the ones that I investigated above.
Have you had trouble yourself?Never noticed any.
Is this server running through a proxy?No.
Is this an all-day-every-day situation?Yes.
Your inital post seemed to include numbers that started low, peaked and then dropped off. Maybe that's the pattern of your traffic?Isn't that the pattern of *everybody's* traffic?
Are the valid:invalid ratios consistent over time?That would involve analysis that I am not, at this moment, prepared to do.
2 bad packets shouldn't keep a visitor from enjoying all of your page elements.Just 1 dropped packet will shaft at least one page element.
Did this just start one day, or has it been a featured issue the entire time you've had this server?
I mean, if so many visitors are experiencing it (although you haven't, yourself), then it seems likely there's a fundamental issue with your box, either some config thing or some hardware thing.
I've known NICs to munge packet data when they're not working properly, and encryption errors or timing issues could come into play. But I've never had invalid packet issues (or visitors receiving incomplete pages) with a server unless it was being attacked.
Back to the research board, I guess ... sorry to not come up with a simple answer for ya, AlexK. I really appreciate the work you've done with blocking bad bots.
has it been a featured issue the entire time you've had this server?
It is worth pointing out, I think, that I've been concentrating on TCP:80 packets. There are also a very large number of UDP and also some TCP:some-port-other-than-80 packets also being rejected (on 24 Jan, 504 TCP:80 IPs and 99 not-TCP:80 IPs). They are definately attacks (unless you can suggest a good reason why UDP packets should be coming at my box).
The conversation has been very useful - the practical value for me has been to perceive that, assuming no bugs--hardware or software--come to light, the TCP:80 packets should be REJECTed rather than DROPped. That one step should improve the perception of the site enormously. I would rather get to the source of the problem, however.
If it's hardware, ...
13 months ago I discovered that a memory module had gone bad in my server [webmasterworld.com], something that my stupid hosts-at-the-time-but-not-for-much-longer never did actually fix (they replaced the bad module, but it was part of a matched-pair, so other, similar, problems began to appear).
...suspect it might be intermittently bad
Yes, hardware can present intermittant problems whilst in the process of breaking down but, the very core of hardware is, once bad always bad. So, hardware presents consistently-bad issues. It is software that manages to give sometimes-good sometimes-bad symptoms.
*All* Invalid packets (in the current set of logs) are inbound.
My server is getting a number of Invalid ICMP packets. These are either Type 3 (61%) (Destination Unreachable) or Type 11 (36%) (Time Exceeded) (119 total in one log, with 3 unknown). Here is a small sample from log.1 (for reasons which become obvious in a later post):
Jan 17 12:25:21 olivia kernel: Invalid packet: IN=eth0 OUT= MAC=00:20:ed:82:c6:f5:00:30:f2:10:b0:00:08:00 SRC=18.104.22.168 DST=22.214.171.124 LEN=68 TOS=0x00 PREC=0xC0 TTL=239 ID=3681 PROTO=ICMP TYPE=3 CODE=1 [SRC=126.96.36.199 DST=188.8.131.52 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=3423 DF PROTO=TCP SPT=80 DPT=1331 WINDOW=31592 RES=0x00 ACK FIN URGP=0 ]
Jan 17 16:29:17 olivia kernel: Invalid packet: IN=eth0 OUT= MAC=00:20:ed:82:c6:f5:00:30:f2:10:b0:00:08:00 SRC=184.108.40.206 DST=220.127.116.11 LEN=56 TOS=0x00 PREC=0x00 TTL=101 ID=30404 PROTO=ICMP TYPE=3 CODE=2 [SRC=18.104.22.168 DST=22.214.171.124 LEN=244 TOS=0x00 PREC=0xC0 TTL=118 ID=4920 PROTO=46 ]
Jan 17 16:51:13 olivia kernel: Invalid packet: IN=eth0 OUT= MAC=00:20:ed:82:c6:f5:00:30:f2:10:b0:00:08:00 SRC=126.96.36.199DST=188.8.131.52 LEN=56 TOS=0x00 PREC=0x00 TTL=238 ID=10789 PROTO=ICMP TYPE=3 CODE=13 [SRC=184.108.40.206 DST=220.127.116.11 LEN=44 TOS=0x00 PREC=0x00 TTL=254 ID=1 PROTO=TCP INCOMPLETE [8 bytes] ]
Jan 17 22:35:36 olivia kernel: Invalid packet: IN=eth0 OUT= MAC=00:20:ed:82:c6:f5:00:30:f2:10:b0:00:08:00 SRC=18.104.22.168 DST=22.214.171.124 LEN=56 TOS=0x00 PREC=0x00 TTL=45 ID=7427 PROTO=ICMP TYPE=3 CODE=1 [SRC=126.96.36.199 DST=188.8.131.52 LEN=644 TOS=0x00 PREC=0x00 TTL=48 ID=19937 DF PROTO=TCP INCOMPLETE [8 bytes] ]
Jan 17 22:37:13 olivia kernel: Invalid packet: IN=eth0 OUT= MAC=00:20:ed:82:c6:f5:00:30:f2:10:b0:00:08:00 SRC=184.108.40.206 DST=220.127.116.11 LEN=56 TOS=0x00 PREC=0x00 TTL=45 ID=9987 PROTO=ICMP TYPE=3 CODE=1 [SRC=18.104.22.168 DST=22.214.171.124 LEN=644 TOS=0x00 PREC=0x00 TTL=48 ID=19939 DF PROTO=TCP INCOMPLETE [8 bytes] ]
Jan 18 03:51:48 olivia kernel: Invalid packet: IN=eth0 OUT= MAC=00:20:ed:82:c6:f5:00:30:f2:10:b0:00:08:00 SRC=126.96.36.199DST=188.8.131.52 LEN=56 TOS=0x00 PREC=0x00 TTL=110 ID=1115 PROTO=ICMP TYPE=11 CODE=1 [SRC=184.108.40.206 DST=220.127.116.11 LEN=1468 TOS=0x00 PREC=0x00 TTL=46 ID=62151 MF PROTO=TCP INCOMPLETE [8 bytes] ]
I switched the DROP off for 48+ hours, but kept the LOG, then compared results. There was just one instance found (not exhaustive) where an IP sent Invalid packets *and* was caught by another rule:
Jan 26 03:39:49 olivia kernel: Invalid packet: IN=eth0 OUT= MAC=00:20:ed:82:c6:f5:00:30:f2:10:b0:00:08:00 SRC=18.104.22.168 DST=22.214.171.124 LEN=52 TOS=0x00 PREC=0x00 TTL=41 ID=62431 PROTO=TCP SPT=64852 DPT=80 WINDOW=65535 RES=0x00 ACK FIN URGP=0(both occurred many times with this IP; above is just a sample)
Jan 26 03:42:05 olivia kernel: New not syn: IN=eth0 OUT= MAC=00:20:ed:82:c6:f5:00:30:f2:10:b0:00:08:00 SRC=126.96.36.199 DST=188.8.131.52 LEN=40 TOS=0x00 PREC=0x00 TTL=41 ID=4270 PROTO=TCP SPT=25275 DPT=80 WINDOW=65535 RES=0x00 ACK URGP=0
Jan 26 03:42:07 olivia kernel: New not syn: IN=eth0 OUT= MAC=00:20:ed:82:c6:f5:00:30:f2:10:b0:00:08:00 SRC=184.108.40.206 DST=220.127.116.11 LEN=40 TOS=0x00 PREC=0x00 TTL=41 ID=44529 PROTO=TCP SPT=25557 DPT=80 WINDOW=65535 RES=0x00 ACK URGP=0
Going through the same exhaustive process as msg #:6+7 also produced little difference: there were perhaps more complete pages loaded, but still the same catalogue of partial page-loads (graphics files missing), multiple requests for the style sheet, etc. etc.. The only thing that I did get for my trouble was an extremely wierd page request [webmasterworld.com].
In the end, it seems that switching the DROP off offers little or no benefit to site visitors, and may clog up the server. So, it has been reinstituted, and the LOG has been dropped instead.
Here is a sight of the relevant part of the iptables script:
# bad_packets chain
# Drop INVALID packets immediately
# 2006-01-28 logging for INVALID switched off -AK
# log is filling with >500 IPs daily due to foll rule
# $IPT -A bad_packets -p ALL -m state --state INVALID -j LOG \
# --log-prefix "Invalid packet: "
$IPT -A bad_packets -p ALL -m state --state INVALID -j DROP
# Then check the tcp packets for additional problems
$IPT -A bad_packets -p tcp -j bad_tcp_packets
# All good, so return
$IPT -A bad_packets -p ALL -j RETURN
# bad_tcp_packets chain
# All tcp packets will traverse this chain.
# Every new connection attempt should begin with
# a syn packet.
# 2006-01-28 added -AK
# Act honourably on behalf of others receiving TCP
# Sequence Number Prediction attacks (these attacks
# rely on the spoofed host DROPping unknown SYN/ACKs
# rather than REJECTing them).
# See [iptables-tutorial.frozentux.net...]
$IPT -A bad_tcp_packets -p tcp --tcp-flags SYN,ACK SYN,ACK \
-m state --state NEW -j REJECT --reject-with tcp-reset
# Fix iptables feature of allowing NEW packets with SYN bit unset
# See [iptables-tutorial.frozentux.net...]
$IPT -A bad_tcp_packets -p tcp! --syn -m state --state NEW -j LOG \
--log-prefix "New not syn: "
$IPT -A bad_tcp_packets -p tcp! --syn -m state --state NEW -j DROP
Thu Jan 26 - one day in the life of a web-server:
sample of 36 of 408 TCP:80 IPs logged due to Invalid Packets:
(547 total when none-TCP:80 packets included)
(40 visitors due to IP-sharing)
5 have no entry in the access_log
18 make 1 page request
4 make 2 page requests
7 make 3 page requests
2 make 4 page requests
1 make 6 page requests
1 make 7 page requests
1 make 10 page requests
1 make 12 page requests
(ignoring the 5 no-access-log, and considering the 35 others):
No of Invalid packets AFTER successful page request: 28
No of Invalid packets DURING successful page request: 7
MSIE 6.0; Windows NT 5.1 26
MSIE 6.0; Windows NT 5.0 3
MSIE 5.5; Windows NT 5.0 2
MSIE 6.0; Windows 98 1
Firefox/1.0.4 Windows NT 5.0 1
Firefox/1.5 Windows NT 5.1 3
IP-Address No of Invalid packets Page-access Time - Browser + OS Packet Time (first-last)
18.104.22.168 1 packet to tcp(80) 03:07:10 MSIE 6.0; Windows 98 03:11:00 Invalid
22.214.171.124 2 packets to tcp(80) 17:59:43 MSIE 6.0; Windows NT 5.1 20:25:48 Invalid
126.96.36.199 2 packets to tcp(80) 20:20:40 MSIE 6.0; Windows NT 5.1 20:28:09 Invalid
188.8.131.52 2 packets to tcp(80) 23:44:16 MSIE 6.0; Windows NT 5.1 23:50:07 Invalid
184.108.40.206 2 packets to tcp(80) 03:04:02 MSIE 6.0; Windows NT 5.1 03:13:01 Invalid
220.127.116.11 2 packets to tcp(80) 15:04:50 MSIE 6.0; Windows NT 5.0 15:09:17 Invalid
18.104.22.168 2 packets to tcp(80) 17:53:15 MSIE 6.0; Windows NT 5.1 18:00:21 Invalid
22.214.171.124 1 packet to tcp(80) 17:45:39 Firefox/1.0.4 Windows NT 5.0 17:53:10 Invalid
126.96.36.199 2 packets to tcp(80) 20:20:42 MSIE 6.0; Windows NT 5.1 20:27:26 Invalid
188.8.131.52 54 packets to tcp(80,80)09:07:41 MSIE 6.0; Windows NT 5.1 09:09:58-09:18:43 (New not syn)
16:33:44 MSIE 6.0; Windows NT 5.0 16:35:59-16:44:48 (New not syn)
16:33:44 16:45:59-16:46:03 (Invalid)
184.108.40.206 16 packets to tcp(80,80)03:44:53 MSIE 6.0; Windows NT 5.1 03:47:09-03:56:00 (New not syn)
220.127.116.11 2 packets to tcp(80) 20:22:20 MSIE 6.0; Windows NT 5.1 20:29:27 Invalid
18.104.22.168 4 packets to tcp(80,80) 13:20:26 MSIE 5.5; Windows NT 5.0 13:22:53-13:25:23 (New not syn)
22.214.171.124 9 packets to tcp(80,80) 12:07:06 MSIE 5.5; Windows NT 5.0 13:22:42-13:22:51 (New not syn)
13:20:24 13:23:41-13:23:45 (Invalid)
126.96.36.199 20 packets to tcp(80,80)13:23:36 MSIE 6.0; Windows NT 5.1 13:25:53-21:50:28 (Invalid)
13:25:53 13:28:09-21:49:13 (New not syn)
188.8.131.52 1 packet to tcp(80) 23:12:38 MSIE 6.0; Windows NT 5.1 23:22:48-02:28:21 Invalid
184.108.40.206 2 packets to tcp(80) 14:04:03 MSIE 6.0; Windows NT 5.1 14:13:07-14:13:09 Invalid
220.127.116.11 1 packet to tcp(80) 23:46:38 MSIE 6.0; Windows NT 5.1 00:03:27 Invalid
18.104.22.168 1 packet to tcp(80) 03:00:22 MSIE 6.0; Windows NT 5.1 03:03:02 Invalid
03:20:24 Firefox/1.5 Windows NT 5.1
22.214.171.124 36 packets to tcp(80) 15:42:50 MSIE 6.0; Windows NT 5.1 15:42:59-15:45:05 Invalid
126.96.36.199 5 packets to tcp(80) 07:11:16 Firefox/1.5 Windows NT 5.1 07:14:03-07:19:47 Invalid
188.8.131.52 22 packets to tcp(80,80)09:57:48 MSIE 6.0; Windows NT 5.1 10:01:13-10:04:58 (New not syn)
09:58:36 10:05:58 Invalid
09:58:51 12:01:00-12:04:46 (New not syn)
09:58:58 12:05:18 Invalid
11:56:57 MSIE 6.0; Windows NT 5.1 15:27:25-15:29:55 (New not syn)
15:25:04 MSIE 6.0; Windows NT 5.1 15:30:52-15:31:00 Invalid
184.108.40.206 1 packet to tcp(80) 17:06:40 Invalid
220.127.116.11 4 packets to tcp(80) 07:06:07 MSIE 6.0; Windows NT 5.1 07:08:54-07:31:23 Invalid
18.104.22.168 1 packet to tcp(80) 07:21:11 MSIE 6.0; Windows NT 5.1 07:23:17 Invalid
22.214.171.124 2 packets to tcp(80) 05:02:25 MSIE 6.0; Windows NT 5.1 05:04:15 Invalid
126.96.36.199 2 packets to tcp(80) 01:47:14 MSIE 6.0; Windows NT 5.1 01:51:20 Invalid
188.8.131.52 1 packet to tcp(80) 12:28:03 Invalid
184.108.40.206 1 packet to tcp(80) 01:45:01 Invalid
220.127.116.11 1 packet to tcp(80) 01:45:01 Invalid
18.104.22.168 1 packet to tcp(80) 12:28:03 Invalid
22.214.171.124 4 packets to tcp(80) 12:56:26 MSIE 6.0; Windows NT 5.0 13:05:42-13:09:47 Invalid
126.96.36.199 18 packets to tcp(80,80)08:32:59 MSIE 6.0; Windows NT 5.1 08:35:16-08:44:01 (New not syn)
188.8.131.52 1 packet to tcp(80) 08:36:00 Firefox/1.5 Windows NT 5.1 09:19:48 Invalid
09:10:21 MSIE 6.0; Windows NT 5.1
184.108.40.206 2 packets to tcp(80) 16:52:44 MSIE 6.0; Windows NT 5.1 17:05:48 Invalid
220.127.116.11 2 packets to tcp(80) 13:11:59 MSIE 6.0; Windows NT 5.1 13:21:46 Invalid
I wanted a send a REJECT for INVALID tcp-packets so that the clients could clean up properly. That would send an ICMP packet Type-3 (Destination Unreachable) with a choice of:
icmp-net-unreachable (Code 0)
icmp-host-unreachable (Code 1)
icmp-port-unreachable (Code 2) (default)
icmp-proto-unreachable (Code 3)
icmp-net-prohibited (Code 9)
icmp-host-prohibited (Code 10)
None of the codes seemed correct. Possibly Code 13 (Communication Administratively Prohibited [RFC1812]) would have been right but, unless I am missing something, none of the codes said "hey, you have sent an INVALID packet" so, I threw up my hands and decided to simply DROP them (which does not send any notification).
Since the packet has got RESET set, I'm pretty sure the other side wasn't hoping for more data :)
The problem starts out in a straight forward way;
- User level observation is an intermittently failing protocol;
But the problem is hard to diagnose;
- System level observation is an INVALID packet match;
iptables -A INPUT -m state --state INVALID -j LOG
--log-level info --log-prefix "INVALID (input):"
iptables -A INPUT -m state --state INVALID -j DROP
xx xx xx:xx:xx localhost kernel: INVALID (input):
IN=eth1 OUT= MAC= SRC=xx.xx.xx.xx DST=xx.xx.xx.xx
LEN=561 TOS=0x0C PREC=0x20 TTL=56 ID=2945 PROTO=TCP
SPT=34920 DPT=80 WINDOW=183 RES=0x00 ACK PSH URGP=0
To the point that it took me a week of on-again off-again testing before I got to an indication of the cause;
- Packet level observation is a TCP packet that reportedly
contains an incorrect checksum;
tcpdump -i eth1 -s1500 -vvv port 80 ¦ grep -i incorrect
xx:xx:xx.577564 IP (tos 0x2c, ttl 56, id 2945, offset 0,
flags [none], proto: TCP (6), length: 561)
sourcehost.34920 > desthost.http: P,
cksum 0x39fd (incorrect (-> 0xfc9e),
1:522(521) ack 1 win 183
Like AlexK, I wasn't prepared to believe that this was typical hacker crap. Unlike AlexK, I found that I could intermittently reproduce the problem using one of my own Internet connections (so I could confirm his and my theory, and create a somewhat slow and painful testing platform -- but slightly faster than "wait-n-see").
So the real piece of missing information for AlexK (and myself originally) was that the INVALID state match doesn't just match on "out of state" packets - it also matches packets with other bad attributes, including an incorrect TCP checksum. This isn't commonly discussed, but is noted around the traps .. ie;
In my case, I saw this problem on all of the hosts on my public network. This is odd as my iptables scripts have evolved across the past 10 years, with this particular install being a slightly improved version again - but with no substantial changes. And this was certainly something that I hadn't seen before.
While testing is still running, in my case, due to the common affect of the problem across multiple hosts and multiple services, and due to the incorrect TCP checksum seen at the host interface, I concluded that;
The second conclusion was going to be the easiest to test, as I didn't have alternative parts for these boxes, not lying around anyway.
Upstream I had a small router-firewall appliance of a sort I'd not used before; I typically deploy iptables boxes on hardened Linux platforms as these seem to be the most robust. This particular router-firewall had a couple of "native" security features which I'd taken advantage of - something the vendor called "Port Scan and DOS Protection". The Port Scan detection was working, so I left them running.
However, given the TCP checksum issue, I concluded that the Denial of Service Protection was probably trying to do its own version of SYN Cookies (or something similar), which was requiring packets to be re-assembled in-line, before reaching the hosts; thus being the most likely cause of my problem.
As it turns out, disabling this service has, so far, proven to resolve my INVALID packet issue.
I hope that this information, along with AlexK's original thread, remain available to help other future surfers who are attempting to resolve this very intricate fault.
AlexK, thanks for posting the detail you did; you helped to shorten my root-cause analysis. If you've been waiting for two years to resolve this problem, then I'm glad I could help (and sorry I wasn't here sooner ;-))
[edited by: encyclo at 11:12 am (utc) on Nov. 10, 2007]
If you've been waiting for two years to resolve this problem
So the real piece of missing information for AlexK (and myself originally) was that the INVALID state match doesn't just match on "out of state" packets - it also matches packets with other bad attributes, including an incorrect TCP checksum.
My site is now on a different box with a different host. In my experience with commercial hosts, the chances of them tracking down badly-configured routers is zero. My current host has a badly-configured DHCP server that causes on-the-fly IP-address renewals to fail; every server's message log fills up with "last message repeated 5 times" messages after 20 or so hours, so I have a cron-job on my server to restart the network each day. Asking them to search out a badly-configured router... I think not!
Finally, and for the benefit of anybody not using IPTables, I recently switched the log-analysis to Full, and finally understood the reason for many of the rejected packets (in the following partial snapshot, understand that most ports on this server are set to DROP packets; this is just one day on a low-volume server):
--------------------- iptables firewall Begin ------------------------
Logged 1129 packets on interface eth0
From 18.104.22.168 - 4 packets
To 22.214.171.124 - 4 packets
Service: epmap (tcp/135) (INPUT packet died:) - 2 packets
Service: netbios-ssn (tcp/139) (INPUT packet died:) - 2 packets
From 126.96.36.199 - 8 packets
To 188.8.131.52 - 8 packets
Service: epmap (tcp/135) (INPUT packet died:) - 1 packet
Service: netbios-ssn (tcp/139) (INPUT packet died:) - 2 packets
Service: microsoft-ds (tcp/445) (INPUT packet died:) - 3 packets
Service: ms-sql-s (tcp/1433) (INPUT packet died:) - 2 packets
From 184.108.40.206 - 7 packets
To 220.127.116.11 - 7 packets
Service: epmap (tcp/135) (INPUT packet died:) - 2 packets
Service: netbios-ssn (tcp/139) (INPUT packet died:) - 3 packets
Service: ms-sql-s (tcp/1433) (INPUT packet died:) - 2 packets
Thanks again, Ian.