Working on Snort_inline 2.8.0.1

The last week I’ve been working on bringing Snort_inline to the Snort 2.8.0.1 level, including it’s IPv6 support. I’m almost ready to commit it to SVN, there are just some issues I need to fix in the inline specific code. The code will get rid of libdnet and use libnet 1.1 for sending reset/reject packets for both IPv4 and IPv6. After committing I will start working on getting the IPv6 features I wrote for NitroSecurity into this tree. This includes more matches, tunnel decoding (including for example the freenet6 tunnel, etc). So stay tuned!

New Snort_inline TCP window normalization code in SVN

A while ago I wrote about why the TCP window scaling normalization in Snort_inline was broken by design. I also wrote about a new solution I was working on and testing that would be uploaded to SVN soon. I just committed the patch to SVN. What it does is add two new options to stream4:

norm_window: normalize the TCP window (disabled by default). This is to protect Snort_inline from being forced to queue too many packets.
max_win_size: maximum size of the scaled TCP window. Packets increasing the window beyond the limit are modified.

This option is disabled by default, and the old wscale normalization code is removed, as are the options that configured it. It runs fine on my gateway, without noticeable slowdowns, but I haven’t done any benchmarking so far. Please try this and let me know how it works for you!

Multiple Snort_inline processes with Vuurmuur

One of the cool things of the Snort_inline project is the support for NFQUEUE. NFQUEUE is the new queuing mechanism to push packets from the kernel to userspace so a userspace program can issue a verdict on it. What makes NFQUEUE cooler than it’s predecessor ip_queue is that it supports multiple queue’s. This means that there can be more than one Snort_inline process inspecting and judging traffic. The challenge is to make sure that each Snort_inline instance sees all traffic belonging to a certain connection so Snort_inline can do stateful inspection on it. Luckily, Vuurmuur makes it very easy.

Normally an ‘accept’ rule in Vuurmuur looks like this:

accept service http from local.lan to world.inet options log

The NFQUEUE equivalent of this rule is:

nfqueue service http from local.lan to world.inet options log,nfqueuenum=”1″

To have ftp handled by another Snort_inline instance, just add a new rule:

nfqueue service ftp from local.lan to world.inet options log,nfqueuenum=”2″

Easy, no? 🙂 Vuurmuur creates the iptables rules that are required. It uses some advanced connmark-fu for it, so the right Snort_inline process receives all packets from a connection. It uses the helper match to make sure related connections are handled by the right queue, such as the ftp data channel. Of course you also need Snort_inline to be ready for it. See this post for more info on that.

The Snort_inline configuration part takes some work. You have to setup your init scripts to start all instances, setup different configs, logging to different locations. You will need multiple Barnyard’s and if using Sguil multiple snort_agent.tcl instances. When updating the rules you need to take care of the multiple processes as well. As said, it takes some work, but it’s rewarding. You can for example setup an extra Snort_inline instance for testing purposes only. Send all traffic from a certain IP to it to try out new rules, config changes, etc. I have set it up to have separate processes monitor my dmz and my lan.

What is possible as well, but not with Vuurmuur so far, is to have a form of poor man’s load balancing by sending new connections to one of multiple processes. This could be done by making use of the ‘ipt_statistics’ iptables module (fmr ipt_random). This allows a rule to be activated only some percent of the time. By using some more connmark-fu it’s possible to have multiple Snort_inline instances to handle different connections of the same type of traffic. I’ll add support for that to a future Vuurmuur release.

Libnet 1.1 IPv6 fixes and additions

Libnet is a cool packet crafting tool, used by Snort to send TCP reset packets and ICMP unreachable packets as part of active responses. Libnet 1.1 supports IPv6 which is what I needed for my work. After some reading and testing there were a few problems. First, while possible to send TCP reset packets, the packets didn’t have a correct checksum and debugging this with valgrind showed lots of memory errors. Second, ICMPv6 was only partly implemented. The libnet_build_* functions for it are missing. This is, by the way, quite a common picture. Many libraries and projects have some support for IPv6, but generally incomplete and less well tested.

For my work on a IPv6 enabled Snort_inline I’ve only fixed the checksum issue and added a libnet_build_icmpv6_unreach() function. The patch against libnet 1.1.3-RC-01 can be found here. It’s development was funded by the great people of NitroSecurity Inc., who are funding my work to bring IPv6 to Snort_inline. The work is not based on Sourcefire‘s recent IPv6 implementation, so it will be interesting to see if and how those codebases can be used to improve each other. The changes to Snort_inline will be made available as well later, WhenItsDone(tm) 🙂 Like with the support for NFQueue, NitroSecurity gives back to the community, which I really appreciate!

The patch: http://www.inliniac.net/files/libnet-1.1.3-RC-01-ipv6.diff.gz

Vuurmuur developments

Last weeks I’ve spend many hours on my Vuurmuur Firewall project. First I’ve been improving the code to prepare for a new release. I’ve added NFQUEUE support to Vuurmuur, so I could use it with nfnetlink enabled Snort_inline. Also the connection killing has been improved. The rules limit options were extended, to allow more flexibility.

Second, with the great help of Adi Kriegisch, I’ve been working on setting up a new build server for Debian and Ubuntu packages. Credits mostly go to Adi, who did most of the work and hosts the server. So many thanks to Adi! The new build server supports all version of Debian from Sarge up and of Ubuntu from Dapper and up.

Third, I have been busy setting up a new site for Vuurmuur. It is based on the Trac software and replaces both the static site at sourceforge.net and the tikiwiki based wiki. The new site can be found at http://www.vuurmuur.org/

Last but not least, and the reason for this post, is that today I’ve released a new version of Vuurmuur. The version, 0.5.73, is the first non-test release since January. This has taken a lot longer than I wanted, but work and other projects and of course ‘real life’ have taken much of the time from Vuurmuur. I don’t think this will change any time soon, unless someone will join the project to contribute some serious amount of coding time, or someone wants to fund me to do so.

On the roadmap (which is a nice feature of Trac btw) one can see the future directions of Vuurmuur. Ideas and comments are welcome. So are bug reports of course 🙂

Window scaling normalization in Snort_inline broken by design

After debugging some connection problems I found that the wscale normalization concept is flawed. I’ll describe here what is wrong with it and then move on to suggest a different solution I’m currently testing. The problem I was seeing is that some connections to some webservers stalled without an apparent reason.

First a quick reminder of why I originally came up with the wscale normalization. Stream4 originally doesn’t look at the window scaling value when determining the TCP window. This causes it to be wrong about the TCP window in about every connection, which is one of the reasons out of window packets are not dropped (this is actually a gaping evasion hole since these packets are not used in stream reassembly). This is why I decided to add window scaling support to the stream4inline extension. This works great and allows the admin to drop out of window packets. There is a problem associated with it though. The maximal window that is possible with wscaling is 1GB. This would mean that Snort_inline would in the worst case have to queue almost 1GB of data in it’s buffers for a single stream. To prevent this being used by an attacker to attack Snort_inline, I wanted give the admin the option to set a maximal wscale size.

So, why doesn’t replacing the wscale value in packets work? I’ll explain that now. First an example without normalization. Say we have client connecting to a server. The client sends in it’s SYN packet a window of 5840 and a wscale of 5. The server replies with a SYN/ACK with window 5792, wscale 9. Both have a unscaled window in their packet since the wscale won’t be used before both sides have received a packet with the wscale option enabled. The client sends an ACK completing the three way handshake, with a window of 183. That means a scaled window of 5856 (183 x 2^5). The client will now send an actual data packet, using the same window. The server ACK’s the data with a packet with a window of 16, meaning a scaled window of 8192 (16 x 2^9).

Now, what happens when we normalize? Consider the same connection, but now Snort_inline normalizes all wscale values above 2, to 2. The client sends in it’s SYN packet: window of 5840, wscale of 5, but due to the normalization, the server receives it as window of 5840, wscale of 2. The server replies with a SYN/ACK with window 5792, wscale 9, but the client receives it as window 5792, wscale 2. The problem here is that neither the client or the server know that it’s wscale value was been modified. Nor is there a way to make it known. So what then happens is this. When the server wants to say it has a window of 8192, it will send a packet with the window field set to 16 (16 * 2^9 = 8192). But, due to the normalization, it actually says it has a window of 64 (16 * 2^2). Likewise, when the client wants to tell the server it has a window of 5856 (window field set to 183), it actually says it has a window of 732 (183 * 2^2). This completely stalls connections. So why did I only see this on some rare connections? That is because most servers on the internet use low wscale values. The server I ran into issues with however, used a value of 9.

The solution I am now testing is normalizing the scaled window. With this idea Snort_inline takes the full scaled window into account and compares it with a maximum value. If it exceeds it, the window value in the packet is modified taking the wscale value into account. I’ve been running like this for about 2 weeks now, and so far I have seen no stalling connections anymore. There is however quite a drawback to this approach. The window size is a constantly changing value that is adapted in almost every TCP packet. Unlike the wscale normalization, that could be done by modifying the SYN and SYN/ACK packets, the new approach in the worst case has to modify and replace almost every single packet in a stream. This will take more resources from Snort_inline.

I’m interested in hearing other possible solutions to this problem or other drawbacks of my new solution. I will be checking my new solution into SVN soon. I will make sure it is disabled by default. To work around the broken wscale normalization just set it to it’s maximum value, so add ‘norm_wscale_max 14’ to your stream4 configuration line.

Snort_inline and out of order packets

In Snort_inline’s stream4 modifications, one of the changes is that out of order TCP packets are treated differently from unmodified stream4. This can cause some new alerts to appear and some unexpected behaviour. So I’ll try to explain what happens here.

First of all let me explain quickly what out of order packets are. To put it simple, TCP packets are send out by the source host in a specific order but can arrive in a different order at the destination. Packetloss, link saturation, routing issues are among many things that can cause this. A Snort_inline specific issue is that when Snort_inline can’t keep up with the packets it needs to process, it will drop packets which causes packetloss. These packets will then have to be resent by the sending host.

Out of order packets become a problem when dealing with stream reassembly. Stream reassembly basically is putting all data from the packets in the right order to get the original data as it was sent. We can’t do stream reassembly if we don’t have all packets. Unmodified stream4 basically ignores gaps in the stream. Designed for passive listening for traffic, it has to deal with packetloss differently than Snort_inline.

Next, some definitions of this functionality in Snort_inline. Out-of-order packets: The number of packets that we have in queue that are out of order for a stream. This means they have a higher sequence number than the next in-sequence packet we are expecting. Out-of-order bytes: The number of bytes of the combined data of the out-of-order packets in the stream. Sequence number hole: A gap between two packets, that can be closed by one or more missing packets.

To prevent Snort_inline from using to much memory on bad connections or when an attacker sends lots of out of order packets, Snort_inline can enforce limits to protect itself. Snort_inline can even force a stream to be completely in-order by dropping all packets that are out of order. Sadly, this has a bad effect on the performance of the connections, so you can set certain limits that balance between performance and protection.

When Snort_inline hits these limits, it will (optionally) fire alerts that look like this:

(spp_stream4) TCP out-of-order packets limit reached for stream
(spp_stream4) TCP out-of-order bytes limit reached for stream
(spp_stream4) TCP sequence number holes limit reached for stream

You can disable the alerts by adding the following option to the preprocessor stream4 line: disable_ooo_alerts. The limits themselves can be adjusted by using the following options: max_seq_holes 2, max_ooo_pkts 25, max_ooo_bytes 7000. These are the values I currently use on my home gateway. I got the idea of implementing these limits from this paper by Vern Paxson. However, it seems to me that his suggestion that at max one sequence hole per stream (even per host) was a bit optimistic. Maybe DSL has more packetloss than the university links he studied.

By default Snort_inline uses the settings that were chosen a bit randomly, so they may not fit your usage. Like with the wscaling, please let me know in a comment what values you use!

Compiling Snort_inline with NFQUEUE support on Ubuntu

I needed to setup the right libraries for Snort_inline development on my fresh Ubuntu Feisty installation, so I decided to write down the procedure for those who think compiling Snort_inline from source is hard. 🙂

Make sure you have build-essential package installed. This makes sure you have a compiler and development packages for glibc and other important libraries. I’m installing the libraries from source to get the latest versions because the latest versions are more stable and perform better than the versions included in Feisty. I’m installing them into /usr because some programs like them there best.

Download the library libnfnetlink version 0.0.25 from http://ftp.netfilter.org/pub/libnfnetlink/

$ tar xvfj libnfnetlink-0.0.25.tar.bz2
$ cd libnfnetlink-0.0.25/
$ ./configure --prefix=/usr
$ make
$ sudo make install

Next download libnetfilter_queue version 0.0.13 from http://ftp.netfilter.org/pub/libnetfilter_queue/

$ tar xvfj libnetfilter_queue-0.0.13.tar.bz2
$ cd libnetfilter_queue-0.0.13
$ ./configure --prefix=/usr
$ make
$ sudo make install

Okay, that sets up the Netfilter libraries.

Next is the installation of libdnet. Do not use the dnet packages from Debian and Ubuntu because this is not the same library as we need for Snort_inline.

Download libdnet 1.11 from http://libdnet.sourceforge.net/

$ tar xzvf libdnet-1.11.tar.gz
$ cd libdnet-1.11
$ ./configure --prefix=/usr
$ make
$ sudo make install

Download snort_inline 2.6.1.5 from http://snort-inline.sourceforge.net/

$ tar xvzf snort_inline-2.6.1.5.tar.gz
$ cd snort_inline-2.6.1.5
$ ./configure --enable-nfnetlink
$ make
$ sudo make install

Test if it works (as root):

# iptables -A OUTPUT -p icmp -j NFQUEUE
# iptables -A INPUT -p icmp -j NFQUEUE
# /usr/local/bin/snort_inline -v -Q -H0

Then ping some host and you should see Snort_inline display the packets. If it doesn’t work, it might be that the kernel modules needed for nfqueue are not yet loaded. Using ‘lsmod’ check if ‘nfnetlink’ and ‘nfnetlink_queue’ are loaded. If not, load them manually.

After this, setup Snort_inline as normal.

TCP Window scaling in Snort_inline

The TCP window field in the TCP header is only 16 bits, so the maximum window size it can handle is only 64kb. A long time ago this was enough, but nowadays it isn’t, by far. Luckily, this is something the window scaling option fixes. Window scaling is very common these days. Your pc or laptop probably uses it by default. Snort’s stream4 however, does not support it. This means that when tracking and reassembling streams, Snort for most connections has no idea about what data is in window and which is out of window. To make matters worse, the packets that are in window when using wscaling, but appear out of window when the wscaling is not accounted for, are never used in the reassembly process. This makes Snort evadable.

One of the goals when creating the stream4inline modifications, was to be able to drop on all TCP anomalies stream4 detects. For this support for window scaling was added to Stream4, so Snort_inline would be able to drop out of window packets. There is however a big problem with window scaling. With window scaling the TCP window possibly increases to a maximum of 1GB (with the maximum wscale value of 14). Stream4 would thus theoretically have to queue up to 1GB of packet data, per stream. While this is something that is unlikely to happen during normal connections, it is possible. This could then be used by an attacker to attack Snort_inline itself.

To prevent this, I added an option to stream4inline that allows the administrator to set a maximum allowable wscale setting. Any higher setting will be normalized away. In these cases the packet is modified and the wscale lowered to the maximum that is allowed. The hosts talking to each other then think the other accepts only the lower wscale and accepts that setting. This can however have some unexpected consequences. If the link that Snort_inline deals with is high speed, high latency or both, setting the wscale value to low can result in serious performance degradation. Connections that are (way) slower than usual is how this issue shows. In these cases the wscale value needs to be increased.

The default value of Snort_inline 2.6.1.5 is a wscale of 2, which is quite low but works fine on my home DSL connection. To change the setting add ‘norm_wscale_max 5’ to your stream4 configuration line. This will allow for a wscale of up to 5. The maximum value is 14. I’d be interested in what values people use on what types and speeds of lines, so please let me know! We can use it to suggest values in the docs or to set a less insane default value 🙂

Snort_inline 2.6.1.5 released

Finally, after many months of development and testing, Snort_inline 2.6.1.5 has been released. It’s the first stable release in almost a year and also the first stable release based on Snort 2.6. William sent the announcement:

snort_inline-2.6.1.5 released

List,

I know it has been a long time since we have had a non-beta release,
but what can I say? Victor and I have both been busy in our personal
and professional lives. If you have been running the version of code
in SVN, there are no major updates with this release other than a
memleak fix for stream4inline. I don't think this gets said often
enough, so I would like to thank Sourcefire for all the hard work they
put into snort and the snort rule sets for which I and the rest of the
community greatly benefit.

Regards,

Will

snort_inline-2.6.1.5
http://snort-inline.sourceforge.net/download.html

Differences between snort in inline mode and snort_inline
http://www.inliniac.net/blog/?p=74

Go and get it! 🙂