Memory leak fixed in stream4inline

A few days ago William told me that if he enabled stream4inline on a busy gateway, Snort_inline would consume all memory within hours. The problem went away when disabling stream4inline, so it made sense that the problem would be in there somewhere.

The first suspect was the reassembly cache. The reassembly cache is used to keep a per stream copy of the reassembled packet in memory. While being memory expensive, it greatly speeds up the sliding window stream reassembly process, especially with small packets. The reason for this being the first and primary suspect is that this is the only place where stream4inline code allocates memory. Reviewing the code however, showed no leaks and adding a debug counter to monitor the memory usage also showed that the leak was not in that code.

Next my investigation focused on parts where stream4 behaves differently in stream4inline mode. I initially focused on what happened when stream4 hit it’s memory limit: the memcap. When the configurable memcap is reached, stream4 nukes 5 random sessions. In stream4inline the option to truncate 15 of the sessions was added, where an attempt is made to clear the memory by removing stored packets no longer needed from a stream. If that fails, 5 random sessions are nuked anyway.

Reviewing the truncating of the sessions didn’t show anything obvious to me so I went on to the killing of the sessions. Descending down the code I finally reached the DropSession function, where the memory cleanup for a session is handled. Here it turned out that the DeleteSpd function, used to clear the stored packets in a stream, was not called in stream4inline mode. The reason for this mistake is that with Snort 2.6.1 support for UDP was added to stream4. The merge with the Snort_inline code went wrong because of extra checks added to the DropSession function.

The stupid thing is that when I did the merge, I was already in doubt about it as a comment showed:

/* XXX did I merge this right??? VJ */

Guess I know the answer now: No 😉

Differences between Snort and Snort_inline

Every few weeks the same question comes up: what is the difference between Snort in inline mode and Snort_inline. This makes sense, because the Snort_inline documentation and website fail to explain it. In this post I will try to highlight the main differences. In general I can say that we try to develop Snort_inline as a patchset on top of Snort. Snort_inline is focused at improving the inline part of Snort. Originally of course, Snort’s inline capabilities were developed in the Snort_inline project. With Snort 2.3.0RC1 they were merged into mainline Snort.

Convenience

We did a number of things to make Snort_inline a little more convenient for inline users.

  • inline is enabled by default in ./configure
  • we got rid of libnet 1.0.2a, switched to libdnet 1.1 instead
  • a snort_inline specific manual page was added, as well as some extra docs
  • a example configuration file for inline use is supplied

Added functionality

  • we support Linux’ new queue’ing mechanism called nfqueue. This was contributed by Nitro Security. Nfqueue supports running multiple copies of Snort_inline to take advantage of SMP and reduce risk of denial of service when Snort_inline should crash.
  • stickydrop preprocessor enables you to add options to the rules to block an ipaddress for a configurable amount of time
  • bait-and-switch preprocessor (Linux only) allows you to redirect traffic from a host to a honeypot based on the rules
  • clamav preprocessor is included (you still need to pass –enable-clamav to ./configure)
  • reinject action for FreeBSD: reinjects an accepted packet into the ipfw list at a specific rule number

Improved for inline use

  • reject action can send RST packets to both source and destination
  • stream4 can drop attacks detected in the reassembled stream. It also enforces the TCP window. It implements a number of ideas from Vern Paxson on TCP reassembly, such as a limit on the number of out of order packets and bytes that are accepted in a stream.
  • some fixes for FreeBSD

As the list shows, if you are interested in Snort running inline, using Snort_inline might be a better choice for you!

Snort_inline updated to 2.6.1.4 in SVN

After moving, which went fine, I now finally have some real coding time again. The last week I have been updating and fixing various parts of Snort_inline. The most important change was the update to Snort version 2.6.1.4, which contains security fixes. William also found an issue with the Stream4inline code. The issue was that the memcap that the admin sets to limit the amount of memory used by stream4 wasn’t properly enforced.

Other fixes that are done is that Snort_inline in nfqueue mode now properly honors signals and also no longer needs the libipq library and headers. There are few changes that will be committed soon. One is an issue that clamav can sometimes return an error when parsing malformed file. Until now the spp_clamav preprocessor would issue a FatalError and cause Snort_inline to die. This is obviously not desirable so the patch makes sure that Snort_inline no longer dies and gives the admin an option to either drop or pass traffic that can’t be inspected.

Last but not least there will be a fix to the nfqueue code that appears to solve the ‘stuck packet problem’ we were seeing under heavy load. A number of people are testing my patch currently so if all goes well that will be commited soon as well.

Checking out the latest code is done with the following command:

svn co https://snort-inline.svn.sourceforge.net/svnroot/snort-inline/trunk

Snort_inline and TCP Segmentation Offloading

Since a short while I have a gigabit setup at home. My laptop has a e1000 Intel NIC, my desktop a Broadcom NIC.While playing with Snort_inline and netpipe-tcp, I noticed something odd. I got tcp packets that had the ‘Don’t Fragment’ option set, but were still bigger than the mtu size of the link. Snort_inline read packets of up to 26kb from the queue, and wireshark and tcpdump were seeing the packets as well. This was only for outgoing packets on the e1000 NIC. The receiving pc saw the packets split up in multiple packets that were honoring the mtu size. This got me thinking that some form of offloading must be taking place and indeed this was the case:

# ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on

It can be disabled by using the following command:

# ethtool -K eth0 tso off

The large packets caused problems in my stream4inline modifications of Stream4. The code can’t handle the case where the packet is bigger than the sliding window size. So I have some work to do 😉

New WordPress issue + Snort and ModSecurity rules

I just read about a new issue with WordPress here at SecurityFocus. It’s a potential credential stealing vulnerability, so I quickly created these ModSecurity 2 rules:

SecDefaultAction “log,deny,status:403,phase:2,t:lowercase,t:escapeSeqDecode”
SecRule REQUEST_FILENAME “/wp-login.php$” “chain,msg:’WORDPRESS wp-login.php redirect_to credentials stealing attempt’,severity:2,t:normalisePath”
SecRule ARGS_NAMES “^redirect_to$” “chain”
SecRule ARGS:redirect_to “(ht|f)tps?://”

I can still login to my WordPress install, so it seems that the rule does no harm. Use at your own risk!

Update: I’ve created a Snort rule as well:

alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:”WORDPRESS wp-login.php redirect_to credentials stealing attempt”; flow:to_server,established; uricontent:”/wp-login.php”; nocase; uricontent:”redirect_to”; pcre:”/redirect_to=(ht|f)tps?://iU”; classtype:web-application-attack; sid:4000003; rev:1;)

Update 2: fixed the Snort rule, thanks to Shirkdog for pointing out that it had some broken pcre in it. The rule is now included in the BleedingThreats ruleset (check here), however that (slightly modified) rule doesn’t detect the attack for me.

Update 3: the Bleeding rule is now fixed. I’ve updated the above rule as well.

Update 4: updated the ModSecurity rule to prevent a possible evasion by prepending tab chars to the redirect url. Thanks to Ryan Barnett for pointing this out.

Experimenting with IPv6

My ISP is one of the few here in the Netherlands that provides a IPv6 tunnel broker. I have played with it some during the last year or so, but now decided to get a little more serious with it. So I’ve decided to enable it for my blog. When opening up my site to IPv6 one thing that is important is security. I will describe the status of IPv6 support of my current setup:

Linux firewalling: IPtables supports IPv6 for quite some time, however it only very recently gained stateful packet filtering support. This hasn’t made it into Debian Sarge or even backports yet, so I’m just using stateless filtering now.

Vuurmuur: my own IPtables frontend has no support for IPv6 at all. I’ve been thinking about adding it for years, but decided to wait at least until stateful support would be available. Next to this my coding time is limited, and many other features are probably more interesting to Vuurmuur users.

Snort/Snort_inline: both Snort and Snort_inline lack support for IPv6. Sourcefire is working on it as far as I know, but no code is available from them. I did find a IPv6 patch for Snort 2.3.3, which can be found here. I ran it in sniffer mode and that works. I haven’t played with it much other than that, but I certainly will in the future.

ModSecurity: my Apache 2 installation has IPv6 enabled by default and ModSecurity 2.x just worked with it without any configuration change! I haven’t looked into how to create rules specific for IPv6 addresses however, so maybe surprises will come up here. I do know from looking at the source that the rbl functionality doesn’t support IPv6 addresses yet, but I haven’t even checked if realtime blacklists exist for IPv6.

Sguil/Modsec2sguil: my modsec2sguil script, that takes ModSecurity alerts and feeds them to Sguil, doesn’t act on the IPv6 alerts because it expects IPv4 addresses. This is not a problem however, since Sguil doesn’t support IPv6 addresses. This makes sense since Snort doesn’t support it either.

So compared to my IPv4 access, protection is somewhat limited. I’m only enabling HTTP for now, so ModSecurity should be able to handle that just fine.

Anyway, it seems to be working fine now, but consider the IPv6 support experimental, as I’m playing with how it all works. So don’t be surpised if it’s broken all of a sudden 😉