Suricata 2.0 and beyond

Today I finally released Suricata 2.0. The 2.0 branch opened in December 2012. In the little over a year that it’s development lasted, we have closed 183 tickets. We made 1174 commits, with the following stats:

582 files changed, 94782 insertions(+), 63243 deletions(-)

So, a significant update! In total, 17 different people made commits. I’m really happy with how much code and features were contributed. When starting Suricata this was what I really hoped for, and it seems to be working!

Eve

The feature I’m most excited about is ‘Eve’. It’s the nickname of a new logging output module ‘Extendible Event Format’. It’s an all JSON event stream that is very easy to parse using 3rd party tools. The heavy lifting has been done by Tom Decanio. Combined with Logstash, Elasticsearch and Kibana, this allows for really easy graphical dashboard creation. This is a nice addition to the existing tools which are generally more alert centered.

kibana300 kibana300map kibana-suri

Splunk support is easy as well, as Eric Leblond has shown:

regit-Screenshot-from-2014-03-05-231712

Looking forward

While doing releases is important and somewhat nice too, the developer in me is always glad when they are over. Leading up to a release there is a slow down of development, when most time is spent on fixing release critical bugs and doing some polishing. This slow down is a necessary evil, but I’m glad when we can start merging bigger changes again.

In the short term, I shooting for a fairly quick 2.0.1 release. There are some known issues that will be addressed in that.

More interestingly from a development perspective is the opening of the 2.1 branch. I’ll likely open that in a few weeks. There are a number of features in progress for 2.1. I’m working on speeding up pcap recording, which is currently quite inefficient. More interestingly, Lua output scripting. A preview of this work is available here  with some example scripts here.

Others are working on nice things as well: improving protocol support for detection and logging, nflog and netmap support, taxii/stix integration, extending our TLS support and more.

I’m hoping the 2.1 cycle will be shorter than the last, but we’ll see how it goes :)

Suricata: Handling of multiple different SYN/ACKs

synackWhen processing the TCP 3 way handshake (3whs), Suricata’s TCP stream engine will closely follow the setup of a TCP connection to make sure the rest of the session can be tracked and reassembled properly. Retransmissions of SYN/ACKs are silently accepted, unless they are different somehow. If the SEQ or ACK values are different they are considered wrong and events are set. The stream events rules will match on this.

I ran into some cases where not the initial SYN/ACK was used by the client, but instead a later one. Suricata however, had accepted the initial SYN/ACK. The result was that every packet from that point was rejected by the stream engine. A 67 packet pcap resulting in 64 stream events.

If people have the stream events enabled _and_ pay attention to them, a noisy session like this should certainly get their attention. However, many people disable the stream events, or choose to ignore them, so a better solution is necessary.

Analysis

In this case the curious thing is that the extra SYN/ACK(s) have different properties: the sequence number is different. As the SYN/ACKs sequence number is used as “initial sequence number” (ISN) in the “to client” direction, it’s crucial to track it correctly. Failing to do so, Suricata will loose track of the stream, causing reassembly to fail. This could lead to missed alerts.

Whats happening on the wire:

TCP SSN 1:

-> SYN: SEQ 10
<- SYN/ACK 1: ACK 11, SEQ 100
<- SYN/ACK 2: ACK 11, SEQ 1000
-> ACK: SEQ 11, ACK 101

TCP SSN 2:

-> SYN: SEQ 10
<- SYN/ACK 1: ACK 11, SEQ 100
<- SYN/ACK 2: ACK 11, SEQ 1000
-> ACK: SEQ 11, ACK 1001

It’s clear that in SSN 1 the client ACKs the first SYN/ACK while in SSN 2 the 2nd SYN/ACK is ACK’d. It’s likely that the first SYN/ACK was lost before it reached the client. Suricata accepts the first though, and rejects any others that are not the same.

Solution

The solution I’ve been working on is to delay judgement on the extra SYN/ACKs until Suricata sees the ACK that completes the 3whs. At that point Suricata knows what the client accepted, and which SYN/ACKs were either ignored, or never received.

Logic in pseudo code:

Normal SYN/ACK coming in:

    UpdateState(p);
    ssn->state = TCP_SYN_RECV;

Extra SYN/ACK packets:

    if (p != ssn) {
        QueueState(p);

On receiving the ACK that completes the 3whs:

    if (ssn->queue_len) {
        q = QueueFindState(p);
        if (q)
            UpdateState(q);
    }
    UpdateState(p);
    ssn->state = TCP_ESTABLISHED;

So when receiving the ACK, Suricata first searches for the proper SYN/ACK on the list. If it’s not found, the ACK will be processed normally, which means it’s checked against the original SYN/ACK. If Suricata did have a queued state, it will first apply it to the SSN. Then the ACK will be processed normally, so that is can complete the 3whs and move the state to ESTABLISHED.

Limitations

Queuing these states takes some memory, and for this reason there is a limit to the number each SSN will accept. This is configurable through a new stream option:

stream:
  max-synack-queued: 5

It defaults to 5. I’ve seen a few (valid) hits against a few terrabytes of traffic, so I think the default is reasonably safe. An event is being set if the limit is exceeded. It can be matched using a stream-event rule:

  alert tcp any any -> any any (msg:"SURICATA STREAM 3way handshake \
      excessive different SYN/ACKs"; stream-event:3whs_synack_flood; \
      sid:2210055; rev:1;)

Performance

This functionality doesn’t affect the regular “fast path” except for a small check to see if we have queued states. However, if the queue list is being used Suricata enters a slow path. Currently this involves an memory allocation per stored queue. It may be interesting to consider using pools here, although a single global pool might be ineffecient. In such a case a lock would have to be used and this might lead to contention, especially in a case where Suricata would be flooded. Per thread pools (519, 520, 521) may be best here.

IPS mode

SYN/ACKs that exceed the limit are dropped if stream.inline is enabled as is the case with all packets that are considered to be bad in some way.

Code

The code is now part of the git master through commit 4c6463f3784f533a07679589dab713096137a439. Feedback welcome through our oisf-devel list.

Important Suricata update

We just released Suricata 1.3.3 which contains some important accuracy fixes. Also, it should be much more robust against out of memory conditions.

For those of you running Suricata in IPS mode, this is important as well. We found that rules that have the drop or reject actions, were not playing well with thresholding.

So upgrading is highly recommended!

Code changes are not too big, largest changes are due to some extra unittests:

 ChangeLog                           |   11 +
 libhtp/htp/dslib.c                  |    4 +-
 libhtp/htp/hooks.c                  |   31 +-
 libhtp/htp/htp_connection.c         |   34 ++-
 libhtp/htp/htp_connection_parser.c  |   25 +-
 libhtp/htp/htp_parsers.c            |    2 +-
 libhtp/htp/htp_request.c            |    4 +-
 libhtp/htp/htp_request_apache_2_2.c |   24 +-
 libhtp/htp/htp_transaction.c        |   68 +++--
 libhtp/htp/htp_util.c               |   35 ++-
 src/alert-debuglog.c                |    4 +-
 src/app-layer.c                     |    9 +-
 src/decode.h                        |    3 +-
 src/detect-detection-filter.c       |   96 ++++++
 src/detect-engine-alert.c           |   37 ++-
 src/detect-engine-hcbd.c            |    5 +
 src/detect-engine-hhd.c             |  121 +++++++-
 src/detect-engine-hsbd.c            |    5 +
 src/detect-engine-iponly.c          |    5 +-
 src/detect-engine-payload.c         |   26 ++
 src/detect-engine-threshold.c       |   15 +-
 src/detect-filemd5.c                |   24 +-
 src/detect-filestore.c              |   11 +-
 src/detect-filestore.h              |    2 +-
 src/detect-pcre.c                   |  485 +----------------------------
 src/detect-threshold.c              |  569 ++++++++++++++++++++++++++++++++++-
 src/detect.c                        |   11 +-
 src/detect.h                        |    2 +-
 src/flow-hash.c                     |   10 +-
 src/flow-timeout.c                  |   10 +-
 src/flow.c                          |    1 -
 src/flow.h                          |   14 +
 src/log-httplog.c                   |    2 +-
 src/runmodes.c                      |    2 +-
 src/source-ipfw.c                   |    1 +
 src/source-pfring.c                 |   20 +-
 src/stream-tcp-reassemble.c         |    4 +-
 src/stream-tcp.c                    |   12 +-
 src/stream.c                        |    3 +-
 src/threads.h                       |    1 +
 src/tmqh-packetpool.c               |    5 +-
 src/util-buffer.h                   |    6 +-
 src/util-debug.c                    |    2 +-
 src/util-host-os-info.c             |   32 +-
 src/util-threshold-config.c         |  210 +++++++++++++
 suricata.yaml.in                    |    6 +-
 46 files changed, 1340 insertions(+), 669 deletions(-)

Setting up an IPS with Fedora 17, Suricata and Vuurmuur

I recently found out that Fedora includes Vuurmuur in it’s repositories. Since Suricata is also included, I figured I would do a quick write up on how to setup a Fedora IPS. While writing it turned more into a real “howto”, so I decided to submit it to Howtoforge.

It can be found here one HowtoForge.

Vuurmuur on Fedora is at the 0.7 version, which is still the current stable. It’s rather old though, and it reminds me again I need to make sure the 0.8 branch gets to a stable release soon. The Suricata included in Fedora 17 is 1.2.1, with 1.3.2 expected to land any day now.

The guide sets the user up from base Fedora install to a working IPS, but doesn’t cover any advanced topics such as rule management, event management etc. Still, I hope it’s useful to some, especially those that are intimidated by Vuurmuur’s and Suricata’s initial learning curves.

Looking forward to feedback! :)

F-Secure AV updates and Suricata IPS

My ISP recently started providing 3 F-Secure AV copies to each of their customers. I installed it but noticed that updates timed out.

It turned out that Suricata, which runs in IPS mode, blocked the update. There were 3 Emerging Threats rules that alerted:

[1:2003614:4] ET VIRUS WinUpack Modified PE Header Inbound
[1:2009557:2] ET TROJAN Yoda’s Protector Packed Binary
[1:2012086:2] ET SHELLCODE Possible Call with No Offset TCP Shellcode

It seems that F-Secure uses some form of packed binaries for their updates that is often used by malware.

To allow the updates to go through without disabling the rules altogether, we can use suppressions. All the alerts happened in streams talking to IP addresses in the 217.110.97.14x range. Whois lookup suggested that F-Secure has 217.110.97.128/25 available, so I decided to suppress the rules for that entire block.

To add the suppressions, I added the following lines to my threshold.conf:

# f-secure update matching
suppress gen_id 1, sig_id 2009557, track by_src, ip 217.110.97.128/25
suppress gen_id 1, sig_id 2012086, track by_src, ip 217.110.97.128/25
suppress gen_id 1, sig_id 2003614, track by_src, ip 217.110.97.128/25

After a Suricata restart, the updates now work fine. If you run Suricata in IDS mode you may still want to add the suppressions to reduce the number of alerts.

Suricata IPS improvements

January has been a productive month for Suricata, especially for the IPS part of it. I’ve quite some time on adding support to the stream engine to operate differently when running inline. This was needed as dropping attacks found in the reassembled stream or the application layer was not reliable. Up until now the stream engine would offer the reassembled stream to the detection engine as soon as it was ACK’d. This meant that by definition the packets containing the data had already passed the IPS device. Simply switching to sending un-ACK’d data to the detection engine would have it’s own set of issues.

To be able to work with un-ACK’d data, we need to make sure we deal with possible evasions properly. The problem, as extensively documented by Judy Novak and Steven Sturges, is that in TCP streams there can be overlapping packets. Those are being dealt with differently based on the receiving OS. If we would need to account for overlaps in the application layer, we would have to be able to tell the HTTP parser for example: “sorry, that last data is wrong, please revert and use the new packet instead”. A nightmare.

The solution I opted for was to not care about destination OS’ for overlaps and such. The approach is fairly simple: once we have accepted a segment, thats what it’s going to be. This means that if we receive a segment later that (partially) overlaps and has different data, it’s data portion will simply be overwritten to be the same as the original segment. This way, the IPS and not an obscure mix of the sender (attacker?) and destination OS, determines the data the destination will see.

Of course the approach comes with some drawbacks. First, we need to keep segments in memory for a longer period of time. This causes significantly higher memory usage. Secondly, if we rewrite a packet, it needs to be reinjected on the wire. As we modified the packet payload a checksum recalculation is required.

In Suricata’s design the application layer parsers, such as our HTTP parser, run on top of the reassembly engine. After the reassembly engine and the app layer parsers are updated, the packet with the associated stream and app layer state is passed on to the detection engine. In the case where we work with ACK’d data, an ACK packet in the opposite direction triggers the reassembly process. If we detect based on that, and decide we need to drop, all we can do is drop the ACK packet as the actual data segment(s) have already passed. This is not good enough in many cases.

In the new code the data segment itself triggers the reassembly process. In this case, if the detection engine decides a drop is required, the packet containing the data itself can be dropped, not just the ACK. The reason we’re not taking the same approach in IDS mode is that we wouldn’t be able to properly deal with the said evasion/overlap issues. The IPS can exactly control what packets pass Suricata. The IDS, being passive, can not.

You can try this code by checking out the current git master. In the suricata.yaml that lives in our git tree you’ll find a new option in the stream config, “stream.inline”. If you enable this, the code as explained above is activated.

Feedback is very welcome!

Suricata 1.0.2 released

After some well deserved vacation I’m getting back up to speed in Suricata development. Luckily most of our dev team continued to work in my absence, making today’s 1.0.2 release possible.

The main focus of this release was fixing the TCP stream engine. Judy Novak found a number of ways to evade detection. See her blog post describing the issues.

The biggest other change is the addition of a new application layer module. The SSH parser parses SSH sessions and stops detection/inspection of the stream after the encrypted part of the session has started. So this is mainly a module focused on reducing the number of packets that need inspection, just like the SSL and TLS modules.

As a bonus though, we introduced two rule keywords that match on the parsed SSH parameters:

ssh.protoversion will match against the ssh protocol version. I’ll give some examples.

ssh.protoversion:2.0

This will match on 2.0 exactly.

ssh.protoversion:2_compat

This will match on 2, but also 1.99 and other versions compatible to “2”.

ssh.protoversion:1.

The last example will match on all versions starting with “1.”, so 1.6, 1.7, etc.

ssh.softwareversion will match on the software version identifier. An example:

ssh.softwareversion:PuTTY

This will match only on session using the PuTTY SSH client.

Other changes include better HTTP accuracy, better IPS functionality.

For the next release we will focus on further improving overall detection accuracy, improving inline mode further, improving performance and specifically improving CUDA performance. As always, we welcome any feedback. Or if you are interested in helping out, please contact us!

Update: added a link to Judy Novak’s blog post on the TCP evasions.

Setting up Suricata 0.9.0 for initial use on Ubuntu Lucid 10.04

The last few days I blogged about compiling Suricata in IDS and IPS mode. Today I’ll write about how to set it up for first use.

Starting with Suricata 0.9.0 the engine can run as an unprivileged user. For this create a new user called “suricata”.

useradd --no-create-home --shell /bin/false --user-group --comment “Suricata IDP account” suricata

This command will create a user and group called “suricata”. It will be unable to login as the shell is set to /bin/false.

The next thing to do is creating a configuration directory. Create /etc/suricata/ and copy the suricata.yaml example config into it. The example configuration can be found in the source archive you used to build Suricata:

mkdir /etc/suricata
cp /path/to/suricata-0.9.0/suricata.yaml /etc/suricata/
cp /path/to/suricata-0.9.0/classification.config /etc/suricata/

Next, create the log directory.

mkdir /var/log/suricata

The log directory needs to be writable for the user and group “suricata”, so change the ownership:

chown suricata:suricata /var/log/suricata

The last step I’ll be describing here is retrieving an initial ruleset. The 2 main rulesets you can use are Emerging Threats (ET) and Sourcefire’s VRT ruleset. Since putting VRT to use is a little bit more complicated I’ll be focussing on ET here.

First, download the emerging rules:

wget http://www.emergingthreats.net/rules/emerging.rules.tar.gz

Go to /etc/suricata/ and extract the rules archive:

cd /etc/suricata/
tar xzvf /path/to/emerging.rules.tar.gz

There is a lot more to rules, such as tuning and staying updated, but thats beyond the scope of this post.

Suricata is now ready to be started:

suricata -c /etc/suricata/suricata.yaml -i eth0 --user suricata --group suricata

If all is setup properly, Suricata will tell you it is now running:

[2087] 9/5/2010 — 18:17:47 – (tm-threads.c:1362) (TmThreadWaitOnThreadInit) — all 8 packet processing threads, 3 management threads initialized, engine started.

There are 3 log files in /var/log/suricata that will be interesting to monitor:

- stats.log: displays statistics on packets, tcp sessions etc.
– fast.log: a alerts log similar to Snort’s fast log.
– http.log: displays HTTP requests in a Apache style format.

This should get you going. There is a lot more to deploying Suricata that I plan to blog on later.

Compiling Suricata 0.9.0 in Ubuntu Lucid 10.04 in IPS (inline) mode

Note: the difference with the 0.8.2 post is that addition of libcap-ng-dev. This allows Suricata to run as an unprivileged user.

Here is how to compile Suricata 0.9.0 in inline mode on Ubuntu Lucid 10.04.

First, make sure you have the “universe” repository enabled. Go to the System menu, Administration, Software Sources. There enable “Community-maintained Open Source Software (universe)”. If you’re not running a gui, edit /etc/apt/sources.list and enable the universe repository there. Don’t forget doing an “apt-get update”.

Install the following packages needed to build Suricata: libpcre3-dev libpcap-dev libyaml-dev zlib1g-dev libnfnetlink-dev libnetfilter-queue-dev libnet1-dev libcap-ng-dev.

apt-get install libpcre3-dev libpcap-dev libyaml-dev zlib1g-dev libnfnetlink-dev libnetfilter-queue-dev libnet1-dev libcap-ng-dev

Download Suricata 0.9.0 here

Extract the suricata-0.9.0.tar.gz file as follows:

tar xzvf suricata-0.9.0.tar.gz

Enter the extracted directory suricata-0.9.0.

Run “./configure –enable-nfqueue”
If “./configure –enable-nfqueue” was succesful, run “make”
If “make” was succesful, run “sudo make install”
Except for Suricata itself, the build process installed “libhtp”. For that to work properly, run “ldconfig”.

Run “suricata -V” and it should report version 0.9.0.

To use Suricata in inline mode, pass -q <queue id> to the command line. Example:

suricata -c /etc/suricata/suricata.yaml -q 0

Suricata 0.9.0 released

Yesterday we released we first release candidate for our upcoming 1.0 release of Suricata. See the announcement on the OISF site here.

Most notable changes are the following new features:

- Support for the http_headers keyword was added
- libhtp was updated to version 0.2.3
- Privilege dropping using libcap-ng is now supported
- Proper support for “pass” rules was added
- Inline mode for Windows was added

Go get the release here: http://www.openinfosecfoundation.org/download/suricata-0.9.0.tar.gz