Suricata 2.0 and beyond

Today I finally released Suricata 2.0. The 2.0 branch opened in December 2012. In the little over a year that it’s development lasted, we have closed 183 tickets. We made 1174 commits, with the following stats:

582 files changed, 94782 insertions(+), 63243 deletions(-)

So, a significant update! In total, 17 different people made commits. I’m really happy with how much code and features were contributed. When starting Suricata this was what I really hoped for, and it seems to be working!

Eve

The feature I’m most excited about is ‘Eve’. It’s the nickname of a new logging output module ‘Extendible Event Format’. It’s an all JSON event stream that is very easy to parse using 3rd party tools. The heavy lifting has been done by Tom Decanio. Combined with Logstash, Elasticsearch and Kibana, this allows for really easy graphical dashboard creation. This is a nice addition to the existing tools which are generally more alert centered.

kibana300 kibana300map kibana-suri

Splunk support is easy as well, as Eric Leblond has shown:

regit-Screenshot-from-2014-03-05-231712

Looking forward

While doing releases is important and somewhat nice too, the developer in me is always glad when they are over. Leading up to a release there is a slow down of development, when most time is spent on fixing release critical bugs and doing some polishing. This slow down is a necessary evil, but I’m glad when we can start merging bigger changes again.

In the short term, I shooting for a fairly quick 2.0.1 release. There are some known issues that will be addressed in that.

More interestingly from a development perspective is the opening of the 2.1 branch. I’ll likely open that in a few weeks. There are a number of features in progress for 2.1. I’m working on speeding up pcap recording, which is currently quite inefficient. More interestingly, Lua output scripting. A preview of this work is available here  with some example scripts here.

Others are working on nice things as well: improving protocol support for detection and logging, nflog and netmap support, taxii/stix integration, extending our TLS support and more.

I’m hoping the 2.1 cycle will be shorter than the last, but we’ll see how it goes :)

Suricata: Handling of multiple different SYN/ACKs

synackWhen processing the TCP 3 way handshake (3whs), Suricata’s TCP stream engine will closely follow the setup of a TCP connection to make sure the rest of the session can be tracked and reassembled properly. Retransmissions of SYN/ACKs are silently accepted, unless they are different somehow. If the SEQ or ACK values are different they are considered wrong and events are set. The stream events rules will match on this.

I ran into some cases where not the initial SYN/ACK was used by the client, but instead a later one. Suricata however, had accepted the initial SYN/ACK. The result was that every packet from that point was rejected by the stream engine. A 67 packet pcap resulting in 64 stream events.

If people have the stream events enabled _and_ pay attention to them, a noisy session like this should certainly get their attention. However, many people disable the stream events, or choose to ignore them, so a better solution is necessary.

Analysis

In this case the curious thing is that the extra SYN/ACK(s) have different properties: the sequence number is different. As the SYN/ACKs sequence number is used as “initial sequence number” (ISN) in the “to client” direction, it’s crucial to track it correctly. Failing to do so, Suricata will loose track of the stream, causing reassembly to fail. This could lead to missed alerts.

Whats happening on the wire:

TCP SSN 1:

-> SYN: SEQ 10
<- SYN/ACK 1: ACK 11, SEQ 100
<- SYN/ACK 2: ACK 11, SEQ 1000
-> ACK: SEQ 11, ACK 101

TCP SSN 2:

-> SYN: SEQ 10
<- SYN/ACK 1: ACK 11, SEQ 100
<- SYN/ACK 2: ACK 11, SEQ 1000
-> ACK: SEQ 11, ACK 1001

It’s clear that in SSN 1 the client ACKs the first SYN/ACK while in SSN 2 the 2nd SYN/ACK is ACK’d. It’s likely that the first SYN/ACK was lost before it reached the client. Suricata accepts the first though, and rejects any others that are not the same.

Solution

The solution I’ve been working on is to delay judgement on the extra SYN/ACKs until Suricata sees the ACK that completes the 3whs. At that point Suricata knows what the client accepted, and which SYN/ACKs were either ignored, or never received.

Logic in pseudo code:

Normal SYN/ACK coming in:

    UpdateState(p);
    ssn->state = TCP_SYN_RECV;

Extra SYN/ACK packets:

    if (p != ssn) {
        QueueState(p);

On receiving the ACK that completes the 3whs:

    if (ssn->queue_len) {
        q = QueueFindState(p);
        if (q)
            UpdateState(q);
    }
    UpdateState(p);
    ssn->state = TCP_ESTABLISHED;

So when receiving the ACK, Suricata first searches for the proper SYN/ACK on the list. If it’s not found, the ACK will be processed normally, which means it’s checked against the original SYN/ACK. If Suricata did have a queued state, it will first apply it to the SSN. Then the ACK will be processed normally, so that is can complete the 3whs and move the state to ESTABLISHED.

Limitations

Queuing these states takes some memory, and for this reason there is a limit to the number each SSN will accept. This is configurable through a new stream option:

stream:
  max-synack-queued: 5

It defaults to 5. I’ve seen a few (valid) hits against a few terrabytes of traffic, so I think the default is reasonably safe. An event is being set if the limit is exceeded. It can be matched using a stream-event rule:

  alert tcp any any -> any any (msg:"SURICATA STREAM 3way handshake \
      excessive different SYN/ACKs"; stream-event:3whs_synack_flood; \
      sid:2210055; rev:1;)

Performance

This functionality doesn’t affect the regular “fast path” except for a small check to see if we have queued states. However, if the queue list is being used Suricata enters a slow path. Currently this involves an memory allocation per stored queue. It may be interesting to consider using pools here, although a single global pool might be ineffecient. In such a case a lock would have to be used and this might lead to contention, especially in a case where Suricata would be flooded. Per thread pools (519, 520, 521) may be best here.

IPS mode

SYN/ACKs that exceed the limit are dropped if stream.inline is enabled as is the case with all packets that are considered to be bad in some way.

Code

The code is now part of the git master through commit 4c6463f3784f533a07679589dab713096137a439. Feedback welcome through our oisf-devel list.

Important Suricata update

We just released Suricata 1.3.3 which contains some important accuracy fixes. Also, it should be much more robust against out of memory conditions.

For those of you running Suricata in IPS mode, this is important as well. We found that rules that have the drop or reject actions, were not playing well with thresholding.

So upgrading is highly recommended!

Code changes are not too big, largest changes are due to some extra unittests:

 ChangeLog                           |   11 +
 libhtp/htp/dslib.c                  |    4 +-
 libhtp/htp/hooks.c                  |   31 +-
 libhtp/htp/htp_connection.c         |   34 ++-
 libhtp/htp/htp_connection_parser.c  |   25 +-
 libhtp/htp/htp_parsers.c            |    2 +-
 libhtp/htp/htp_request.c            |    4 +-
 libhtp/htp/htp_request_apache_2_2.c |   24 +-
 libhtp/htp/htp_transaction.c        |   68 +++--
 libhtp/htp/htp_util.c               |   35 ++-
 src/alert-debuglog.c                |    4 +-
 src/app-layer.c                     |    9 +-
 src/decode.h                        |    3 +-
 src/detect-detection-filter.c       |   96 ++++++
 src/detect-engine-alert.c           |   37 ++-
 src/detect-engine-hcbd.c            |    5 +
 src/detect-engine-hhd.c             |  121 +++++++-
 src/detect-engine-hsbd.c            |    5 +
 src/detect-engine-iponly.c          |    5 +-
 src/detect-engine-payload.c         |   26 ++
 src/detect-engine-threshold.c       |   15 +-
 src/detect-filemd5.c                |   24 +-
 src/detect-filestore.c              |   11 +-
 src/detect-filestore.h              |    2 +-
 src/detect-pcre.c                   |  485 +----------------------------
 src/detect-threshold.c              |  569 ++++++++++++++++++++++++++++++++++-
 src/detect.c                        |   11 +-
 src/detect.h                        |    2 +-
 src/flow-hash.c                     |   10 +-
 src/flow-timeout.c                  |   10 +-
 src/flow.c                          |    1 -
 src/flow.h                          |   14 +
 src/log-httplog.c                   |    2 +-
 src/runmodes.c                      |    2 +-
 src/source-ipfw.c                   |    1 +
 src/source-pfring.c                 |   20 +-
 src/stream-tcp-reassemble.c         |    4 +-
 src/stream-tcp.c                    |   12 +-
 src/stream.c                        |    3 +-
 src/threads.h                       |    1 +
 src/tmqh-packetpool.c               |    5 +-
 src/util-buffer.h                   |    6 +-
 src/util-debug.c                    |    2 +-
 src/util-host-os-info.c             |   32 +-
 src/util-threshold-config.c         |  210 +++++++++++++
 suricata.yaml.in                    |    6 +-
 46 files changed, 1340 insertions(+), 669 deletions(-)

Setting up Suricata 0.9.0 for initial use on Ubuntu Lucid 10.04

The last few days I blogged about compiling Suricata in IDS and IPS mode. Today I’ll write about how to set it up for first use.

Starting with Suricata 0.9.0 the engine can run as an unprivileged user. For this create a new user called “suricata”.

useradd --no-create-home --shell /bin/false --user-group --comment “Suricata IDP account” suricata

This command will create a user and group called “suricata”. It will be unable to login as the shell is set to /bin/false.

The next thing to do is creating a configuration directory. Create /etc/suricata/ and copy the suricata.yaml example config into it. The example configuration can be found in the source archive you used to build Suricata:

mkdir /etc/suricata
cp /path/to/suricata-0.9.0/suricata.yaml /etc/suricata/
cp /path/to/suricata-0.9.0/classification.config /etc/suricata/

Next, create the log directory.

mkdir /var/log/suricata

The log directory needs to be writable for the user and group “suricata”, so change the ownership:

chown suricata:suricata /var/log/suricata

The last step I’ll be describing here is retrieving an initial ruleset. The 2 main rulesets you can use are Emerging Threats (ET) and Sourcefire’s VRT ruleset. Since putting VRT to use is a little bit more complicated I’ll be focussing on ET here.

First, download the emerging rules:

wget http://www.emergingthreats.net/rules/emerging.rules.tar.gz

Go to /etc/suricata/ and extract the rules archive:

cd /etc/suricata/
tar xzvf /path/to/emerging.rules.tar.gz

There is a lot more to rules, such as tuning and staying updated, but thats beyond the scope of this post.

Suricata is now ready to be started:

suricata -c /etc/suricata/suricata.yaml -i eth0 --user suricata --group suricata

If all is setup properly, Suricata will tell you it is now running:

[2087] 9/5/2010 — 18:17:47 – (tm-threads.c:1362) (TmThreadWaitOnThreadInit) — all 8 packet processing threads, 3 management threads initialized, engine started.

There are 3 log files in /var/log/suricata that will be interesting to monitor:

- stats.log: displays statistics on packets, tcp sessions etc.
- fast.log: a alerts log similar to Snort’s fast log.
- http.log: displays HTTP requests in a Apache style format.

This should get you going. There is a lot more to deploying Suricata that I plan to blog on later.

Compiling Suricata 0.9.0 in Ubuntu Lucid 10.04 in IDS mode

Note: the difference with the 0.8.2 post is that addition of libcap-ng-dev. This allows Suricata to run as an unprivileged user.

Ubuntu Lucid 10.04 LTS looks like a good platform for running an IDS on. It’s up to date and has long term support. Here is how to compile and install Suricata 0.9.0 on it.

Install the following packages needed to build Suricata: libpcre3-dev libpcap-dev libyaml-dev zlib1g-dev libcap-ng-dev.

apt-get install libpcre3-dev libpcap-dev libyaml-dev zlib1g-dev libcap-ng-dev

Download Suricata 0.9.0 here

Extract the suricata-0.9.0.tar.gz file as follows:

tar xzvf suricata-0.9.0.tar.gz

Enter the extracted directory suricata-0.9.0.

Run “./configure”
Note that you may get a warning about libnet 1.1 that is missing. You can ignore that, it’s only used in IPS/inline mode currently.
If “./configure” was succesful, run “make”
If “make” was succesful, run “sudo make install”
Except for Suricata itself, the build process installed “libhtp”. For that to work properly, run “ldconfig”.

Run “suricata -V” and it should report version 0.9.0.

To use Suricata in IDS mode, pass -i to the command line. Example

suricata -c /etc/suricata/suricata.yaml -i eth0

Suricata 0.9.0 released

Yesterday we released we first release candidate for our upcoming 1.0 release of Suricata. See the announcement on the OISF site here.

Most notable changes are the following new features:

- Support for the http_headers keyword was added
- libhtp was updated to version 0.2.3
- Privilege dropping using libcap-ng is now supported
- Proper support for “pass” rules was added
- Inline mode for Windows was added

Go get the release here: http://www.openinfosecfoundation.org/download/suricata-0.9.0.tar.gz

Compiling Suricata 0.8.2 in Ubuntu Lucid 10.04 in IDS mode

The newly released Ubuntu Lucid 10.04 LTS looks like a good platform for running an IDS on. It’s up to date and has long term support. Here is how to compile and install Suricata 0.8.2 on it.

Install the following packages needed to build Suricata: libpcre3-dev libpcap-dev libyaml-dev zlib1g-dev.

apt-get install libpcre3-dev libpcap-dev libyaml-dev zlib1g-dev

Download Suricata 0.8.2 here

Extract the suricata-0.8.2.tar.gz file as follows:

tar xzvf suricata-0.8.2.tar.gz

Enter the extracted directory suricata-0.8.2.

Run “./configure”
Note that you may get a warning about libnet 1.1 that is missing. You can ignore that, it’s only used in IPS/inline mode currently.
If “./configure” was succesful, run “make”
If “make” was succesful, run “sudo make install”
Except for Suricata itself, the build process installed “libhtp”. For that to work properly, run “ldconfig”.

Run “suricata -V” and it should report version 0.8.2.

To use Suricata in IDS mode, pass -i to the command line. Example

suricata -c /etc/suricata/suricata.yaml -i eth0

Suricata 0.8.2 released

Today the OISF development team released 0.8.2 of the Suricata IDS/IPS engine. I feel this is definitely the best release so far. Read the announcement here. In short, stability was improved, memory footprint reduced, performance improved and new features were added.

One of the tools we used to help improve the engine is a fuzzer created by Will Metcalf, our QA lead. In short, the script takes a pcap file, runs it through editcap (part of wireshark) altering a number of random bytes, then feeds the altered pcap file to Suricata. This resulted in many interesting corner cases. Naturally the script makes sure you don’t forget to enable “ulimit -c unlimited” and such :) More on that script can be found on Will’s blog node5.

For the next period we’ll be working on resolving a number of open issues. There are still a number of improvements we need to make to the relation between our app layer decoding modules and our detection engine. Next to this we’re still missing support for a number of rule keywords, such as asn1 and http_headers. We’re also working on getting our CUDA accelaration into a more usable shape. This release improved it slightly, by making it work on x86_64, but it’s still not useful in production environments.

So as usual, enough to do! Meanwhile, we’re looking for feedback on our release!