Crossing the Streams in Suricata

At it’s core, Suricata is a packet processor. It reads packets and pushes them through a configurable pipeline. The 2nd most important processing unit in Suricata is the flow. In Suricata we use the term flow for the bidirectional flows of packets with the same 5 tuple (proto, src ip, dst ip, sp, dp. Vlans can be added as well). In fact, much of Suricata’s threading effort revolves around the flow. In the 2 main runmodes, autofp and workers, flow based load balancing makes sure that a all packets of a single flow always go through the same threading pipeline. In workers this means one single thread, in autofp 2: the capture thread and a stream/detect/output thread.

Flows are the central unit for out ‘app layer’ parsing. Protocol parsers like HTTP don’t even have access to the original packet. It all runs on top of the stream engine, which tracks TCP flows in … our flow structure.

Another place where the flow is crucial is in many of the rules. Rules extensively use the concept of ‘flowbits’. This allows one rule to ‘flag’ a flow, and then another to check this flag. In Emerging Threats many hundreds of rules use this logic.

Ever since we started Suricata, we’ve been talking about what some called ‘global flowbits’. A bit of a strange and contradictory name, but pretty much rule writers wanted the logic of flowbits, but then applied to other units as well. So a few weeks ago I (finally) decided to check if I could quickly implement ‘hostbits’. As Suricata already has a scalable ‘host table’, it was easy add the storage of ‘bits’ there. In a few hours I had the basics working and made it public: see this pull request.

Although I got some nice feedback, I was mostly interested in what the ET folks would think, since they would be the main consumers. While presenting the work I also mentioned the xbits ideas by Michael Rash and the response was “wow, do we have ip_pair tracking now?”. Ehh, no, just ip/host based… “Ah well, I guess that is nice too”. Not exactly the response I hoped for :)

IP pair tracking is not something Suricata already did. But as the need was clear I decided to have a look at it. Turned out it was quite simple to do. The IPPair tracker is much like the Host tracking. It’s only done on demand, which sets it apart from the Flow tracking which is done unconditionally. In this case only the new keyword is making use of the IP Pair storage.

So, what I have implemented is pretty much ‘xbits’. It supports tracking by ‘ip_src’, ‘ip_dst’ and ‘ip_pair’. It uses the syntax as suggested by Michael Rash:

xbits:<set|unset|isset|isnotset|toggle>,<bitname>,\
      track <ip_src|ip_dst|ip_pair>,expire <seconds>

It’s only lightly tested, so I would appreciate testing feedback!

You’ll find the code here in PR 1275 at github. This should normally end up in Suricata 2.1, which will come out early next year.

Suricata Training Tour

After a lot of preparations, it’s finally going to happen: official Suricata trainings!

In the next couple of months I’ll be doing at least 3 sessions: a home match (Amsterdam), a workshop in Luxembourg and a session at DeepSec. Next to this, we’re planning various US based sessions on the East coast and West coast.

I’m really looking forward to doing these sessions. Other than the official content, there will be plenty of room for questions and discussions.

Hope to see you soon! :)

Suricata Flow Logging

Pretty much from the start of the project, Suricata has been able to track flows. In Suricata the term ‘flow’ means the bidirectional flow of packets with the same 5 tuple. Or 7 tuple when vlan tags are counted as well.

Such a flow is created when the first packet comes in and is stored in the flow hash. Each new packet does a hash look-up and attaches the flow to the packet. Through the packet’s flow reference we can access all that is stored in the flow: TCP session, flowbits, app layer state data, protocol info, etc.

When a flow hasn’t seen any packets in a while, a separate thread times it out. This ‘Flow Manager’ thread constantly walks the hash table and looks for flows that are timed out. The time a flow is considered ‘active’ depends on the protocol, it’s state and the configuration settings.

In Suricata 2.1, flows will optionally be logged when they time out. This logging is available through a new API, with an implementation for ‘Eve’ JSON output already developed. Actually, 2 implementations:

  1. flow — logs bidirectional records
  2. netflow — logs unidirectional records

As the flow logging had to be done at flow timeout, the Flow Manager had to drive it. Suricata 2.0 and earlier had a single Flow Manager thread. This was hard coded, and in some cases it was clearly a bottleneck. It wasn’t uncommon to see this thread using more CPU than the packet workers.

So adding more tasks to the Flow Manager, especially something as expensive as output, was likely going to make things worse. To address this, 2 things are now done:

  1. multiple flow manager support
  2. offloading of part of the flow managers tasks to a new class of management threads

The multiple flow managers simply divide up the hash table. Each thread manages it’s own part of it. The new class of threads is called ‘Flow Recycler’. It takes care of the actual flow cleanup and recycling. This means it’s taking over a part of the old Flow Manager’s tasks. In addition, if enabled, these threads are tasked with performing the actual flow logging.

As the flow logging follows the ‘eve’ format, passing it into Elasticsearch, Logstash and Kibana (ELK) is trivial. If you already run such a setup, the only thing that is need is enabling the feature in your suricata.yaml.

kibana-flow

kibana-netflowThe black netflow dashboard is available here: http://www.inliniac.net/files/NetFlow.json

Many thanks to the FireEye Forensics Group (formerly nPulse Technologies) for funding this work.

Detecting OpenSSL Heartbleed with Suricata

The OpenSSL heartbleed vulnerability is a pretty serious weakness in OpenSSL that can lead to information disclosure, in some cases even to to private key leaking. Please see this post here http://blog.existentialize.com/diagnosis-of-the-openssl-heartbleed-bug.html for more info.

This is a case where an IDS is able to detect the vuln, even though we’re talking about TLS.

LUA

I’ve written a quick and dirty LUA script to detect it:

alert tls any any -> any any ( \
    msg:"TLS HEARTBLEED malformed heartbeat record"; \
    flow:established,to_server; dsize:>7; \
    content:"|18 03|"; depth:2; lua:tls-heartbleed.lua; \
    classtype:misc-attack; sid:3000001; rev:1;)

The script:

function init (args)
    local needs = {}
    needs["payload"] = tostring(true)
    return needs
end

function match(args)
    local p = args['payload']
    if p == nil then
        --print ("no payload")
        return 0
    end
 
    if #p < 8 then
        --print ("payload too small")
    end
    if (p:byte(1) ~= 24) then
        --print ("not a heartbeat")
        return 0
    end
 
    -- message length
    len = 256 * p:byte(4) + p:byte(5)
    --print (len)
 
    -- heartbeat length
    hb_len = 256 * p:byte(7) + p:byte(8)

    -- 1+2+16
    if (1+2+16) >= len  then
        print ("invalid length heartbeat")
        return 1
    end

    -- 1 + 2 + payload + 16
    if (1 + 2 + hb_len + 16) > len then
        print ("heartbleed attack detected: " .. (1 + 2 + hb_len + 16) .. " > " .. len)
        return 1
    end
    --print ("no problems")
    return 0
end
return 0

Regular rules

Inspired by the FOX-IT rules from http://blog.fox-it.com/2014/04/08/openssl-heartbleed-bug-live-blog/, here are some non-LUA rules:

Detect a large response.

alert tls any any -> any any ( \
    msg:"TLS HEARTBLEED heartbeat suspiciuous large record"; \
    flow:established,to_client; dsize:>7; \
    content:"|18 03|"; depth:2; \
    byte_test:2,>,200,3,big; classtype:misc-attack; \
    sid:3000002; rev:1;)

Detect a large response following a large request (flow bit is either set by the LUA rule above or by the rule that follows):

alert tls any any -> any any ( \
    msg:"TLS HEARTBLEED heartbeat attack likely succesful"; \
    flowbits:isset,TLS.heartbleed; \
    flow:established,to_client; dsize:>7; \
    content:"|18 03|"; depth:2; byte_test:2,>,200,3,big; \
    classtype:misc-attack; \
    sid:3000003; rev:1;)

Detect a large request, set flowbit:

alert tls any any -> any any ( \
    msg:"TLS HEARTBLEED heartbeat suspiciuous large request"; \
    flow:established,to_server; content:"|18 03|"; depth:2; \
    content:"|01|"; distance:3; within:1; \
    byte_test:2,>,200,0,big,relative; \
    flowbits:set,TLS.heartbleed; \
    classtype:misc-attack; sid:3000004; rev:1;)

Suricata TLS parser

Pierre Chifflier has written detection logic for the Suricata TLS parser. This is in our git master and will be part of 2.0.1. If you run this code, enable these rules:

alert tls any any -> any any ( \
    msg:"SURICATA TLS overflow heartbeat encountered, possible exploit attempt (heartbleed)"; \
    flow:established; app-layer-event:tls.overflow_heartbeat_message; \
    flowint:tls.anomaly.count,+,1; classtype:protocol-command-decode; \
    reference:cve,2014-0160; sid:2230012; rev:1;)
alert tls any any -> any any ( \
    msg:"SURICATA TLS invalid heartbeat encountered, possible exploit attempt (heartbleed)"; \
    flow:established; app-layer-event:tls.invalid_heartbeat_message; \
    flowint:tls.anomaly.count,+,1; classtype:protocol-command-decode; \
    reference:cve,2014-0160; sid:2230013; rev:1;)

Ticket: https://redmine.openinfosecfoundation.org/issues/1173
Pull Request: https://github.com/inliniac/suricata/pull/924

Other Resources

- My fellow country (wo)men of Fox-IT have Snort rules here: http://blog.fox-it.com/2014/04/08/openssl-heartbleed-bug-live-blog/ These rules detect suspiciously large heartbeat response sizes
– Oisf-users has a thread: https://lists.openinfosecfoundation.org/pipermail/oisf-users/2014-April/003603.html
– Emerging Threats has a thread: https://lists.emergingthreats.net/pipermail/emerging-sigs/2014-April/024049.html
– Sourcefire has made rules available as well http://vrt-blog.snort.org/2014/04/heartbleed-memory-disclosure-upgrade.html These should work on Suricata as well.

Update 1:
– Pierre Chifflier correctly noted that hb_len doesn’t contain the ‘type’ and ‘size’ fields (3 bytes total), while ‘len’ does. So updated the check.
Update 2:
– Yonathan Klijnsma pointed me at the difference between the request and the response: https://twitter.com/ydklijnsma/status/453514484074962944. I’ve updated the rule to only inspect the script against requests.
Update 3:
– Better rule formatting
– Add non-LUA rules as well
Update 4:
– ET is going to add these rules: https://lists.emergingthreats.net/pipermail/emerging-sigs/2014-April/024056.html
Update 5:
– Updated the LUA script after feedback from Ivan Ristic. The padding issue was ignored.
Update 6:
– Added Pierre Chifflier’s work on detecting this in the Suricata TLS parser.
– Added reference to Sourcefire VRT rules

Suricata 2.0 and beyond

Today I finally released Suricata 2.0. The 2.0 branch opened in December 2012. In the little over a year that it’s development lasted, we have closed 183 tickets. We made 1174 commits, with the following stats:

582 files changed, 94782 insertions(+), 63243 deletions(-)

So, a significant update! In total, 17 different people made commits. I’m really happy with how much code and features were contributed. When starting Suricata this was what I really hoped for, and it seems to be working!

Eve

The feature I’m most excited about is ‘Eve’. It’s the nickname of a new logging output module ‘Extendible Event Format’. It’s an all JSON event stream that is very easy to parse using 3rd party tools. The heavy lifting has been done by Tom Decanio. Combined with Logstash, Elasticsearch and Kibana, this allows for really easy graphical dashboard creation. This is a nice addition to the existing tools which are generally more alert centered.

kibana300 kibana300map kibana-suri

Splunk support is easy as well, as Eric Leblond has shown:

regit-Screenshot-from-2014-03-05-231712

Looking forward

While doing releases is important and somewhat nice too, the developer in me is always glad when they are over. Leading up to a release there is a slow down of development, when most time is spent on fixing release critical bugs and doing some polishing. This slow down is a necessary evil, but I’m glad when we can start merging bigger changes again.

In the short term, I shooting for a fairly quick 2.0.1 release. There are some known issues that will be addressed in that.

More interestingly from a development perspective is the opening of the 2.1 branch. I’ll likely open that in a few weeks. There are a number of features in progress for 2.1. I’m working on speeding up pcap recording, which is currently quite inefficient. More interestingly, Lua output scripting. A preview of this work is available here  with some example scripts here.

Others are working on nice things as well: improving protocol support for detection and logging, nflog and netmap support, taxii/stix integration, extending our TLS support and more.

I’m hoping the 2.1 cycle will be shorter than the last, but we’ll see how it goes :)

tcpreplay on Intel 82576

For my Suricata QA setup, I’m using tcpreplay on a dual port gigabit NIC. The idea is to blast out packets on one port and then have Suricata listen on the other part.

For the traffic replay I’m using tcpreplay 3.4.4 from the Ubuntu archive. As I have a lot of pcaps to process I intend to use the –topspeed option to keep runtimes as low as possible. This will result in approximately ~500Mbps on this box, as the pcaps come from a nas.

While validating the replay results, I noticed that there was a lot of packet reordering going on. This seemed odd as tcpreplay replays packets in order. The docs seemed to suggest the driver/NIC does this: http://tcpreplay.synfin.net/wiki/FAQ#tcpreplayissendingpacketsoutoforder

It turned out that this is caused by the driver using multiple tx-queues.

dmesg:

[    1.143444] igb 0000:03:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)

With the help of Luca Deri I was able to reduce the number of queues.

To do this, the igb driver module needs to be passed an option, RSS=1. However, the igb driver that comes with Ubuntu 13.10 (which has version 5.0.5k) does not support this option.

The latest version is needed, which can be downloaded from http://sourceforge.net/projects/e1000/files/igb%20stable/5.1.2/

After installing it, remove the current module and load the new module with the RSS option:

modprobe -r igb
modprobe igb RSS=1

Confirm the result in dmesg:

[  834.376632] igb 0000:03:00.1: Using MSI-X interrupts. 1 rx queue(s), 1 tx queue(s)

With this, tcpreplay at topspeed will not result in reordered packets.

Many thanks to Luca Deri for putting me on the right track here.