Tag Archives: luajit

Detecting OpenSSL Heartbleed with Suricata

Posted on by

The OpenSSL heartbleed vulnerability is a pretty serious weakness in OpenSSL that can lead to information disclosure, in some cases even to to private key leaking. Please see this post here http://blog.existentialize.com/diagnosis-of-the-openssl-heartbleed-bug.html for more info.

This is a case where an IDS is able to detect the vuln, even though we’re talking about TLS.

LUA

I’ve written a quick and dirty LUA script to detect it:

alert tls any any -> any any ( \
    msg:"TLS HEARTBLEED malformed heartbeat record"; \
    flow:established,to_server; dsize:>7; \
    content:"|18 03|"; depth:2; lua:tls-heartbleed.lua; \
    classtype:misc-attack; sid:3000001; rev:1;)

The script:

function init (args)
    local needs = {}
    needs["payload"] = tostring(true)
    return needs
end

function match(args)
    local p = args['payload']
    if p == nil then
        --print ("no payload")
        return 0
    end
 
    if #p < 8 then
        --print ("payload too small")
    end
    if (p:byte(1) ~= 24) then
        --print ("not a heartbeat")
        return 0
    end
 
    -- message length
    len = 256 * p:byte(4) + p:byte(5)
    --print (len)
 
    -- heartbeat length
    hb_len = 256 * p:byte(7) + p:byte(8)

    -- 1+2+16
    if (1+2+16) >= len  then
        print ("invalid length heartbeat")
        return 1
    end

    -- 1 + 2 + payload + 16
    if (1 + 2 + hb_len + 16) > len then
        print ("heartbleed attack detected: " .. (1 + 2 + hb_len + 16) .. " > " .. len)
        return 1
    end
    --print ("no problems")
    return 0
end
return 0

Regular rules

Inspired by the FOX-IT rules from http://blog.fox-it.com/2014/04/08/openssl-heartbleed-bug-live-blog/, here are some non-LUA rules:

Detect a large response.

alert tls any any -> any any ( \
    msg:"TLS HEARTBLEED heartbeat suspiciuous large record"; \
    flow:established,to_client; dsize:>7; \
    content:"|18 03|"; depth:2; \
    byte_test:2,>,200,3,big; classtype:misc-attack; \
    sid:3000002; rev:1;)

Detect a large response following a large request (flow bit is either set by the LUA rule above or by the rule that follows):

alert tls any any -> any any ( \
    msg:"TLS HEARTBLEED heartbeat attack likely succesful"; \
    flowbits:isset,TLS.heartbleed; \
    flow:established,to_client; dsize:>7; \
    content:"|18 03|"; depth:2; byte_test:2,>,200,3,big; \
    classtype:misc-attack; \
    sid:3000003; rev:1;)

Detect a large request, set flowbit:

alert tls any any -> any any ( \
    msg:"TLS HEARTBLEED heartbeat suspiciuous large request"; \
    flow:established,to_server; content:"|18 03|"; depth:2; \
    content:"|01|"; distance:3; within:1; \
    byte_test:2,>,200,0,big,relative; \
    flowbits:set,TLS.heartbleed; \
    classtype:misc-attack; sid:3000004; rev:1;)

Suricata TLS parser

Pierre Chifflier has written detection logic for the Suricata TLS parser. This is in our git master and will be part of 2.0.1. If you run this code, enable these rules:

alert tls any any -> any any ( \
    msg:"SURICATA TLS overflow heartbeat encountered, possible exploit attempt (heartbleed)"; \
    flow:established; app-layer-event:tls.overflow_heartbeat_message; \
    flowint:tls.anomaly.count,+,1; classtype:protocol-command-decode; \
    reference:cve,2014-0160; sid:2230012; rev:1;)
alert tls any any -> any any ( \
    msg:"SURICATA TLS invalid heartbeat encountered, possible exploit attempt (heartbleed)"; \
    flow:established; app-layer-event:tls.invalid_heartbeat_message; \
    flowint:tls.anomaly.count,+,1; classtype:protocol-command-decode; \
    reference:cve,2014-0160; sid:2230013; rev:1;)

Ticket: https://redmine.openinfosecfoundation.org/issues/1173
Pull Request: https://github.com/inliniac/suricata/pull/924

Other Resources

– My fellow country (wo)men of Fox-IT have Snort rules here: http://blog.fox-it.com/2014/04/08/openssl-heartbleed-bug-live-blog/ These rules detect suspiciously large heartbeat response sizes
– Oisf-users has a thread: https://lists.openinfosecfoundation.org/pipermail/oisf-users/2014-April/003603.html
– Emerging Threats has a thread: https://lists.emergingthreats.net/pipermail/emerging-sigs/2014-April/024049.html
– Sourcefire has made rules available as well http://vrt-blog.snort.org/2014/04/heartbleed-memory-disclosure-upgrade.html These should work on Suricata as well.

Update 1:
– Pierre Chifflier correctly noted that hb_len doesn’t contain the ‘type’ and ‘size’ fields (3 bytes total), while ‘len’ does. So updated the check.
Update 2:
– Yonathan Klijnsma pointed me at the difference between the request and the response: https://twitter.com/ydklijnsma/status/453514484074962944. I’ve updated the rule to only inspect the script against requests.
Update 3:
– Better rule formatting
– Add non-LUA rules as well
Update 4:
– ET is going to add these rules: https://lists.emergingthreats.net/pipermail/emerging-sigs/2014-April/024056.html
Update 5:
– Updated the LUA script after feedback from Ivan Ristic. The padding issue was ignored.
Update 6:
– Added Pierre Chifflier’s work on detecting this in the Suricata TLS parser.
– Added reference to Sourcefire VRT rules

More on Suricata lua flowints

Posted on by

This morning I added flowint lua functions for incrementing and decrementing flowints. From the commit:

Add flowint lua functions for incrementing and decrementing flowints.

First use creates the var and inits to 0. So a call:

    a = ScFlowintIncr(0)

Results in a == 1.

If the var reached UINT_MAX (2^32), it’s not further incremented. If the
var reaches 0 it’s not decremented further.

Calling ScFlowintDecr on a uninitialized var will init it to 0.

Example script:

    function init (args)
        local needs = {}
        needs["http.request_headers"] = tostring(true)
        needs["flowint"] = {"cnt_incr"}
        return needs
    end

    function match(args)
        a = ScFlowintIncr(0);
        if a == 23 then
            return 1
        end

        return 0
    end
    return 0

This script matches the 23rd time it’s invoked on a flow.

Compared to yesterday’s flowint script and the earlier flowvar based counting script, this performs better:

   Num      Rule         Gid      Rev      Ticks        %      Checks   Matches  Max Ticks   Avg Ticks   Avg Match   Avg No Match
  -------- ------------ -------- -------- ------------ ------ -------- -------- ----------- ----------- ----------- -------------- 
  1        1            1        0        2434188332   59.71  82249    795      711777      29595.35    7683.20     29809.22   
  2        2            1        0        1015328580   24.91  82249    795      154398      12344.57    3768.66     12428.27   
  3        3            1        0        626858067    15.38  82249    795      160731      7621.47     3439.91     7662.28

The rules:

alert http any any -> any any (msg:"LUAJIT HTTP flowvar match"; luajit:lua_flowvar_cnt.lua; flow:to_server; sid:1;)
alert http any any -> any any (msg:"LUAJIT HTTP flowint match"; luajit:lua_flowint_cnt.lua; flow:to_server; sid:2;)
alert http any any -> any any (msg:"LUAJIT HTTP flowint incr match"; luajit:lua_flowint_incr_cnt.lua; flow:to_server; sid:3;)

Please comment, discuss, review etc on the oisf-devel list.

Suricata Lua scripting flowint access

Posted on by

A few days ago I wrote about my Emerging Threats sponsored work to support flowvars from Lua scripts in Suricata.

Today, I updated that support. Flowvar ‘sets’ are now real time. This was needed to fix some issues where a script was invoked multiple times in single rule, which can happen with some buffers, like HTTP headers.

Also, I implemented flowint support. Flowints in Suricata are integers stored in the flow context.

Example script:

function init (args)
    local needs = {}
    needs["http.request_headers"] = tostring(true)
    needs["flowint"] = {"cnt"}
    return needs
end

function match(args)
    a = ScFlowintGet(0);
    if a then
        ScFlowintSet(0, a + 1)
    else
        ScFlowintSet(0, 1)
    end 
        
    a = ScFlowintGet(0);
    if a == 23 then
        return 1
    end 
    
    return 0
end 

return 0

It does the same thing as this flowvar script:

function init (args)
    local needs = {}
    needs["http.request_headers"] = tostring(true)
    needs["flowvar"] = {"cnt"}
    return needs
end

function match(args)
    a = ScFlowvarGet(0);
    if a then
        a = tostring(tonumber(a)+1)
        ScFlowvarSet(0, a, #a)
    else
        a = tostring(1)
        ScFlowvarSet(0, a, #a)
    end 
    
    if tonumber(a) == 23 then
        return 1
    end
    
    return 0
end

return 0

Only, at about half the cost:

   Num      Rule         Gid      Rev      Ticks        %      Checks   Matches  Max Ticks   Avg Ticks   Avg Match   Avg No Match
  -------- ------------ -------- -------- ------------ ------ -------- -------- ----------- ----------- ----------- -------------- 
  1        1            1        0        2392221879   70.56  82249    795      834993      29085.12    6964.14     29301.02   
  2        2            1        0        998297994    29.44  82249    795      483810      12137.51    4019.44     12216.74

Suricata Lua scripting flowvar access

Posted on by

Funded by Emerging Threats, I’ve been working on giving the lua scripts access to flowvars.

Currently only “flowvars” are done, “flowints” will be next. Please review the code at:
https://github.com/inliniac/suricata/tree/dev-lua-flowvar

Pcre based flowvar capturing is done in a post-match fashion. If the rule containing the “capture” matches, the var is stored in the flow.

For lua scripting, this wasn’t what the rule writers wanted. In this case, the flowvars are stored in the flow regardless of a rule match.

The way a script can start using flowvars is by first registering which one it needs access to:

function init (args)
    local needs = {}
    needs["http.request_headers.raw"] = tostring(true)
    needs["flowvar"] = {"cnt"}
    return needs
end

More than one can be registered, e.g.:

    needs["flowvar"] = {"cnt", "somevar", "anothervar" }

The maximum is 15 per script. The order of the vars matters. As Suricata uses id’s internally, to use the vars you have to use id’s as well. The first registered var has id 0, 2nd 1 and so on:

function match(args)
    a = ScFlowvarGet(0);
    if a then
        print ("We have an A: " .. (a))
        a = tostring(tonumber(a)+1)
        print ("A incremented to: " .. (a))
        ScFlowvarSet(0, a, #a)
    else
        print "Init A to 1"
        a = tostring(1)
        ScFlowvarSet(0, a, #a)
    end

    print ("A is " .. (a))
    if tonumber(a) == 23 then
        print "Match!"
        return 1
    end

    return 0
end

You can also use a var:

function init (args)
    local needs = {}
    needs["http.request_headers.raw"] = tostring(true)
    needs["flowvar"] = {"blah", "cnt"}
    return needs
end

local var_cnt = 1

function match(args)
    a = ScFlowvarGet(var_cnt);
    if a then
        print ("We have an A: " .. (a))
        a = tostring(tonumber(a)+1)
        print ("A incremented to: " .. (a))
        ScFlowvarSet(var_cnt, a, #a)
    else
        print "Init A to 1"
        a = tostring(1)
        ScFlowvarSet(var_cnt, a, #a)
    end

    print ("A is " .. (a))
    if tonumber(a) == 23 then
        print "Match!"
        return 1
    end

    return 0
end

Flowvars are set at the end of the rule’s inspection, so after the script has run.

When multiple stores are done from the script and/or pcre, the last match will win. So if order matters, rule priority can be used to control inspection order.

Thoughts, comments, and code review highly welcomed at the oisf-devel list.

Suricata 1.4 is out

Posted on by

About 5 months after 1.3 came out we’ve released 1.4, and we’ve been quite busy. Eric Leblond’s post here has all the stats and graphs. There are three big new features: unix socket, ip reputation and luajit. For each of these the same is true: it’s usesable now, but it’s the potential that we’re most excited about. Over the next months we’ll be extending each of those to be even more useful. We’re very much interested in ideas and feedback.

Performance obviously matters to many in the IDS world, and here too we have improved Suricata quite a bit again. We now have Suricata 1.4 running on a ISP 10gbit/s network on commodity hardware with a large ET ruleset. Of course, YMMV, but we’re definitely making a lot of progress here.

Sometimes the little things matter a lot as well. A minor new feature is that live “drop” stats are the the stats.log now:

capture.kernel_packets    | AFPacketem21              | 13640581
capture.kernel_drops      | AFPacketem21              | 442864
capture.kernel_packets    | AFPacketem22              | 7073228
capture.kernel_drops      | AFPacketem22              | 9449
capture.kernel_packets    | AFPacketem23              | 10528970
capture.kernel_drops      | AFPacketem23              | 148281
capture.kernel_packets    | AFPacketem24              | 7212584
capture.kernel_drops      | AFPacketem24              | 12643
capture.kernel_packets    | AFPacketem25              | 9763439
capture.kernel_drops      | AFPacketem25              | 17874
capture.kernel_packets    | AFPacketem26              | 10464106
capture.kernel_drops      | AFPacketem26              | 20378
capture.kernel_packets    | AFPacketem27              | 8869182
capture.kernel_drops      | AFPacketem27              | 18336
capture.kernel_packets    | AFPacketem28              | 7925045
capture.kernel_drops      | AFPacketem28              | 258168

This is supported for AF_PACKET, PF_RING and libpcap.

Last August we’ve added Suricata to github to make it easier to participate. Also, the code review tools associated with the pull requests are very useful. Github has been an unexpected success for us. At the time of writing there are 24 forks of Suricata on it, I’ve processed about 250 pull requests. The patches that have been submitted range from small fixes to full blown features, and more are on the way. I’m very grateful for these contributions and everyone’s patience with me.

Now that 1.4 is out, we’ll be taking it slow over the holidays. The team has been working like crazy, and everyone deserves a break. So the next weeks we’ll focus on further consolidation, fixing bugs that no doubt will pop up. Other than that, things will be slow. After the holidays we’ll start planning for the next milestone. Again, your ideas and contributions are very welcome! 🙂

Suricata 1.4 development update

Posted on by

Today, a day after 1.3.2, we’ve released 1.4beta2. While 1.3.2 is an important update for those running 1.3.1 or lower, today’s release is where things get exciting. A lot of things were improved and added. Let me show some numbers first.

The 1.4beta2 release is a pretty big update over 1.4beta1 as it touches over 5k lines of code:

234 files changed, 5033 insertions(+), 3759 deletions(-)

Compared to 1.4beta2 vs yesterday’s 1.3.2 it’s clear over 11k lines of code are touched:

262 files changed, 11406 insertions(+), 5794 deletions(-)

Personally, I’ve been working on two main area’s: defrag engine and the luajit integration, and a couple of other things.

Defrag

The defrag engine was the last major subsystem that still used a Big Lock. Defrag uses so called “trackers” to track fragments belonging to a single IP packet. These trackers are stored in a hash table. 1.3 and prior used a hash that had no locking, so it relied on a Big Lock to protect it’s operations. Suricata has had fine grained hashes for flow and host tables for some time already, so it made sense to port defrag over as well.

Luajit

I’ve written about the luajit a couple of times already. While the basic functionality debuted in beta1, the code has been completely overhauled. The most important change that is user visible is the integration with the various HTTP inspection engines. This did result in a limitation though, for now you can just inspect one HTTP buffer per script.

A weird challenge with luajit is that it’s “state” needs to be in the 32 bit part of memory. The reason isn’t clear to me, but this gave us some trouble. Some users use many rules and agressive pattern matcher settings. When after this memory usage the luajit states had to be alloc’d, it failed. I’ve worked around this by allocating a bunch of states in advance, hoping they’ll end up in the proper memory. We’ll see how that will work.

Misc

I’ve also largely rewritten the optional rule profiling to perform better. Here too, a Big Lock was removed. The accounting is now first done on a per thread basis, and only merged at detection engine shut down. Another nice feature is that it will now print the profiling stats during a live rule reload as well.

Next, I’ve improved performance of the decode, stream and app layer event keywords. They were quite expensive as they were checked quite often. I’ve now added a prefilter check to the detection engine’s prefilter stage. Helps quite a bit!

Finally, I’ve been working on getting global and rule threshold play well together. This work isn’t done yet, but some real progress has been made. Work is tracker here and documentation lives here.

So all in all quite a bit of changes. Please help us test this so we can move to a stable and high performing 1.4! 🙂

Suricata luajit update

Posted on by

After an exciting week of meeting and working with the team around the RAID conference, time for another lua update.

The keyword supports an interesting set of buffers now:

packet
payload

http.uri
http.uri.raw
http.request_line
http.request_headers
http.request_headers.raw
http.request_cookie
http.request_user_agent
http.request_body

http.response_headers
http.response_headers.raw
http.response_body
http.response_cookie

The http keywords are now integrated into their respective inspection engines. This led to one important limitation for now: you can only inspect one such buffer per script.

We pass the inspection offset to the script as well for these. In the lua script you can access it as follows:

function match(args)
    a = tostring(args["http.request_headers.raw"])
    o = args["offset"]

    s = a:sub(o)
    print (s)

    return 0
end

In a buffer “Mozilla/5.0” and a signature “content:Mozilla;”, “s” in the script will contain “/5.0”. At this moment there is no way yet to pass back an offset from the script to the inspection engine.

On the performance side things are looking good as well. At RAID Will Metcalf converted a set of 6 ETpro sigs to a single lua script. It resulted in better detection accuracy and better performance. That work is still private, but we’ll get some real world scripts public soon! 🙂

Update 10/4: this code is now available for testing in the new Suricata 1.4beta2 release!

First impressions of lua(jit) performance in Suricata

Posted on by

Today I decided to look into the potential performance of the luajit keyword a bit. It’s important to know if this can perform at reasonable speeds so that we can actually use it in real deployments. Even if we can’t the feature may still be appealing though, for offline pcap analysis.

So far, the results are rather encouraging.

First, I added 2 buffers today: http.uri, which contains the normalized uri (same buffer as the http_uri content modifier inspects) and http.request_line, which is the request line given to us by libhtp. This contains method, separators, uri, HTTP version.

Next I created 5 rules. A pure Lua rule (1), a Lua rule with content prefilter (2), a Lua rule with content and pcre prefilter (3), a pcre rule with content prefilter (4) and a pure pcre rule (5).

alert http any any -> any any (msg:"LUAJIT HTTP POST test, pure lua"; luajit:test2.lua; sid:1;)
alert http any any -> any any (msg:"LUAJIT HTTP POST test, content prefilter"; content:"POST"; http_method; content:".php"; http_uri; luajit:test2.lua; sid:2;)
alert http any any -> any any (msg:"LUAJIT HTTP POST test, pcre prefilter"; content:"POST"; http_method; content:".php"; http_uri; pcre:"/^POST\s+\/.*\.php\s+HTTP\/1\.0\r\n/m"; luajit:test2.lua; sid:3;)
alert http any any -> any any (msg:"LUAJIT HTTP POST test, pcre no lua"; content:"POST"; http_method; content:".php"; http_uri; pcre:"/^POST\s+\/.*\.php\s+HTTP\/1\.0\r\n/m"; sid:4;)
alert http any any -> any any (msg:"LUAJIT HTTP POST test, pure pcre"; pcre:"/^POST\s+\/.*\.php\s+HTTP\/1\.0\r\n/m"; sid:5;)

and the following Lua script:

function init (args)
    local needs = {}
    needs["http.request_line"] = tostring(true)
    return needs
end

-- match if packet and payload both contain HTTP
function match(args)
    a = tostring(args["http.request_line"])
    if #a > 0 then
        if a:find("^POST%s+/.*%.php%s+HTTP/1.0$") then
            return 1
        end
    end
  
    return 0
end

return 0

The script does a pattern match (regex even) against the request line, something I’d consider quite expensive.

So, how does this perform? Here are the rule perf stats:

   Num      Rule        Avg Ticks   Avg Match   Avg No Match
  -------- ------------ ----------- ----------- -------------- 
  1        5            12113.53    7198.08     12114.28   
  2        3            11638.15    39842.23    9424.83    
  3        2            10682.71    35497.08    10194.56   
  4        1            8812.31     15841.85    8807.01    
  5        4            8536.46     20074.97    7630.97

Pure pcre rules are bad, we all know that, but they end up being most expensive in this test which surprises me. The pure Lua rule is quite a bit cheaper and even ends up below the prefilted Lua rules. Only the content+pcre (no Lua) rule is slightly faster.

So far things look rather good for the lua keyword. Who knows, maybe it can even be used on live traffic.

The work continues! 🙂

Suricata lua continued

Posted on by

Today I improved the lua jit support in Suricata further. The scripts will now need to express their “needs” through an “init” function in the script that is called only at Suricata startup.

The “init” function fills a lua table. This will allow the user to indicate what buffers the script needs to inspect. The script will then only be invoked when these buffers are actually available, so the script won’t have to worry about whether or not some data is unavailable or not. Also, only these buffers are passed to the script, so safing the overhead of copying unnecessary buffers.

The earlier example extended:

function init (args)
    local needs = {}
    needs["packet"] = tostring(true)
    needs["payload"] = tostring(true)
    return needs
end

-- match if packet and payload both contain HTTP
function match(args)
    pkt = 0
    pay = 0
    
    for k,v in pairs(args) do
        if tostring(k) == "packet" then
            a = tostring(v)
            if #a > 0 then
                if a:find("HTTP") then
                    pkt = 1
                end
            end
        elseif tostring(k) == "payload" then
            a = tostring(v)
            if #a > 0 then
                if a:find("HTTP") then
                    pay = 1
                end
            end
        end

        if pay == 1 and pkt == 1 then
            return 1
        end
    end

    return 0
end

return 0

The new part is:

function init (args)
    local needs = {}
    needs["packet"] = tostring(true)
    needs["payload"] = tostring(true)
    return needs
end

Currently only “packet”, which is the complete raw packet including protocol headers such as ethernet, and “payload” are available. I will extend this to include “stream”, “http.uri”, “http.headers” and so on. Also, I was thinking that the script could also require a flowvar or flowint.

Another change is that the user can also return a table instead of just a 1 or 0. In that table “retval” is used as the return code. The idea here is that later we’ll be able to return flowvars and flowints as well. Example:

function init (args)
    local needs = {}
    needs["packet"] = tostring(true)
    return needs
end

-- return match via table
function match(args)
    local result = {}
    result["retval"] = tostring(1)
    return result
end

Here the the script always matches, and passes the match back to Suricata through a table.

So far mostly internal changes and no extra detection features, but it’s getting some more shape. Looking forward to feedback! 🙂

First beta for Suricata 1.4

Posted on by

The first test release for the new Suricata 1.4 branch as just been released. Some really exciting stuff was added. Let me highlight some of it:

AF_PACKET IPS mode: Eric Leblond has been working on extending the passive AF_PACKET support to support IPS as well. Eric has documented the new feature on his blog.

TLS logging and certificate storage: created by contributor Jean-Paul Roliers under guidance of Eric Leblond. As a bonus, a rule keyword to match on certifcate fingerprints.

Custom HTTP logging: contributor Ignacio Sanchez created a new output mode for our HTTP log module, allowing the admin to customize the log message format. He has made it compatible to Apache’s mod_log_config. For more information, see our wiki page.

Tunnel decoding: Michel Saborde opened a bunch of tickets for Teredo, IPv4-in-IPv6 and IPv6-in-IPv6 tunneling. Saved a lot of time in Eric’s implementation.

There is more, like the luajit keyword I wrote about yesterday here.

So there are a lot of changes. Git gives us the following numbers: “106 files changed, 6966 insertions(+), 2259 deletions(-)” in just 3 weeks. This means the release is definitely beta quality, so use with care.

Grab it here: http://www.openinfosecfoundation.org/download/suricata-1.4beta1.tar.gz

Next week the team will be in Amsterdam for the RAID 2012 conference. After that we’ll continue to work towards 1.4beta2. For an idea of what is coming, check the milestone.

Until than, have fun with this new beta. Many thanks to our generous contributors!