Monthly Archives: September 2012

Suricata luajit update

Posted on by

After an exciting week of meeting and working with the team around the RAID conference, time for another lua update.

The keyword supports an interesting set of buffers now:

packet
payload

http.uri
http.uri.raw
http.request_line
http.request_headers
http.request_headers.raw
http.request_cookie
http.request_user_agent
http.request_body

http.response_headers
http.response_headers.raw
http.response_body
http.response_cookie

The http keywords are now integrated into their respective inspection engines. This led to one important limitation for now: you can only inspect one such buffer per script.

We pass the inspection offset to the script as well for these. In the lua script you can access it as follows:

function match(args)
    a = tostring(args["http.request_headers.raw"])
    o = args["offset"]

    s = a:sub(o)
    print (s)

    return 0
end

In a buffer “Mozilla/5.0” and a signature “content:Mozilla;”, “s” in the script will contain “/5.0”. At this moment there is no way yet to pass back an offset from the script to the inspection engine.

On the performance side things are looking good as well. At RAID Will Metcalf converted a set of 6 ETpro sigs to a single lua script. It resulted in better detection accuracy and better performance. That work is still private, but we’ll get some real world scripts public soon! 🙂

Update 10/4: this code is now available for testing in the new Suricata 1.4beta2 release!

First impressions of lua(jit) performance in Suricata

Posted on by

Today I decided to look into the potential performance of the luajit keyword a bit. It’s important to know if this can perform at reasonable speeds so that we can actually use it in real deployments. Even if we can’t the feature may still be appealing though, for offline pcap analysis.

So far, the results are rather encouraging.

First, I added 2 buffers today: http.uri, which contains the normalized uri (same buffer as the http_uri content modifier inspects) and http.request_line, which is the request line given to us by libhtp. This contains method, separators, uri, HTTP version.

Next I created 5 rules. A pure Lua rule (1), a Lua rule with content prefilter (2), a Lua rule with content and pcre prefilter (3), a pcre rule with content prefilter (4) and a pure pcre rule (5).

alert http any any -> any any (msg:"LUAJIT HTTP POST test, pure lua"; luajit:test2.lua; sid:1;)
alert http any any -> any any (msg:"LUAJIT HTTP POST test, content prefilter"; content:"POST"; http_method; content:".php"; http_uri; luajit:test2.lua; sid:2;)
alert http any any -> any any (msg:"LUAJIT HTTP POST test, pcre prefilter"; content:"POST"; http_method; content:".php"; http_uri; pcre:"/^POST\s+\/.*\.php\s+HTTP\/1\.0\r\n/m"; luajit:test2.lua; sid:3;)
alert http any any -> any any (msg:"LUAJIT HTTP POST test, pcre no lua"; content:"POST"; http_method; content:".php"; http_uri; pcre:"/^POST\s+\/.*\.php\s+HTTP\/1\.0\r\n/m"; sid:4;)
alert http any any -> any any (msg:"LUAJIT HTTP POST test, pure pcre"; pcre:"/^POST\s+\/.*\.php\s+HTTP\/1\.0\r\n/m"; sid:5;)

and the following Lua script:

function init (args)
    local needs = {}
    needs["http.request_line"] = tostring(true)
    return needs
end

-- match if packet and payload both contain HTTP
function match(args)
    a = tostring(args["http.request_line"])
    if #a > 0 then
        if a:find("^POST%s+/.*%.php%s+HTTP/1.0$") then
            return 1
        end
    end
  
    return 0
end

return 0

The script does a pattern match (regex even) against the request line, something I’d consider quite expensive.

So, how does this perform? Here are the rule perf stats:

   Num      Rule        Avg Ticks   Avg Match   Avg No Match
  -------- ------------ ----------- ----------- -------------- 
  1        5            12113.53    7198.08     12114.28   
  2        3            11638.15    39842.23    9424.83    
  3        2            10682.71    35497.08    10194.56   
  4        1            8812.31     15841.85    8807.01    
  5        4            8536.46     20074.97    7630.97

Pure pcre rules are bad, we all know that, but they end up being most expensive in this test which surprises me. The pure Lua rule is quite a bit cheaper and even ends up below the prefilted Lua rules. Only the content+pcre (no Lua) rule is slightly faster.

So far things look rather good for the lua keyword. Who knows, maybe it can even be used on live traffic.

The work continues! 🙂

Suricata lua continued

Posted on by

Today I improved the lua jit support in Suricata further. The scripts will now need to express their “needs” through an “init” function in the script that is called only at Suricata startup.

The “init” function fills a lua table. This will allow the user to indicate what buffers the script needs to inspect. The script will then only be invoked when these buffers are actually available, so the script won’t have to worry about whether or not some data is unavailable or not. Also, only these buffers are passed to the script, so safing the overhead of copying unnecessary buffers.

The earlier example extended:

function init (args)
    local needs = {}
    needs["packet"] = tostring(true)
    needs["payload"] = tostring(true)
    return needs
end

-- match if packet and payload both contain HTTP
function match(args)
    pkt = 0
    pay = 0
    
    for k,v in pairs(args) do
        if tostring(k) == "packet" then
            a = tostring(v)
            if #a > 0 then
                if a:find("HTTP") then
                    pkt = 1
                end
            end
        elseif tostring(k) == "payload" then
            a = tostring(v)
            if #a > 0 then
                if a:find("HTTP") then
                    pay = 1
                end
            end
        end

        if pay == 1 and pkt == 1 then
            return 1
        end
    end

    return 0
end

return 0

The new part is:

function init (args)
    local needs = {}
    needs["packet"] = tostring(true)
    needs["payload"] = tostring(true)
    return needs
end

Currently only “packet”, which is the complete raw packet including protocol headers such as ethernet, and “payload” are available. I will extend this to include “stream”, “http.uri”, “http.headers” and so on. Also, I was thinking that the script could also require a flowvar or flowint.

Another change is that the user can also return a table instead of just a 1 or 0. In that table “retval” is used as the return code. The idea here is that later we’ll be able to return flowvars and flowints as well. Example:

function init (args)
    local needs = {}
    needs["packet"] = tostring(true)
    return needs
end

-- return match via table
function match(args)
    local result = {}
    result["retval"] = tostring(1)
    return result
end

Here the the script always matches, and passes the match back to Suricata through a table.

So far mostly internal changes and no extra detection features, but it’s getting some more shape. Looking forward to feedback! 🙂

First beta for Suricata 1.4

Posted on by

The first test release for the new Suricata 1.4 branch as just been released. Some really exciting stuff was added. Let me highlight some of it:

AF_PACKET IPS mode: Eric Leblond has been working on extending the passive AF_PACKET support to support IPS as well. Eric has documented the new feature on his blog.

TLS logging and certificate storage: created by contributor Jean-Paul Roliers under guidance of Eric Leblond. As a bonus, a rule keyword to match on certifcate fingerprints.

Custom HTTP logging: contributor Ignacio Sanchez created a new output mode for our HTTP log module, allowing the admin to customize the log message format. He has made it compatible to Apache’s mod_log_config. For more information, see our wiki page.

Tunnel decoding: Michel Saborde opened a bunch of tickets for Teredo, IPv4-in-IPv6 and IPv6-in-IPv6 tunneling. Saved a lot of time in Eric’s implementation.

There is more, like the luajit keyword I wrote about yesterday here.

So there are a lot of changes. Git gives us the following numbers: “106 files changed, 6966 insertions(+), 2259 deletions(-)” in just 3 weeks. This means the release is definitely beta quality, so use with care.

Grab it here: http://www.openinfosecfoundation.org/download/suricata-1.4beta1.tar.gz

Next week the team will be in Amsterdam for the RAID 2012 conference. After that we’ll continue to work towards 1.4beta2. For an idea of what is coming, check the milestone.

Until than, have fun with this new beta. Many thanks to our generous contributors!

Suricata development training update

Posted on by

The Suricata development training at RAID 2012 next week is going to happen, so please all RSVP. It’s free!

If you’re planning to attend, please let me know what topics you are interested in. We have core devs in the room, so we can go hardcore on everything from the threading to packet capture to CUDA to pattern matching… also more straightforward stuff like extending Suricata with new keywords, log modules, etc.

Let me know! Oh and RSVP! 🙂

Suricata lua (jit) script keyword

Posted on by

So Will started bugging me (again) on doing scripting from Suricata and I gave in. Just committed extremely immature, incomplete, experimental luajit scripting support.

What it does is that it adds a new keyword, “luajit”. There is one argument, a script name. That script is then loaded from your rules directory and ran against a packet. No flow, http or any of that right now, just packets.

Example rule:
alert tcp any any -> any any (msg:"LUAJIT test"; luajit:test.lua; sid:1;)

This loads the script from /etc/suricata/rules/test.lua

The script has a “match” function that will return either 1 or 0. 1 for match, 0 for no match.

Example script:

-- match string HTTP in packet payload
function match(args)
    for k,v in pairs(args) do
       if tostring(k) == "payload" then
            a = tostring(v)
            if #a > 0 then
                if a:find("HTTP") then
                    return 1
                end
            end
        end
    end

    return 0
end

return 0
-- eof

The fun thing is that it works, but the best joke is that on my box this simple script makes no performance impact at all.

Currently only “payload” and “packet” keys are available. More will follow, or not. This is research stuff, and if we run into some major obstacle we’ll remove it or change it completely. Until then, let me know how you feel about it 🙂

Oh yeah, to enable add:
--with-libluajit-includes=/usr/include/luajit-2.0/ --with-libluajit-libraries=/usr/lib/x86_64-linux-gnu/
To your ./configure line. Adapt for your distro.

Happy scripting! 🙂