tcpreplay on Intel 82576

For my Suricata QA setup, I’m using tcpreplay on a dual port gigabit NIC. The idea is to blast out packets on one port and then have Suricata listen on the other part.

For the traffic replay I’m using tcpreplay 3.4.4 from the Ubuntu archive. As I have a lot of pcaps to process I intend to use the –topspeed option to keep runtimes as low as possible. This will result in approximately ~500Mbps on this box, as the pcaps come from a nas.

While validating the replay results, I noticed that there was a lot of packet reordering going on. This seemed odd as tcpreplay replays packets in order. The docs seemed to suggest the driver/NIC does this: http://tcpreplay.synfin.net/wiki/FAQ#tcpreplayissendingpacketsoutoforder

It turned out that this is caused by the driver using multiple tx-queues.

dmesg:

[    1.143444] igb 0000:03:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)

With the help of Luca Deri I was able to reduce the number of queues.

To do this, the igb driver module needs to be passed an option, RSS=1. However, the igb driver that comes with Ubuntu 13.10 (which has version 5.0.5k) does not support this option.

The latest version is needed, which can be downloaded from http://sourceforge.net/projects/e1000/files/igb%20stable/5.1.2/

After installing it, remove the current module and load the new module with the RSS option:

modprobe -r igb
modprobe igb RSS=1

Confirm the result in dmesg:

[  834.376632] igb 0000:03:00.1: Using MSI-X interrupts. 1 rx queue(s), 1 tx queue(s)

With this, tcpreplay at topspeed will not result in reordered packets.

Many thanks to Luca Deri for putting me on the right track here.

Disabling Threading in Tcl8.5 in Debian

sguil_logo_h

I’ve been spending the holidays to upgrade some of my own servers. One of them is the Sguil server I use. Until now it ran Debian Squeeze. On Debian Squeeze you could use tcl8.3, which has threading disabled. For Sguil tcl threading needs to be disabled:

ERROR: This version of tcl was compile with threading enabled. Sguil is NOT compatible with threading.

This is a compile time option in TCL, and the Debian Wheezy packages have it enabled by default. Here are the steps to create your own tcl deb with threading disabled:

# apt-get install dpkg-dev
# apt-get install devscripts

Get the tcl8.5 source package and build deps:

# apt-get source tcl8.5
# apt-get build-dep tcl8.5
# cd tcl8.5-8.5.11/

Next, edit the debian/rules file to disable threading. Remove the line:

                      --enable-threads \

Then, build the package:

# debuild -us -uc

And finally install the package:

# cd ..
# dpkg -i ../tcl8.5_8.5.11-2_amd64.deb

I followed this guide here at Debian Administration. It has some more detail on rebuilding debs.

Suricata Development Update

SuricataWith the holidays approaching and the 1.4.7 and 2.0beta2 releases out, I thought it was a good moment for some reflection on how development is going.

I feel things are going very well. It’s great to work with a group that approaches this project from different angles. OISF has budget have people work on overall features, quality and support. Next to that, our consortium supporters help develop the project: Tilera’s Ken Steele is working on the Tile hardware support, doing lots optimizations. Many of which benefit performance and overall quality for the whole project. Tom Decanio of Npulse is doing great work on the output side, unifying the outputs to be machine readable. Jason Ish of Emulex/Endace is helping out the configuration API, defrag, etc. Others, both from the larger community and our consortium, are helping as well.

QA

At our last meetup in Luxembourg, we’ve spend quite a bit of time discussing how we can improve the quality of Suricata. Since then, we’ve been working hard to add better and more regression and quality testing.

We’ve been using a Buildbot setup for some time now, where on a number of platforms we do basic build testing. First, this was done only against the git master(s). Eric has then created a new method using a script call prscript. It’s purpose is to push a git branch to our buildbot _before_ it’s even considered for inclusion.

Recently, with cooperation of Emerging Threats, we’ve been extending this setup to include a large set of rule+pcap matches that are checked against each commit. This too is part of the pre-include QA process.

There are many more plans to extend this setup further. I’ve set up a private buildbot instance to serve as a staging area. Things we’ll be adding soon:
– valgrind testing
– DrMemory testing
– clang/scan-build
– cppcheck

Ideally, each of those tools would report 0 issues, but thats hard in practice. Sometimes there are false positives. Most tools support some form of suppression, so one of the tasks is to create those.

We’ve spend some time updating our documents regarding contributing to our code base. Please take a moment to a general contribution page, aimed at devs new to the project.

Next to this, this document describes quality requirements for our code, commits and pull requests.

Suricata 2.0

Our roadmap shows a late January 2.0 final release. It might slip a little bit, as we have a few larger changes to make:
– a logging API rewrite is in progress
“united” output, an all JSON log method written by Tom Decanio of Npulse [5]
app-layer API cleanup and update that Anoop is working on [6]

Wrapping up, I think 2013 was a very good year for Suricata. 2014 will hopefully be even better. We will be announcing some new support soon, are improving our training curicullum and will just be working hard to make Suricata better.

But first, the holidays. Cheers!

Suricata profiling per keyword

Last week I’ve added some more profiling options to Suricata. It’s part of the current git master. It’s enabled only when --enable-profiling and then through the suricata.yaml:

profiling:
  # per keyword profiling
  keywords:
    enabled: yes
    filename: keyword_perf.log
    append: yes

This will output a table similar to below:

--------------------------------------------------------------------------
Date: 11/7/2013 -- 15:13:11
--------------------------------------------------------------------------
Stats for: total
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
threshold        355324491   190574   409      72276       1864.00     3625.00     1860.00    
content          1274592063  534328   196738   312321      2385.00     2424.00     2362.00    
pcre             56626031    11149    824      254562      5079.00     12234.00    4507.00    
byte_test        153287955   128254   32109    67989       1195.00     1658.00     1040.00    
byte_jump        3676404     2041     2041     15939       1801.00     1801.00     0.00       
flow             38276182    22842    22842    63987       1675.00     1675.00     0.00       
isdataat         580764      558      556      2427        1040.00     1040.00     1017.00    
dsize            2212029     2062     2061     3711        1072.00     1072.00     789.00     
flowbits         1677209     874      870      9873        1919.00     1923.00     884.00     
itype            1653        2        1        1386        826.00      267.00      1386.00    
icode            27383781    93827    2        25545       291.00      1021.00     291.00     
flags            192751968   245519   189709   255639      785.00      753.00      892.00     
urilen           6149297     6142     1099     28299       1001.00     1395.00     915.00     
byte_extract     143091      78       78       7743        1834.00     1834.00     0.00       
--------------------------------------------------------------------------
Stats for: packet
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
flow             38276182    22842    22842    63987       1675.00     1675.00     0.00       
dsize            2212029     2062     2061     3711        1072.00     1072.00     789.00     
flowbits         351171      294      290      5526        1194.00     1198.00     884.00     
itype            1653        2        1        1386        826.00      267.00      1386.00    
icode            27383781    93827    2        25545       291.00      1021.00     291.00     
flags            192751968   245519   189709   255639      785.00      753.00      892.00     
--------------------------------------------------------------------------
Stats for: packet/stream payload
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
content          1203990910  512902   183628   312321      2347.00     2365.00     2337.00    
pcre             28087301    6598     54       254562      4256.00     12279.00    4190.00    
byte_test        153287955   128254   32109    67989       1195.00     1658.00     1040.00    
byte_jump        3676404     2041     2041     15939       1801.00     1801.00     0.00       
isdataat         578172      556      554      2427        1039.00     1039.00     1017.00    
byte_extract     143091      78       78       7743        1834.00     1834.00     0.00       
--------------------------------------------------------------------------
Stats for: http uri
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
content          44775802    13102    8351     60993       3417.00     3257.00     3698.00    
pcre             18284421    3646     97       61338       5014.00     8916.00     4908.00    
isdataat         2592        2        2        1725        1296.00     1296.00     0.00       
urilen           6149297     6142     1099     28299       1001.00     1395.00     915.00     
--------------------------------------------------------------------------
Stats for: http raw uri
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
pcre             9534        2        0        4953        4767.00     0.00        4767.00    
--------------------------------------------------------------------------
Stats for: http client body
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
content          1556904     441      181      58476       3530.00     2874.00     3986.00    
pcre             63924       6        6        17358       10654.00    10654.00    0.00       
--------------------------------------------------------------------------
Stats for: http headers
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
content          23688244    7631     4348     31098       3104.00     3311.00     2829.00    
pcre             9998970     859      667      71904       11640.00    12727.00    7862.00    
--------------------------------------------------------------------------
Stats for: http stat code
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
content          80052       39       20       3699        2052.00     2199.00     1898.00    
--------------------------------------------------------------------------
Stats for: http method
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
content          476334      203      201      27240       2346.00     2351.00     1846.00    
--------------------------------------------------------------------------
Stats for: http cookie
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
content          23817       10       9        2763        2381.00     2384.00     2358.00    
pcre             181881      38       0        13095       4786.00     0.00        4786.00    
--------------------------------------------------------------------------
Stats for: post-match
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
flowbits         1326038     580      580      9873        2286.00     2286.00     0.00       
--------------------------------------------------------------------------
Stats for: threshold
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
threshold        355324491   190574   409      72276       1864.00     3625.00     1860.00

The first part has the totals for all keywords. After this the stats are broken down per buffer type.

Part of this work was sponsored by Emerging Threats.

Fixing “error: m4_defn: undefined macro: _m4_divert_diversion”

Ran into a problem with autotools today, thought I’d share my solution. First, the error only happened on an old system:

$ bash autogen.sh 
Found libtoolize
Remember to add `AC_PROG_LIBTOOL' to `configure.ac'.
libtoolize: `config.guess' exists: use `--force' to overwrite
libtoolize: `config.sub' exists: use `--force' to overwrite
libtoolize: `ltmain.sh' exists: use `--force' to overwrite
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal --force -I m4
configure.ac:1: error: m4_defn: undefined macro: _m4_divert_diversion
configure.ac:1: the top level
autom4te: /usr/bin/m4 failed with exit status: 1
aclocal: autom4te failed with exit status: 1
autoreconf: aclocal failed with exit status: 1

The autogen.sh file is this one: https://github.com/inliniac/suricata/blob/master/autogen.sh

First lines of configure.ac:

AC_CONFIG_HEADERS([config.h])
AC_INIT(suricata, 2.0dev)
AC_CONFIG_SRCDIR([src/suricata.c])
AC_CONFIG_MACRO_DIR(m4)

Solution? Move AC_CONFIG_HEADERS to below AC_INIT, like so:

AC_INIT(suricata, 2.0dev)
AC_CONFIG_HEADERS([config.h])
AC_CONFIG_SRCDIR([src/suricata.c])
AC_CONFIG_MACRO_DIR(m4)

More on Suricata lua flowints

This morning I added flowint lua functions for incrementing and decrementing flowints. From the commit:

Add flowint lua functions for incrementing and decrementing flowints.

First use creates the var and inits to 0. So a call:

    a = ScFlowintIncr(0)

Results in a == 1.

If the var reached UINT_MAX (2^32), it’s not further incremented. If the
var reaches 0 it’s not decremented further.

Calling ScFlowintDecr on a uninitialized var will init it to 0.

Example script:

    function init (args)
        local needs = {}
        needs["http.request_headers"] = tostring(true)
        needs["flowint"] = {"cnt_incr"}
        return needs
    end

    function match(args)
        a = ScFlowintIncr(0);
        if a == 23 then
            return 1
        end

        return 0
    end
    return 0

This script matches the 23rd time it’s invoked on a flow.

Compared to yesterday’s flowint script and the earlier flowvar based counting script, this performs better:

   Num      Rule         Gid      Rev      Ticks        %      Checks   Matches  Max Ticks   Avg Ticks   Avg Match   Avg No Match
  -------- ------------ -------- -------- ------------ ------ -------- -------- ----------- ----------- ----------- -------------- 
  1        1            1        0        2434188332   59.71  82249    795      711777      29595.35    7683.20     29809.22   
  2        2            1        0        1015328580   24.91  82249    795      154398      12344.57    3768.66     12428.27   
  3        3            1        0        626858067    15.38  82249    795      160731      7621.47     3439.91     7662.28    

The rules:

alert http any any -> any any (msg:"LUAJIT HTTP flowvar match"; luajit:lua_flowvar_cnt.lua; flow:to_server; sid:1;)
alert http any any -> any any (msg:"LUAJIT HTTP flowint match"; luajit:lua_flowint_cnt.lua; flow:to_server; sid:2;)
alert http any any -> any any (msg:"LUAJIT HTTP flowint incr match"; luajit:lua_flowint_incr_cnt.lua; flow:to_server; sid:3;)

Please comment, discuss, review etc on the oisf-devel list.

Suricata Lua scripting flowint access

A few days ago I wrote about my Emerging Threats sponsored work to support flowvars from Lua scripts in Suricata.

Today, I updated that support. Flowvar ‘sets’ are now real time. This was needed to fix some issues where a script was invoked multiple times in single rule, which can happen with some buffers, like HTTP headers.

Also, I implemented flowint support. Flowints in Suricata are integers stored in the flow context.

Example script:

function init (args)
    local needs = {}
    needs["http.request_headers"] = tostring(true)
    needs["flowint"] = {"cnt"}
    return needs
end

function match(args)
    a = ScFlowintGet(0);
    if a then
        ScFlowintSet(0, a + 1)
    else
        ScFlowintSet(0, 1)
    end 
        
    a = ScFlowintGet(0);
    if a == 23 then
        return 1
    end 
    
    return 0
end 

return 0

It does the same thing as this flowvar script:

function init (args)
    local needs = {}
    needs["http.request_headers"] = tostring(true)
    needs["flowvar"] = {"cnt"}
    return needs
end

function match(args)
    a = ScFlowvarGet(0);
    if a then
        a = tostring(tonumber(a)+1)
        ScFlowvarSet(0, a, #a)
    else
        a = tostring(1)
        ScFlowvarSet(0, a, #a)
    end 
    
    if tonumber(a) == 23 then
        return 1
    end
    
    return 0
end

return 0

Only, at about half the cost:

   Num      Rule         Gid      Rev      Ticks        %      Checks   Matches  Max Ticks   Avg Ticks   Avg Match   Avg No Match
  -------- ------------ -------- -------- ------------ ------ -------- -------- ----------- ----------- ----------- -------------- 
  1        1            1        0        2392221879   70.56  82249    795      834993      29085.12    6964.14     29301.02   
  2        2            1        0        998297994    29.44  82249    795      483810      12137.51    4019.44     12216.74   

Suricata: Handling of multiple different SYN/ACKs

synackWhen processing the TCP 3 way handshake (3whs), Suricata’s TCP stream engine will closely follow the setup of a TCP connection to make sure the rest of the session can be tracked and reassembled properly. Retransmissions of SYN/ACKs are silently accepted, unless they are different somehow. If the SEQ or ACK values are different they are considered wrong and events are set. The stream events rules will match on this.

I ran into some cases where not the initial SYN/ACK was used by the client, but instead a later one. Suricata however, had accepted the initial SYN/ACK. The result was that every packet from that point was rejected by the stream engine. A 67 packet pcap resulting in 64 stream events.

If people have the stream events enabled _and_ pay attention to them, a noisy session like this should certainly get their attention. However, many people disable the stream events, or choose to ignore them, so a better solution is necessary.

Analysis

In this case the curious thing is that the extra SYN/ACK(s) have different properties: the sequence number is different. As the SYN/ACKs sequence number is used as “initial sequence number” (ISN) in the “to client” direction, it’s crucial to track it correctly. Failing to do so, Suricata will loose track of the stream, causing reassembly to fail. This could lead to missed alerts.

Whats happening on the wire:

TCP SSN 1:

-> SYN: SEQ 10
<- SYN/ACK 1: ACK 11, SEQ 100
<- SYN/ACK 2: ACK 11, SEQ 1000
-> ACK: SEQ 11, ACK 101

TCP SSN 2:

-> SYN: SEQ 10
<- SYN/ACK 1: ACK 11, SEQ 100
<- SYN/ACK 2: ACK 11, SEQ 1000
-> ACK: SEQ 11, ACK 1001

It’s clear that in SSN 1 the client ACKs the first SYN/ACK while in SSN 2 the 2nd SYN/ACK is ACK’d. It’s likely that the first SYN/ACK was lost before it reached the client. Suricata accepts the first though, and rejects any others that are not the same.

Solution

The solution I’ve been working on is to delay judgement on the extra SYN/ACKs until Suricata sees the ACK that completes the 3whs. At that point Suricata knows what the client accepted, and which SYN/ACKs were either ignored, or never received.

Logic in pseudo code:

Normal SYN/ACK coming in:

    UpdateState(p);
    ssn->state = TCP_SYN_RECV;

Extra SYN/ACK packets:

    if (p != ssn) {
        QueueState(p);

On receiving the ACK that completes the 3whs:

    if (ssn->queue_len) {
        q = QueueFindState(p);
        if (q)
            UpdateState(q);
    }
    UpdateState(p);
    ssn->state = TCP_ESTABLISHED;

So when receiving the ACK, Suricata first searches for the proper SYN/ACK on the list. If it’s not found, the ACK will be processed normally, which means it’s checked against the original SYN/ACK. If Suricata did have a queued state, it will first apply it to the SSN. Then the ACK will be processed normally, so that is can complete the 3whs and move the state to ESTABLISHED.

Limitations

Queuing these states takes some memory, and for this reason there is a limit to the number each SSN will accept. This is configurable through a new stream option:

stream:
  max-synack-queued: 5

It defaults to 5. I’ve seen a few (valid) hits against a few terrabytes of traffic, so I think the default is reasonably safe. An event is being set if the limit is exceeded. It can be matched using a stream-event rule:

  alert tcp any any -> any any (msg:"SURICATA STREAM 3way handshake \
      excessive different SYN/ACKs"; stream-event:3whs_synack_flood; \
      sid:2210055; rev:1;)

Performance

This functionality doesn’t affect the regular “fast path” except for a small check to see if we have queued states. However, if the queue list is being used Suricata enters a slow path. Currently this involves an memory allocation per stored queue. It may be interesting to consider using pools here, although a single global pool might be ineffecient. In such a case a lock would have to be used and this might lead to contention, especially in a case where Suricata would be flooded. Per thread pools (519, 520, 521) may be best here.

IPS mode

SYN/ACKs that exceed the limit are dropped if stream.inline is enabled as is the case with all packets that are considered to be bad in some way.

Code

The code is now part of the git master through commit 4c6463f3784f533a07679589dab713096137a439. Feedback welcome through our oisf-devel list.