Disabling Threading in Tcl8.5 in Debian

sguil_logo_h

I’ve been spending the holidays to upgrade some of my own servers. One of them is the Sguil server I use. Until now it ran Debian Squeeze. On Debian Squeeze you could use tcl8.3, which has threading disabled. For Sguil tcl threading needs to be disabled:

ERROR: This version of tcl was compile with threading enabled. Sguil is NOT compatible with threading.

This is a compile time option in TCL, and the Debian Wheezy packages have it enabled by default. Here are the steps to create your own tcl deb with threading disabled:

# apt-get install dpkg-dev
# apt-get install devscripts

Get the tcl8.5 source package and build deps:

# apt-get source tcl8.5
# apt-get build-dep tcl8.5
# cd tcl8.5-8.5.11/

Next, edit the debian/rules file to disable threading. Remove the line:

                      --enable-threads \

Then, build the package:

# debuild -us -uc

And finally install the package:

# cd ..
# dpkg -i ../tcl8.5_8.5.11-2_amd64.deb

I followed this guide here at Debian Administration. It has some more detail on rebuilding debs.

Suricata Development Update

SuricataWith the holidays approaching and the 1.4.7 and 2.0beta2 releases out, I thought it was a good moment for some reflection on how development is going.

I feel things are going very well. It’s great to work with a group that approaches this project from different angles. OISF has budget have people work on overall features, quality and support. Next to that, our consortium supporters help develop the project: Tilera’s Ken Steele is working on the Tile hardware support, doing lots optimizations. Many of which benefit performance and overall quality for the whole project. Tom Decanio of Npulse is doing great work on the output side, unifying the outputs to be machine readable. Jason Ish of Emulex/Endace is helping out the configuration API, defrag, etc. Others, both from the larger community and our consortium, are helping as well.

QA

At our last meetup in Luxembourg, we’ve spend quite a bit of time discussing how we can improve the quality of Suricata. Since then, we’ve been working hard to add better and more regression and quality testing.

We’ve been using a Buildbot setup for some time now, where on a number of platforms we do basic build testing. First, this was done only against the git master(s). Eric has then created a new method using a script call prscript. It’s purpose is to push a git branch to our buildbot _before_ it’s even considered for inclusion.

Recently, with cooperation of Emerging Threats, we’ve been extending this setup to include a large set of rule+pcap matches that are checked against each commit. This too is part of the pre-include QA process.

There are many more plans to extend this setup further. I’ve set up a private buildbot instance to serve as a staging area. Things we’ll be adding soon:
– valgrind testing
– DrMemory testing
– clang/scan-build
– cppcheck

Ideally, each of those tools would report 0 issues, but thats hard in practice. Sometimes there are false positives. Most tools support some form of suppression, so one of the tasks is to create those.

We’ve spend some time updating our documents regarding contributing to our code base. Please take a moment to a general contribution page, aimed at devs new to the project.

Next to this, this document describes quality requirements for our code, commits and pull requests.

Suricata 2.0

Our roadmap shows a late January 2.0 final release. It might slip a little bit, as we have a few larger changes to make:
– a logging API rewrite is in progress
“united” output, an all JSON log method written by Tom Decanio of Npulse [5]
app-layer API cleanup and update that Anoop is working on [6]

Wrapping up, I think 2013 was a very good year for Suricata. 2014 will hopefully be even better. We will be announcing some new support soon, are improving our training curicullum and will just be working hard to make Suricata better.

But first, the holidays. Cheers!

Suricata profiling per keyword

Last week I’ve added some more profiling options to Suricata. It’s part of the current git master. It’s enabled only when --enable-profiling and then through the suricata.yaml:

profiling:
  # per keyword profiling
  keywords:
    enabled: yes
    filename: keyword_perf.log
    append: yes

This will output a table similar to below:

--------------------------------------------------------------------------
Date: 11/7/2013 -- 15:13:11
--------------------------------------------------------------------------
Stats for: total
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
threshold        355324491   190574   409      72276       1864.00     3625.00     1860.00    
content          1274592063  534328   196738   312321      2385.00     2424.00     2362.00    
pcre             56626031    11149    824      254562      5079.00     12234.00    4507.00    
byte_test        153287955   128254   32109    67989       1195.00     1658.00     1040.00    
byte_jump        3676404     2041     2041     15939       1801.00     1801.00     0.00       
flow             38276182    22842    22842    63987       1675.00     1675.00     0.00       
isdataat         580764      558      556      2427        1040.00     1040.00     1017.00    
dsize            2212029     2062     2061     3711        1072.00     1072.00     789.00     
flowbits         1677209     874      870      9873        1919.00     1923.00     884.00     
itype            1653        2        1        1386        826.00      267.00      1386.00    
icode            27383781    93827    2        25545       291.00      1021.00     291.00     
flags            192751968   245519   189709   255639      785.00      753.00      892.00     
urilen           6149297     6142     1099     28299       1001.00     1395.00     915.00     
byte_extract     143091      78       78       7743        1834.00     1834.00     0.00       
--------------------------------------------------------------------------
Stats for: packet
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
flow             38276182    22842    22842    63987       1675.00     1675.00     0.00       
dsize            2212029     2062     2061     3711        1072.00     1072.00     789.00     
flowbits         351171      294      290      5526        1194.00     1198.00     884.00     
itype            1653        2        1        1386        826.00      267.00      1386.00    
icode            27383781    93827    2        25545       291.00      1021.00     291.00     
flags            192751968   245519   189709   255639      785.00      753.00      892.00     
--------------------------------------------------------------------------
Stats for: packet/stream payload
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
content          1203990910  512902   183628   312321      2347.00     2365.00     2337.00    
pcre             28087301    6598     54       254562      4256.00     12279.00    4190.00    
byte_test        153287955   128254   32109    67989       1195.00     1658.00     1040.00    
byte_jump        3676404     2041     2041     15939       1801.00     1801.00     0.00       
isdataat         578172      556      554      2427        1039.00     1039.00     1017.00    
byte_extract     143091      78       78       7743        1834.00     1834.00     0.00       
--------------------------------------------------------------------------
Stats for: http uri
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
content          44775802    13102    8351     60993       3417.00     3257.00     3698.00    
pcre             18284421    3646     97       61338       5014.00     8916.00     4908.00    
isdataat         2592        2        2        1725        1296.00     1296.00     0.00       
urilen           6149297     6142     1099     28299       1001.00     1395.00     915.00     
--------------------------------------------------------------------------
Stats for: http raw uri
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
pcre             9534        2        0        4953        4767.00     0.00        4767.00    
--------------------------------------------------------------------------
Stats for: http client body
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
content          1556904     441      181      58476       3530.00     2874.00     3986.00    
pcre             63924       6        6        17358       10654.00    10654.00    0.00       
--------------------------------------------------------------------------
Stats for: http headers
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
content          23688244    7631     4348     31098       3104.00     3311.00     2829.00    
pcre             9998970     859      667      71904       11640.00    12727.00    7862.00    
--------------------------------------------------------------------------
Stats for: http stat code
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
content          80052       39       20       3699        2052.00     2199.00     1898.00    
--------------------------------------------------------------------------
Stats for: http method
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
content          476334      203      201      27240       2346.00     2351.00     1846.00    
--------------------------------------------------------------------------
Stats for: http cookie
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
content          23817       10       9        2763        2381.00     2384.00     2358.00    
pcre             181881      38       0        13095       4786.00     0.00        4786.00    
--------------------------------------------------------------------------
Stats for: post-match
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
flowbits         1326038     580      580      9873        2286.00     2286.00     0.00       
--------------------------------------------------------------------------
Stats for: threshold
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
threshold        355324491   190574   409      72276       1864.00     3625.00     1860.00

The first part has the totals for all keywords. After this the stats are broken down per buffer type.

Part of this work was sponsored by Emerging Threats.

Fixing “error: m4_defn: undefined macro: _m4_divert_diversion”

Ran into a problem with autotools today, thought I’d share my solution. First, the error only happened on an old system:

$ bash autogen.sh 
Found libtoolize
Remember to add `AC_PROG_LIBTOOL' to `configure.ac'.
libtoolize: `config.guess' exists: use `--force' to overwrite
libtoolize: `config.sub' exists: use `--force' to overwrite
libtoolize: `ltmain.sh' exists: use `--force' to overwrite
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal --force -I m4
configure.ac:1: error: m4_defn: undefined macro: _m4_divert_diversion
configure.ac:1: the top level
autom4te: /usr/bin/m4 failed with exit status: 1
aclocal: autom4te failed with exit status: 1
autoreconf: aclocal failed with exit status: 1

The autogen.sh file is this one: https://github.com/inliniac/suricata/blob/master/autogen.sh

First lines of configure.ac:

AC_CONFIG_HEADERS([config.h])
AC_INIT(suricata, 2.0dev)
AC_CONFIG_SRCDIR([src/suricata.c])
AC_CONFIG_MACRO_DIR(m4)

Solution? Move AC_CONFIG_HEADERS to below AC_INIT, like so:

AC_INIT(suricata, 2.0dev)
AC_CONFIG_HEADERS([config.h])
AC_CONFIG_SRCDIR([src/suricata.c])
AC_CONFIG_MACRO_DIR(m4)

More on Suricata lua flowints

This morning I added flowint lua functions for incrementing and decrementing flowints. From the commit:

Add flowint lua functions for incrementing and decrementing flowints.

First use creates the var and inits to 0. So a call:

    a = ScFlowintIncr(0)

Results in a == 1.

If the var reached UINT_MAX (2^32), it’s not further incremented. If the
var reaches 0 it’s not decremented further.

Calling ScFlowintDecr on a uninitialized var will init it to 0.

Example script:

    function init (args)
        local needs = {}
        needs["http.request_headers"] = tostring(true)
        needs["flowint"] = {"cnt_incr"}
        return needs
    end

    function match(args)
        a = ScFlowintIncr(0);
        if a == 23 then
            return 1
        end

        return 0
    end
    return 0

This script matches the 23rd time it’s invoked on a flow.

Compared to yesterday’s flowint script and the earlier flowvar based counting script, this performs better:

   Num      Rule         Gid      Rev      Ticks        %      Checks   Matches  Max Ticks   Avg Ticks   Avg Match   Avg No Match
  -------- ------------ -------- -------- ------------ ------ -------- -------- ----------- ----------- ----------- -------------- 
  1        1            1        0        2434188332   59.71  82249    795      711777      29595.35    7683.20     29809.22   
  2        2            1        0        1015328580   24.91  82249    795      154398      12344.57    3768.66     12428.27   
  3        3            1        0        626858067    15.38  82249    795      160731      7621.47     3439.91     7662.28    

The rules:

alert http any any -> any any (msg:"LUAJIT HTTP flowvar match"; luajit:lua_flowvar_cnt.lua; flow:to_server; sid:1;)
alert http any any -> any any (msg:"LUAJIT HTTP flowint match"; luajit:lua_flowint_cnt.lua; flow:to_server; sid:2;)
alert http any any -> any any (msg:"LUAJIT HTTP flowint incr match"; luajit:lua_flowint_incr_cnt.lua; flow:to_server; sid:3;)

Please comment, discuss, review etc on the oisf-devel list.