Monthly Archives: July 2012

Suricata development training

Posted on by

We’re considering to offer a Suricata development training day around the next OISF brainstorm meeting. That would be in Amsterdam around the RAID conference, in early September.

Topics we could cover:

– code/development overview
– create/extend detect module
– create/extend output module
– app layer module
– proto detection
– …

The training would probably be free as it’s an excercise for us as well, so we’d just want honest feedback in return 🙂

Nothing is set in stone at this point, but I wanted to throw the idea around already. If you’re interested in joining this session, please let us know! If there is enough interest we may just make this happen!

Suricata on Myricom capture cards

Posted on by

Myricom and OISF just announced that Myricom joined to OISF consortium to support the development of Suricata. The good folks at Myricom already sent me one of their cards earlier. In this post I’ll describe how you can use these cards already, even though Suricata doesn’t have native Myricom support yet. So in this guide I’ll describe using the Myricom libpcap support.

Getting started

I’m going to assume you installed the card properly, installed the Sniffer driver and made sure that all works. Make sure that in your dmesg you see that the card is in sniffer mode:

[ 2102.860241] myri_snf INFO: eth4: Link0 is UP
[ 2101.341965] myri_snf INFO: eth5: Link0 is UP

I have installed the Myricom runtime and libraries in /opt/snf

Compile Suricata against Myricom’s libpcap:

./configure --with-libpcap-includes=/opt/snf/include/ --with-libpcap-libraries=/opt/snf/lib/ --prefix=/usr --sysconfdir=/etc --localstatedir=/var
make
sudo make install

Next, configure the amount of ringbuffers. I’m going to work with 8 here, as my quad core + hyper threading has 8 logical CPU’s.

pcap:
  - interface: eth5
    threads: 8
    buffer-size: 512kb
    checksum-checks: no

The 8 threads setting makes Suricata create 8 reader threads for eth5. The Myricom driver makes sure each of those is attached to it’s own ringbuffer.

Then start Suricata as follows:
SNF_NUM_RINGS=8 SNF_FLAGS=0x1 suricata -c suricata.yaml -i eth5 --runmode=workers

If you want 16 ringbuffers, update the “threads” variable in your yaml to 16 and start Suricata:
SNF_NUM_RINGS=16 SNF_FLAGS=0x1 suricata -c suricata.yaml -i eth5 --runmode=workers

It looks like you can use any number of ringbuffers, so not limited to a power of 2 for example.

Example with CPU affinity

You can also use Suricata’s built in CPU affinity settings to assign a worker to a cpu/core. In this example I’ll create 7 worker threads that will each run on their own logical CPU. The remaining CPU can then be used by the management threads, most importantly the flow manager.

max-pending-packets: 8192

detect-engine:
  - sgh-mpm-context: full

mpm-algo: ac-bs

threading:
  set-cpu-affinity: yes
  cpu-affinity:
    - management-cpu-set:
      cpu: [ "0" ]
    - detect-cpu-set:
      cpu: [ "1-7" ]
      mode: "exclusive"
      prio:
        default: "high"

pcap:
  - interface: eth5
    buffer-size: 512kb
    threads: 7
    checksum-checks: no

Then start Suricata with:
SNF_NUM_RINGS=7 SNF_FLAGS=0x1 suricata -c suricata.yaml -i eth5 --runmode=workers

This configuration will reserve cpu0 for the management threads and will assign a worker thread (and thus a ringbuffer) to cpu1 to cpu7. Note that I added a few more performance tricks to it. This config with 7 cpu pinned threads appears to be a little faster than the case where CPU affinity is not used.

Myricom has a nice traffic replay tool as well. This replays a pcap at 1Gbps:
snf_replay -r 1.0 -i 10 -p1 /path/to/pcap

Final remarks

The Myricom card already works nicely with Suricata. Because of the way their libpcap code works, we can already use the ringbuffers feature of the card. Myricom does also offer a native API. Later this year, together with Myricom, we’ll be looking into adding support for it.

Suricata http_user_agent vs http_header

Posted on by

One of the new features in Suricata 1.3 is a new content modifier called http_user_agent. This allows rule writers to match on the User-Agent header in HTTP requests more efficiently. The new keyword is documented in the OISF wiki. In this post, I’ll show it’s efficiency with two examples.

Example 1: rarely matching UA

Consider a signature where the match if on a part of the UA that is very rare, so not part of regular User Agents. In my example “abc”.

The signature looks like this:
alert http any any -> any any (msg:"User-Agent abc http_header"; content:"User-Agent: "; http_header; nocase; content:"abc"; http_header; distance:0; pcre:"/User-Agent:[^\n]*abc/iH"; sid:1; rev:1;)

The http_user_agent variant looks much simpler:
alert http any any -> any any (msg:"User-Agent abc http_user_agent"; content:"abc"; http_user_agent; sid:2; rev:1;)

Now when running this against a pcap with over 12.500 HTTP requests, neither signature matched. However, signature 1 was inspected 209752 times! This high number is because the request headers are inspected one-by-one. Signature 2 wasn’t inspected at all, as it never made it past the multi pattern matching stage (mpm).

When looking at pcap runtime, running with only the http_user_agent version is about 10% faster.

Example 2: commonly matching UA

So, what if we want to match on something that is quite common? In other words, the signature will have frequent matches?

First, the http_header signature:
alert http any any -> any any (msg:"User-Agent MSIE 6 http_header"; content:"User-Agent: "; http_header; nocase; content:"MSIE 6"; http_header; distance:0; pcre:"/User-Agent:[^\n]*MSIE 6/iH"; sid:3; rev:1;)
The http_user_agent variant:
alert http any any -> any any (msg:"User-Agent MSIE 6 http_user_agent"; content:"MSIE 6"; http_user_agent; sid:4; rev:1;)

In this case both signatures do match, just over 10.000 times even. The stats look like this:

Each of the inspections of signature 4, the http_user_agent variant, is actually a match. This makes sense as we look for a simple string and the mpm will only invoke the signature if that string is found. It’s clear that the http_header variant takes way more resources. Here too, when looking at pcap runtime, running with only the http_user_agent version is approximately 10% faster.

Final remarks

It’s quite clear that the http_user_agent keyword is much more efficient that inspecting all the HTTP headers. But other than efficiency, the http_user_agent also allows for much easier to read rules.

The Emerging Threats project will likely fork their Suricata ruleset for 1.3 (see this blog post). Even though this will be a significant effort on their side, it’s pretty clear to me the performance effect will be noticeable!

Suricata 1.3 released

Posted on by

Today, almost half a year after the last “stable” release, we released Suricata 1.3. I think this release is a big step forward with regard to maturity of Suricata. Performance and scalability have been much improved, just like accuracy and stability.

The official announcement can be found on the OISF site

In the last 6 months a lot of code has been changed:

384 files changed, 44332 insertions(+), 18478 deletions(-)

These changes have been made by 11 committers, only four of which were paid by OISF. The others were either developers from supporting vendors or great community members. I’d like to thank everyone for their contribution!

With the 1.3 release, for some people work only just started. I think this would be an ideal time for the Emerging Threats project to fork their Suricata ruleset. The new set for 1.3 could then start taking advantage of features like http_user_agent, file_data, file keywords, tls/ssl keywords, etc. One of the new features in 1.3, the rule analyzer, should be really helpful for the rule writer folks.

Looking towards the future, we’re planning for some nice new features and improvement. First, the TLS/SSL handling will be further improved. The guys are working on certificate fingerprint matching, storing certs to disk and more. We’ll also continue to improve our IPv6 support. Of course, performance work is always on our agenda, so also for the time to come. See our roadmap here.

Finally, if you’re interested discussing the roadmap with us in person, the RAID 2012 conference in Amsterdam next fall is a good opportunity. Most of the team will be present.