RAID 2011 Thoughts

The last few days I’ve been at the Recent Advances in Intrusion Detection (RAID) conference in California. Overall it has been a very pleasant and interesting experience. The nice California weather was certainly helping a lot!

I’ve seen all talks and some were very interesting. However, being a Suricata IDS developer, I was not just interested in research for the hell of it, but I was actively scouting for ideas we could implement into Suricata. In this respect the conference was highly disappointing. Although with some of the talks I thought the idea was applicable in general security, like Erik Bosmans high speed memory tainting detection, I found nothing like that for NIDS.

Most inspiring part of the conference was spending an evening with Seth Hall, one of the Bro IDS engineers. Bro has a very different approach to inspecting the network than Suricata. Actually, I should say Suricata does it differently as Bro has been around much longer than Suricata. :) The conversation was all about sharing of ideas and experiences, and finding common grounds for actual cooperation.

A couple of notes from that conversation. First, Bro supports Unified2/Barnyard2 now, as input (so actually Barnyard2 can output to Bro). This means it can extend it’s analysis to include Suricata generated events. Second, we might try to have Suricata and Bro work together, where Suricata would be controlled by Brocolli. This way Bro could benefit from Suricata’s high speed signature matching engine, functionality Bro doesn’t have, and Suricata could benefit from Bro’s higher level understanding of the network. Finally, Bro’s binpack effort to define protocol parsers in a higher level language that can then be compiled into native code looks interesting as well. It would probably take quite a bit of changes to get this all going, but it might just be worth it.

Then there was the panel at the conference with Martin Roesch, Seth Hall and myself. A lot of people expected fireworks, but no such thing happened. Everyone was polite, respectful and friendly. It never really turned into a real discussion though, it was more a Q&A with the audience. Dominique Karg blogged about the panel here.

It was good to talk to Martin Roesch. The OISF – Sourcefire relation has definitely not started well, so it was good to have normal conversations and such. I offered Marty to work together, especially on SCADA detection. As was announced earlier, OISF will maintain the Digital Bond Quickdraw SCADA parsers and keywords, not only for Suricata, but also for Snort. Hopefully we can start a more constructive relationship on this topic, and elsewhere.

Some final thoughts on RAID. It was well organized and it was great to meet so many smart(er) people thinking about generally the same topics as I do. On the negative side I do feel disappointed over the apparent disconnect between the academic world and the more real world focused efforts like Suricata, Snort and tools like Streamdb, Sguil, Snortby, Squert, etc. But maybe I’m just lacking the vision to put the theory to practice.

The current tools out there may not be considered sufficient by everyone for every task. However, if RAID was a good benchmark, I fear we’ll have to settle for those for a while. Thats not necessarily a bad thing as fore-mentioned tools are under active development and continue to improve steadily.

On Suricata performance

Lots of fuzz in the media about Suricata’s performance versus Snort yesterday. Some claiming Suricata is much faster, others claiming Snort is much faster.

At this point I really don’t care much. What the Suricata development by the OISF has shown in my opinion is that we’ve managed to create a very promising new Open Source project out here. In little over a year, funded for about $600k by the US government and with heavy (and growing) industry support, we’ve produced a new IDS/IPS engine mostly compatible with Snort but build on a all new code base an incorporating some very interesting fresh ideas. We’re already seeing a community form around our project with a lot of support from that new community.

So about this performance fuzz. Who to believe? Is Suricata faster than Snort? Yes, no, ehhh, depends on how you look at it. Is Suricata faster than Snort on a single core cycle for cycle, tick for tick? No. It’s pretty clear we aren’t, I didn’t expect us to be either. But we scale. We’ve had reports of running on a 32 core box and scaling to use all cores. There Suricata is much faster. Like Martin Roesch wrote on the VRT blog one can set up Snort on a box to one have instance of Snort per core (or multiple per core). This is in fact the way many appliance builders get to high speeds with it. While this may be feasible for appliance builders, admins we talked to that run their own IDS/IPS think it’s a management nightmare.

As we’re a new project with a fresh codebase, there is going to be a lot of low hanging fruit in performance optimizations. I’ll give an example here. On a test pcap, with a reduced ruleset (about 10k rules), Suricata took about 400s to inspect. Then with a bigger ruleset (about 14k rules), it suddenly took 1600s! After a little bit of cache profiling it turned out that the part of the engine where the address part of a signature was inspected was horribly cache inefficient. In less than an afternoon I rewrote it to be more efficient. Result, the same test now completes in under 600s. This code is in the current git master and will be in 1.0.1.

My point here being that there will be lots of room for optimizations, and not just minor stuff. So far we’ve mostly focused on being accurate (we still have work to do here) and having the algorithms be correct. Hardly any tuning has been done. In our last OISF meeting we’ve gotten a few very interesting help offers for serious performance testing and tuning on some really big boxes, state of the art CUDA hardware, 10GBit labs, etc. So I expect a lot of progress in the months to follow.

It’s clear that we have work to do. What I’m really excited about is how fast that work is progressing, how much help we’re getting both from our brand new community and the industry, and the openness of our development process.

On a final note, during the development of this project we’ve found a lot of bugs and issues in other tools. Will Metcalf, who runs our QA, has been reporting many issues in Snort and VRT sigs to Sourcefire, in Emerging Threats sigs to the ET community. We’ve found bugs in other tools as well, for example in a neat library called libcap-ng. So everyone benefits from our work! :)

Ohloh

Ohloh is a pretty cool site for keeping track of projects and programmers. It’s an easy way to keep track of the development in a project and gives a nice indication of how actively it’s being developed. It has some social networkish features too, such as individual developers giving each other “kudos”.

The code analysis is pretty nice: it gives statistics on code base size, growth, comment ratio, languages used, etc. Per developer it tracks quite a few stats as well.

It also does a estimate of the cost of a project. For the Suricata project it currently estimates cost of 2.1 million USD. Actual cost are significantly less than that, less than half of that. So either we are severely underpaid or the calculation is off quite a bit :)

The per developer code statistics show that I’ve “touched” 131k lines of code out of 148k which confirms what I already knew: I need some vacation…

Anyway, check it out. Vuurmuur is on there, as are Snort and ModSecurity.

Oh by the way, Suricata 1.0 coming out tomorrow!

Checking out SourceForge’s Marketplace

I’ve registered myself as a seller of services on SourceForge’s Open Source Marketplace. I’ve done so offering software development services for the Snort, Snort_inline and Vuurmuur projects. I was wondering if anyone has any experience (good or bad) with the Marketplace system, either as a buyer or seller of services. Let me know!

Available for contract work

This year there will be a lot of work that needs to be done for the Open Infosec Foundation. And like I wrote a few days ago, a lot of work is already being done. However, most of it is unpaid at this time as it will be some months before our funding comes in. So at least until then I’m available and looking for contract work.

For the last two years I’ve been doing work as a contractor in the (open source) security field. My experience is mostly in coding in C and Perl, primarily on Snort and Snort_inline. Recently I created the (Perl language) SidReporter program for Emerging Threats. Areas I worked in: IPv6 IDS/IPS coding, signature writing, Web Application Firewalls, threading, bandwidth accounting, and more…

Checkout my LinkedIn profile for more info. My resume is available on request.

If you have some work or know someone that does, please let me know!

Snort_inline updated to 2.8.2.1 in SVN

This morning I updated our Snort_inline codebase with SourceFire’s just released 2.8.2.1 version. See the original changelogs here: 2.8.1, 2.8.2, 2.8.2.1.

Also Richard Bejtlich and Nr have good posts about the improvements of the last versions. See Richards post about a fixed frag3 vulnerability here and see Nr’s post here.

Please note that our SVN code has seen limited testing so far, so be careful! Please report any issues!

Snort_inline 2.8 status

A while ago I wrote about porting Snort_inline to 2.8.0.1. That worked well, however we are still trying to resolve some issues. Especially in stickydrop, that is just broken right now. Also, SourceFire released 2.8.0.2 last week, so we need to update to that too.

First however, I will be traveling to California this week. I will be meeting Will there, so I’ll try to get him to fix that damn code ;-)

Snort_inline updated to 2.8.0.1 in SVN

I’ve just committed an update to Snort_inline’s SVN. It brings it to the Snort 2.8.0.1 level. It supports both IPv4 and IPv6 on IPQ and NFQ. I have not been able to test IPFW on IPv6, so I don’t think that will work currently.

This update removes the libdnet dependency and replaces it with libnet 1.1. To be able to send ICMPv6 unreachable packets you will need the libnet 1.1 patch I wrote a while ago. You can find that here. Get the latest Snort_inline by checking out SVN:

svn co https://snort-inline.svn.sourceforge.net/svnroot/snort-inline/trunk

Consider the code to be of beta quality for now, so be careful with it. Please report any problems with it!

Again, a big thank you to NitroSecurity for funding this work!

Working on Snort_inline 2.8.0.1

The last week I’ve been working on bringing Snort_inline to the Snort 2.8.0.1 level, including it’s IPv6 support. I’m almost ready to commit it to SVN, there are just some issues I need to fix in the inline specific code. The code will get rid of libdnet and use libnet 1.1 for sending reset/reject packets for both IPv4 and IPv6. After committing I will start working on getting the IPv6 features I wrote for NitroSecurity into this tree. This includes more matches, tunnel decoding (including for example the freenet6 tunnel, etc). So stay tuned!

Libnet 1.1 IPv6 fixes and additions

Libnet is a cool packet crafting tool, used by Snort to send TCP reset packets and ICMP unreachable packets as part of active responses. Libnet 1.1 supports IPv6 which is what I needed for my work. After some reading and testing there were a few problems. First, while possible to send TCP reset packets, the packets didn’t have a correct checksum and debugging this with valgrind showed lots of memory errors. Second, ICMPv6 was only partly implemented. The libnet_build_* functions for it are missing. This is, by the way, quite a common picture. Many libraries and projects have some support for IPv6, but generally incomplete and less well tested.

For my work on a IPv6 enabled Snort_inline I’ve only fixed the checksum issue and added a libnet_build_icmpv6_unreach() function. The patch against libnet 1.1.3-RC-01 can be found here. It’s development was funded by the great people of NitroSecurity Inc., who are funding my work to bring IPv6 to Snort_inline. The work is not based on Sourcefire‘s recent IPv6 implementation, so it will be interesting to see if and how those codebases can be used to improve each other. The changes to Snort_inline will be made available as well later, WhenItsDone(tm) :) Like with the support for NFQueue, NitroSecurity gives back to the community, which I really appreciate!

The patch: http://www.inliniac.net/files/libnet-1.1.3-RC-01-ipv6.diff.gz