Tag Archives: Matt Jonkman

Closing in on Suricata 1.4

Posted on by

I just made Suricata 1.4rc1 available with some pretty exciting features: unix socket mode and IP reputation.

Unix socket

First of all, Eric Leblond’s work on the Unix socket was merged. The unix socket work consists of two parts. The unix socket protocol implementation and a new runmode.

The protocol implementation is based on JSON messages over unix socket. Eric will be fully documenting it soon. Currently the commands are limited to shutting down and getting some basic stats. This part isn’t very exciting yet, but the groundwork for many future extensions has been laid.

The part that is exciting right now, is the unix socket runmode. That this does is start Suricata with all the rules and such, and then it waits for commands on the unix socket. Then the commands will be a pcap filename – log directory pair. This pcap will then be inspected against the rules and the logs go into the log directory supplied. As this can be easily scripted (a python script is provided), it’s a very fast way to test your pcap collections, as the overhead of starting and stopping is skipped.

This may initialy appeal mostly for those of you doing sandnetting and malware analysis, where tens of thousands of pcaps and automatically processed every hour or day, I think this could grow into a feature for a wider audience as well. For example, I could see use in Sguil or Snorby, or pretty much every event manager with full packet capture support, adding an option to scan a pcap associated with an event again. Maybe against _all_ rules, instead of the tuned set running on the live sensors. Maybe you can re-inspect old sessions against the current rules this way to find hits on attacks that were 0-days at the time, etc.

I think there could be many possibilities.

IP Reputation

A slightly more polished version of the code I discussed here is now available in this release. It’s one of those things where it will be very interesting to see how people will put it to use.

Matt Jonkman just wrote some of his ideas to the Emerging Threats mailing list: one of the ideas Matt wrote about is to amend weak rules with reputation data. So if you have a signature that is phrone to false positives, you probably disable it currently. But what if you combine it with reputation data? If the weak rule fires on a sketchy ip, it may be a more reliable alert.

We’ll see how this plays out.

1.4 final

We’re hoping that if nothing big happens, we can do a mid-December 1.4 final release. So please consider running this new release. It’s running very stable on quite a number of places, ISP networks, Lab networks, home networks, sandnetting networks, etc. But we need much more testing to find issues and/or gain confidence that we have found the most important issues. Thanks for helping out!

IP Reputation in Suricata

Posted on by

Disclaimer: this work was sponsored by Emerging Threats Pro.

One thing we’ve been talking about for many years at OISF is IP Reputation. The basic idea is that many organizations have information about specific IP-addresses. This information may be that a host is infected, acts as a spam relay or many other things. We’ve always thought it might be useful to apply this info to the IDS directly.

In the last weeks I’ve developed code to load IP reputation information into Suricata. This code is now part of the Suricata git master, so it’s available to all.

The work consisted of 3 main parts: data load, internal data structures and a rule keyword.

Data loading

The data I worked with was provided by Emerging Threats Pro. The data format is very simple. Two types of CSV files, one to define a mapping between category names and id’s and the other to define the scores for hosts in the categories.

The data formats are documented here: IP Reputation Format.

Internal Data Structures

To store the data in memory I hooked into our “Hosts” API. The Hosts API is a hash table like the Flow table that can be used to store data per host. It’s in use for Tagging and Thresholding. I added storage for IP Reputation to it.

Rule keyword

A new rule keyword to match on the reputation data was introduced: “iprep”. The keyword allows a rule to match on a specific category. Example:

alert ... (flow:to_server; iprep:src,Bot,>,10;)

This will generate an alert if the SRC IP of the host talking to a server is known to have a score of >10 in the “Bot” category.

The keyword is compatible to Suricata’s concept of “IP-only” rules. These are rules that do not inspect packet content or flow state and can thus be inspected once per flow direction instead of for each packet.

Speed

I’ve been playing with data sets of up to a million entries. Loading it takes hardly any time and I’m confident larger numbers will work just fine. The host table just needs bigger memcaps and hash sizes.

At runtime, the speed depends mostly on the rules. A pure “iprep” rule is quite expensive when not IP-only, although this is mostly due to the frequency of the checks. Such rules will be checked against large numbers of packets.

When created as a IP-only rule, things change. Such rules are checked only once per flow direction, so overhead appears to be minimal in this case.

Data

The data I used from Emerging Threats Pro is not available for free, so for those who want to test creating your own data is required right now. Matt Jonkman from Emerging Threats Pro will make a free feed available within a few weeks though. Of course you could also get the paid data from Emerging Threats Pro. 🙂

Update 29/11/2012

This feature is part of the just released 1.4rc1 version, please help us test it!

DeepSec

Posted on by

Last month I attended the DeepSec conference in Vienna. I enjoyed it a great deal. It was good to be back in Vienna. Had a few good meetings with my friend Adi with who I work on the Vuurmuur project.

I assisted Matt Jonkman in his Snort Signature writing class. We had a nice group of people and using the Emerging Threats SandNet we could deal with pretty interesting samples to write signatures for. Even though my expertise is more on the code level of Snort I felt I could still contribute something to the sessions.

On the last day Matt and I did the first Open Infosec Foundation brainstorm session. I think it was very useful and the crowd was very responsive. After this encouraging experience we are planning to attend more conferences to do similar sessions. Suggestions about which conferences would be interesting (and why) are very welcome!

First SidReporter statistics available

Posted on by

Matt Jonkman just announced that the first stats of SidReporter are available here. Matt writes:

These will become more interesting the more sites we have reporting, so please consider running  the client. It’s painless, anonymous, and will contribute to us greatly improving the signature base we all use.

It will be interesting to see what data this can bring us. Congrats Matt!

First OISF brainstorming session on Deepsec

Posted on by

Next November I will be attending Deepsec in Vienna. Matt Jonkman is giving a workshop there and I will be helping/assisting him with it, it’s called ‘Protocol Analysis for Writing Snort Signatures’. If you’re interested, sign up for it! While we are there we will also host the first brainstorming session for OISF. The idea is to get together with everyone thats interested and talk about how our next generation IDS/IPS should look like. But it’s not just about the technology, we also seek input about how to organize the project, about licensing, etc. So if you’re at Deepsec and got some time to spare, be sure to join us in the brainstorming session!

SidReporter beta2 released

Posted on by

A little over a week ago the second beta of the SidReporter from Emerging Threats was released (see http://www.emergingthreats.net/content/view/95/1/). I’ve been working with Matt Jonkman to setup this new project at Emerging Threats, mostly in writing the reporter scripts. I think it’s an exciting new project that could provide the community with great information. As Matt wrote on the initial announcement:

“As mentioned a few weeks ago, we’ve been working to bring out tool to anonymously report IDS/IPS hits. Similar to DShield’s firewall log reporting, we believe we can make some incredible data inferences with this information, as well as help improve the quality of our signatures while giving us all feedback to tune our rulesets.

But that’s just the start. As with DShield’s data, I think we’ll run into benefits to the community that we can’t even imagine until we start to look at the data.”

The next step for the reporter is adding support for getting the events from Sguil. Expect to see that soon!

New Snortsam patch for Snort 2.8.0.1

Posted on by

Matt Jonkman of Emerging Threats asked me to have a look at the existing Snortsam 2.8.0.1 patch as people were continuing to report problems with it. I updated it to compile without compiler warnings, build cleanly with debugging enabled, build cleanly with Snort’s IPv6 support enabled and added a check so it won’t act on alerts in IPv6 packets since the Snortsam framework does not support IPv6. Finally I removed the patch script so it’s provided as a ‘normal’ diff. Here is the patch: http://www.inliniac.net/files/snortsam-2.8.0.1.diff

Here are the instructions for getting your Snort 2.8.0.1 source patched:

Make sure you have a clean Snort 2.8.0.1 tree, then patch it:

cd snort-2.8.0.1
patch -p1 < ../snortsam-2.8.0.1.diff

Next, run ‘autojunk.sh’ to update the build system (you need to have libtoolize, aclocal, autoheader, autoconf and automake installed). After this, configure and build Snort normally:

./configure <your configure options>
make
make install

Thats it.

Thanks to Matt Jonkman of Emerging Threats for paying me to do this and CunningPike for doing the first iterations of the patch!

Matt Jonkman leaves Bleeding Edge

Posted on by

Matt Jonkman is stepping out of the Bleeding Edge project. He announced this here. Apparently Sensory Networks, one of the sponsors of the project, now owns it. It will be interesting to see if they will continue it, and if so, how. Honestly, I’m a bit skeptical, since to my knowledge not many Sensory people are directly involved at this moment. Still I believe Sensory consists of good people. I did a contract job for them about a year ago, and enjoyed working with them.

I think I speak for many if I say “Thanks” for all the hard work Jonkman has done for Bleeding, and I really look forward to new projects he will start or get involved in! Thanks Matt!