Tag Archives: iprep

Suricata 1.4 is out

Posted on by

About 5 months after 1.3 came out we’ve released 1.4, and we’ve been quite busy. Eric Leblond’s post here has all the stats and graphs. There are three big new features: unix socket, ip reputation and luajit. For each of these the same is true: it’s usesable now, but it’s the potential that we’re most excited about. Over the next months we’ll be extending each of those to be even more useful. We’re very much interested in ideas and feedback.

Performance obviously matters to many in the IDS world, and here too we have improved Suricata quite a bit again. We now have Suricata 1.4 running on a ISP 10gbit/s network on commodity hardware with a large ET ruleset. Of course, YMMV, but we’re definitely making a lot of progress here.

Sometimes the little things matter a lot as well. A minor new feature is that live “drop” stats are the the stats.log now:

capture.kernel_packets    | AFPacketem21              | 13640581
capture.kernel_drops      | AFPacketem21              | 442864
capture.kernel_packets    | AFPacketem22              | 7073228
capture.kernel_drops      | AFPacketem22              | 9449
capture.kernel_packets    | AFPacketem23              | 10528970
capture.kernel_drops      | AFPacketem23              | 148281
capture.kernel_packets    | AFPacketem24              | 7212584
capture.kernel_drops      | AFPacketem24              | 12643
capture.kernel_packets    | AFPacketem25              | 9763439
capture.kernel_drops      | AFPacketem25              | 17874
capture.kernel_packets    | AFPacketem26              | 10464106
capture.kernel_drops      | AFPacketem26              | 20378
capture.kernel_packets    | AFPacketem27              | 8869182
capture.kernel_drops      | AFPacketem27              | 18336
capture.kernel_packets    | AFPacketem28              | 7925045
capture.kernel_drops      | AFPacketem28              | 258168

This is supported for AF_PACKET, PF_RING and libpcap.

Last August we’ve added Suricata to github to make it easier to participate. Also, the code review tools associated with the pull requests are very useful. Github has been an unexpected success for us. At the time of writing there are 24 forks of Suricata on it, I’ve processed about 250 pull requests. The patches that have been submitted range from small fixes to full blown features, and more are on the way. I’m very grateful for these contributions and everyone’s patience with me.

Now that 1.4 is out, we’ll be taking it slow over the holidays. The team has been working like crazy, and everyone deserves a break. So the next weeks we’ll focus on further consolidation, fixing bugs that no doubt will pop up. Other than that, things will be slow. After the holidays we’ll start planning for the next milestone. Again, your ideas and contributions are very welcome! 🙂

Closing in on Suricata 1.4

Posted on by

I just made Suricata 1.4rc1 available with some pretty exciting features: unix socket mode and IP reputation.

Unix socket

First of all, Eric Leblond’s work on the Unix socket was merged. The unix socket work consists of two parts. The unix socket protocol implementation and a new runmode.

The protocol implementation is based on JSON messages over unix socket. Eric will be fully documenting it soon. Currently the commands are limited to shutting down and getting some basic stats. This part isn’t very exciting yet, but the groundwork for many future extensions has been laid.

The part that is exciting right now, is the unix socket runmode. That this does is start Suricata with all the rules and such, and then it waits for commands on the unix socket. Then the commands will be a pcap filename – log directory pair. This pcap will then be inspected against the rules and the logs go into the log directory supplied. As this can be easily scripted (a python script is provided), it’s a very fast way to test your pcap collections, as the overhead of starting and stopping is skipped.

This may initialy appeal mostly for those of you doing sandnetting and malware analysis, where tens of thousands of pcaps and automatically processed every hour or day, I think this could grow into a feature for a wider audience as well. For example, I could see use in Sguil or Snorby, or pretty much every event manager with full packet capture support, adding an option to scan a pcap associated with an event again. Maybe against _all_ rules, instead of the tuned set running on the live sensors. Maybe you can re-inspect old sessions against the current rules this way to find hits on attacks that were 0-days at the time, etc.

I think there could be many possibilities.

IP Reputation

A slightly more polished version of the code I discussed here is now available in this release. It’s one of those things where it will be very interesting to see how people will put it to use.

Matt Jonkman just wrote some of his ideas to the Emerging Threats mailing list: one of the ideas Matt wrote about is to amend weak rules with reputation data. So if you have a signature that is phrone to false positives, you probably disable it currently. But what if you combine it with reputation data? If the weak rule fires on a sketchy ip, it may be a more reliable alert.

We’ll see how this plays out.

1.4 final

We’re hoping that if nothing big happens, we can do a mid-December 1.4 final release. So please consider running this new release. It’s running very stable on quite a number of places, ISP networks, Lab networks, home networks, sandnetting networks, etc. But we need much more testing to find issues and/or gain confidence that we have found the most important issues. Thanks for helping out!

IP Reputation in Suricata

Posted on by

Disclaimer: this work was sponsored by Emerging Threats Pro.

One thing we’ve been talking about for many years at OISF is IP Reputation. The basic idea is that many organizations have information about specific IP-addresses. This information may be that a host is infected, acts as a spam relay or many other things. We’ve always thought it might be useful to apply this info to the IDS directly.

In the last weeks I’ve developed code to load IP reputation information into Suricata. This code is now part of the Suricata git master, so it’s available to all.

The work consisted of 3 main parts: data load, internal data structures and a rule keyword.

Data loading

The data I worked with was provided by Emerging Threats Pro. The data format is very simple. Two types of CSV files, one to define a mapping between category names and id’s and the other to define the scores for hosts in the categories.

The data formats are documented here: IP Reputation Format.

Internal Data Structures

To store the data in memory I hooked into our “Hosts” API. The Hosts API is a hash table like the Flow table that can be used to store data per host. It’s in use for Tagging and Thresholding. I added storage for IP Reputation to it.

Rule keyword

A new rule keyword to match on the reputation data was introduced: “iprep”. The keyword allows a rule to match on a specific category. Example:

alert ... (flow:to_server; iprep:src,Bot,>,10;)

This will generate an alert if the SRC IP of the host talking to a server is known to have a score of >10 in the “Bot” category.

The keyword is compatible to Suricata’s concept of “IP-only” rules. These are rules that do not inspect packet content or flow state and can thus be inspected once per flow direction instead of for each packet.

Speed

I’ve been playing with data sets of up to a million entries. Loading it takes hardly any time and I’m confident larger numbers will work just fine. The host table just needs bigger memcaps and hash sizes.

At runtime, the speed depends mostly on the rules. A pure “iprep” rule is quite expensive when not IP-only, although this is mostly due to the frequency of the checks. Such rules will be checked against large numbers of packets.

When created as a IP-only rule, things change. Such rules are checked only once per flow direction, so overhead appears to be minimal in this case.

Data

The data I used from Emerging Threats Pro is not available for free, so for those who want to test creating your own data is required right now. Matt Jonkman from Emerging Threats Pro will make a free feed available within a few weeks though. Of course you could also get the paid data from Emerging Threats Pro. 🙂

Update 29/11/2012

This feature is part of the just released 1.4rc1 version, please help us test it!