Suricata and PCRE performance

Update: Will Metcalf pointed out I was missing the –enable-utf8 –enable-unicode-properties flags from PCRE, so added these & updated the numbers. Thanks Will.

In the Emerging Threats community the following if often heard: “PCRE is evil”. With this people refer to signatures that use “pure” PCRE matches, meaning without anchoring it to a content pattern match.

A while ago Will Metcalf initiated work to get Suricata to support a new PCRE feature by Herczeg Zoltán: SLJIT. Since then, support for this has found it’s way into the official PCRE release, currently at version 8.20-RC3.

I decided to run a quick benchmark to see how much difference there would be. The results are quite amazing. In my test I used an older Intel Core2 E6600 2.4Ghz on Ubuntu 10.10, a 416MB pcap full of badness (sandnet traffic) and a slightly older ruleset of 11.972 signatures.

The results:

suricata, OS default pcre (8.02)...................: 78s
suricata, pcre-8.20-RC3 (no jit), -O2..............: 80s
suricata, pcre-8.20-RC3 (no jit), -O3 -march=native: 72s
suricata, pcre-8.20-RC3 (jit), -O2.................: 53s

I played some more with GCC 4.6.1 and various optimization levels, but this was the best result so far. Quite surprising because in the past I saw some improvements from using the newer GCC over the OS default of 4.4.5.

Want to try the new PCRE without messing up your system?
./configure --prefix=/opt/pcre-8.20-RC3/ --enable-jit --enable-utf8 --enable-unicode-properties
make
sudo make install

Then recompile Suricata as well:
./configure --enable-pcre-jit --with-libpcre-libraries=/opt/pcre-8.20-RC3/lib/ --with-libpcre-includes=/opt/pcre-8.20-RC3/include/
make
sudo make install

You’ll need the Suricata code from git to take advantage of this.

Please give it a try, it’s free performance!

7 thoughts on “Suricata and PCRE performance

  1. And since this is Inliniac (*Everything* inline 8-), and I still get to deal with snort, I’ll add that snort also benefits from the addition of SLJITt. Tested it with snort-2.9.1.1 and am seeing about 8-10% improvement – not as good as Suricata, so far, but better than nothing! Nice stuff, Zoltan, Will and Victor!!

  2. To benefit fully from the JIT support some minor changes to Suricata were needed. Nothing more than passing PCRE_STUDY_JIT_COMPILE to pcre_study(). If the Snort devs do the same there’s a good chance it’s performance increase will be bigger as well.

  3. You might want to check out re2wrap, a C-wrapper for RE2 used by OS inferno under MIT license. Or at least I think it might help get you started. It’s on code.google.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s