Update: Will Metcalf pointed out I was missing the –enable-utf8 –enable-unicode-properties flags from PCRE, so added these & updated the numbers. Thanks Will.
In the Emerging Threats community the following if often heard: “PCRE is evil”. With this people refer to signatures that use “pure” PCRE matches, meaning without anchoring it to a content pattern match.
A while ago Will Metcalf initiated work to get Suricata to support a new PCRE feature by Herczeg Zoltán: SLJIT. Since then, support for this has found it’s way into the official PCRE release, currently at version 8.20-RC3.
I decided to run a quick benchmark to see how much difference there would be. The results are quite amazing. In my test I used an older Intel Core2 E6600 2.4Ghz on Ubuntu 10.10, a 416MB pcap full of badness (sandnet traffic) and a slightly older ruleset of 11.972 signatures.
The results:
suricata, OS default pcre (8.02)...................: 78s
suricata, pcre-8.20-RC3 (no jit), -O2..............: 80s
suricata, pcre-8.20-RC3 (no jit), -O3 -march=native: 72s
suricata, pcre-8.20-RC3 (jit), -O2.................: 53s
I played some more with GCC 4.6.1 and various optimization levels, but this was the best result so far. Quite surprising because in the past I saw some improvements from using the newer GCC over the OS default of 4.4.5.
Want to try the new PCRE without messing up your system?
./configure --prefix=/opt/pcre-8.20-RC3/ --enable-jit --enable-utf8 --enable-unicode-properties
make
sudo make install
Then recompile Suricata as well:
./configure --enable-pcre-jit --with-libpcre-libraries=/opt/pcre-8.20-RC3/lib/ --with-libpcre-includes=/opt/pcre-8.20-RC3/include/
make
sudo make install
You’ll need the Suricata code from git to take advantage of this.
Please give it a try, it’s free performance!
Inliniac 
Nice!!!
Thnak you for the measurement. Really nice results.
And since this is Inliniac (*Everything* inline 8-), and I still get to deal with snort, I’ll add that snort also benefits from the addition of SLJITt. Tested it with snort-2.9.1.1 and am seeing about 8-10% improvement – not as good as Suricata, so far, but better than nothing! Nice stuff, Zoltan, Will and Victor!!
To benefit fully from the JIT support some minor changes to Suricata were needed. Nothing more than passing PCRE_STUDY_JIT_COMPILE to pcre_study(). If the Snort devs do the same there’s a good chance it’s performance increase will be bigger as well.
JIT is very nice, but why Suricata don’t try RE2 from Google?
We are aware of RE2, the problem is that Suricata is in C, the lib in C++. Using C++ libs/code from C programs is not trivial.
You might want to check out re2wrap, a C-wrapper for RE2 used by OS inferno under MIT license. Or at least I think it might help get you started. It’s on code.google.