Vuurmuur Development Update

Over the holidays I’ve spent some time refreshing the Vuurmuur code. One major thing that is now done is that the 3 different ‘projects’ (libvuurmuur, vuurmuur and vuurmuur-conf) are now merged into a single ‘project’. This means that a single ‘./configure && make && make install’ now installs everything.

When I originally started Vuurmuur I had much bigger dreams for it than eventually materialized. Also, I didn’t understand autotools very well, so it was easier to keep the project split up. At some point there were even 5 projects!

One very convenient consequence is that development can now be done without system wide installation of the libs. This may sound trivial, but it really speeds things up.

I’ve updated the install script and the debian scripts for this new model as well.

QA

A second point is the use of better QA.

  1. Travis-CI integration. This tests gcc/clang builds for compilation warnings and errors, the install script, debian package generation
  2. Scan-build and cppcheck. Vuurmuur is now clean in scan-build 3.9 and cppcheck 1.77.
  3. Coverity Scan. I’ve registered Vuurmuur with Coverity’s Scan program. Initially there were quite a few issues, although most of them minor. I’ve fixed all of them so now Vuurmuur is clean for Coverity as well.
  4. ASAN/UBSAN: I’m running Vuurmuur with address and undefined behavior sanitizers enabled. Fixed a few issues because of that.

Error handling

One major source of issues with the static checkers was the error handling in vuurmuur_conf. This lead to many completely untested code paths, usually for things like memory allocation failure or other ‘internal’ errors. I’ve simplified that handling enormously, by simply adding a class of ‘fatal’ errors that simply exit vuurmuur_conf in such conditions. This has lead to a smaller and cleaner code base.

User visible changes

Most of the changes are internal, but a few things are user visible.

  1. removal of QUEUE support. ip_queue is long dead and has been replaced with NFQUEUE.
  2. proper sorting of connections in Connection Viewer.
  3. default to black background in vuurmuur_conf

I’m hoping to push out a new release soon(ish). Time contraints will continue to be a big issue though. So if anyone wants to help out, please let me know.

Suricata bits, ints and vars

Since the beginning of the project we’ve spoken about variables on multiple levels. Of course flowbits defined by the Snort language came first, but other flow based variables quickly followed: flowints for basic counting, and vars for extracting data using pcre expressions.

I’ve always thought of the pcre data extraction using substring capture as a potentially powerful feature. However the implementation was lacking. The extracted data couldn’t really be used for much.

Internals

Recently I’ve started work to address this. The first thing that needed to be done was to move the mapping between variable names, such as flowbit names, and the internal id’s out of the detection engine. The detection engine is not available in the logging engines and logging of the variables was one of my goals.

This is a bit tricky as we want a lock less data structure to avoid runtime slow downs. However rule reloads need to be able to update it. The solution I’ve created has a read only look up structure after initialization that is ‘hot swapped’ with the new data at reload.

PCRE

The second part of the work is to allow for more flexible substring capture. There are 2 limitations in the current code: first, only single substring can be captured per rule. Second, the names of the variables were limited by libpcre. 32 chars with hardly any special chars like dots. The way to express these names has been a bit of a struggle.

The old way looks like this:

pcre:"/(?P.*)<somename>/";

This create a flow based variable named ‘somename’ that is filled by this pcre expression. The ‘flow_’ prefix can be replaced by ‘pkt_’ to create a packet based variable.

In the new method the names are no longer inside the regular expression, but they come after the options:

pcre:"/([a-z]+)\/[a-z]+\/(.+)\/(.+)\/changelog$/GUR, \
    flow:ua/ubuntu/repo,flow:ua/ubuntu/pkg/base,     \
    flow:ua/ubuntu/pkg/version";

After the regular pcre regex and options, a comma separated lists of variable names. The prefix here is ‘flow:’ or ‘pkt:’ and the names can contain special characters now. The names map to the capturing substring expressions in order.

Key/Value

While developing this a logical next step became extraction of key/value pairs. One capture would be the key, the second the value. The notation is similar to the last:

pcre:"^/([A-Z]+) (.*)\r\n/G, pkt:key,pkt:value";

‘key’ and ‘value’ are simply hardcoded names to trigger the key/value extraction.

Logging

Things start to get interesting when logging is added. First, by logging flowbits existing rulesets can benefit.

{
  "timestamp": "2009-11-24T06:53:35.727353+0100",
  "flow_id": 1192299914258951,
  "event_type": "alert",
  "src_ip": "69.49.226.198",
  "src_port": 80,
  "dest_ip": "192.168.1.48",
  "dest_port": 1077,
  "proto": "TCP",
  "tx_id": 0,
  "alert": {
    "action": "allowed",
    "gid": 1,
    "signature_id": 2018959,
    "rev": 2,
    "signature": "ET POLICY PE EXE or DLL Windows file download HTTP",
    "category": "Potential Corporate Privacy Violation",
    "severity": 1
  },
  "http": {
    "hostname": "69.49.226.198",
    "url": "/x.exe",
    "http_user_agent": "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)",
    "http_content_type": "application/octet-stream",
    "http_method": "GET",
    "protocol": "HTTP/1.1",
    "status": 200,
    "length": 23040
  },
  "vars": {
    "flowbits": {
      "exe.no.referer": true,
      "http.dottedquadhost": true,
      "ET.http.binary": true
    }
  }
}

When rules are created to extract info and set specific ‘information’ flowbits, logging can create value:

"vars": {
  "flowbits": {
    "port/http": true,
    "ua/os/windows": true,
    "ua/tool/msie": true
  },
  "flowvars": {
    "ua/tool/msie/version": "6.0",
    "ua/os/windows/version": "5.1"
  }
}
"http": {
  "hostname": "db.local.clamav.net",
  "url": "/daily-15405.cdiff",
  "http_user_agent": "ClamAV/0.97.5 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)",
  "http_content_type": "application/octet-stream",
  "http_method": "GET",
  "protocol": "HTTP/1.0",
  "status": 200,
  "length": 1688
},
"vars": {
  "flowbits": {
    "port/http": true,
    "ua/os/linux": true,
    "ua/arch/x64": true,
    "ua/tool/clamav": true
  },
  "flowvars": {
     "ua/tool/clamav/version": "0.97.5"
  }
}

In the current code the alert and http logs are showing the ‘vars’.

Next to this, a ‘eve.vars’ log is added, which is a specific output of vars independent of other logs.

Use cases

Some of the use cases could be to add more information to logs without having to add code. For example, I have a set of rules that set of rules that extracts the packages are installed by apt-get or for which Ubuntu’s updater gets change logs:

"vars": {
  "flowbits": {
    "port/http": true,
    "ua/tech/python/urllib": true
  },
  "flowvars": {
    "ua/tech/python/urllib/version": "2.7",
    "ua/ubuntu/repo": "main",
    "ua/ubuntu/pkg/base": "libxml2",
    "ua/ubuntu/pkg/version": "libxml2_2.7.8.dfsg-5.1ubuntu4.2"
  }
}

It could even be used as a simple way to ‘parse’ protocols and create logging for them.

Performance

Using rules to extract data from traffic is not going to be cheap for 2 reasons. First, Suricata’s performance mostly comes from avoiding inspecting rules. It has a lot of tricks to make sure as little rules as possible are evaluated. Likewise, the rule writers work hard to make sure their rules are only evaluated if they have a good chance of matching.

The rules that extract data from user agents or URI’s are going to be matching very often. So even if the rules are written to be efficient they will still be evaluated a lot.

Secondly, extraction currently can be done through PCRE and through Lua scripts. Neither of which are very fast.

Testing the code

Check out this branch https://github.com/inliniac/suricata/pull/2468 or it’s replacements.

Bonus: unix socket hostbits

Now that variable names can exist outside of the detection engine, it’s also possible to add unix socket commands that modify them. I created this for ‘hostbits’. The idea here is to simply use hostbits to implement white/blacklists. A set of unix socket commands will be added to manage add/remove them. The existing hostbits implementation handles expiration and matching.

To block on the blacklist:

drop ip any any -> any any (hostbits:isset,blacklist; sid:1;)

To pass all traffic on the whitelist:

pass ip any any -> any any (hostbits:isset,whitelist; sid:2;)

Both rules are ‘ip-only’ compatible, so will be efficient.

A major advantage of this approach is that the black/whitelists can be
modified from ruleset themselves, just like any hostbit.

E.g.:

alert tcp any any -> any any (content:"EVIL"; \
    hostbits:set,blacklist; sid:3;)

A new ‘list’ can be created this way by simply creating a rule that
references a hostbit name.

Unix Commands

Unix socket commands to add and remove hostbits need to be added.

Add:

suricatasc -c "add-hostbit <ip> <hostbit> <expire>"
suricatasc -c "add-hostbit 1.2.3.4 blacklist 3600"

If an hostbit is added for an existing hostbit, it’s expiry timer is updated.

Hostbits expire after the expiration timer passes. They can also be manually removed.

Remove:

suricatasc -c "remove-hostbit <ip> <hostbit>"
suricatasc -c "remove-hostbit 1.2.3.4 blacklist"

Feedback & Future work

I’m looking forward to getting some feedback on a couple of things:

  • log output structure and logic. The output needs to be parseable by things like ELK, Splunk and jq.
  • pcre options notation
  • general feedback about how it runs

Some things I’ll probably add:

  • storing extracted data into hosts, ippairs
  • more logging

Some other ideas:

  • extraction using a dedicated keyword, so outside of pcre
  • ‘int’ extraction

Let me know what you think!

Fuzzing Suricata with pcaps

Yesterday I wrote about fuzzing Suricata with AFL. Today I’m going to show another way. Since early in the project, we’ve shipped a perl based fuzzer called ‘wirefuzz’. The tool is very simple. It takes a list of pcaps, changes random bits in them using Wiresharks editcap and runs them through Suricata. Early in the project Will Metcalf, who wrote the tool, found a lot of issues with it.

Since it’s random based fuzzing, the fuzzing is quite shallow. It is still a great way of stressing the decoder layers of Suricata though, as we need to be able to process all junk input correctly.

Lately we had an issue that I thought should have been found using fuzzing: #1653, and indeed, when I started fuzzing the code I found the issue within an hour. Pretty embarrassing.

Another reason to revisit is Address Sanitizer. It’s great because it’s so unforgiving. If it finds something it blows up. This is great for fuzzing. It’s recommended to use AFL with Asan as well. Wirefuzz does support a valgrind mode, but that is very slow. With Asan things are quite fast again, while doing much more thorough checking.

So I decided to spend some time on improving this tool so that I can add it to my CI set up.

Here is how to use it.

git clone https://github.com/inliniac/suricata -b dev-fuzz-v3.1
cd suricata
git clone https://github.com/OISF/libhtp -b 0.5.x
bash autogen.sh
export CFLAGS="-fsanitize=address"
./configure --disable-shared --sysconfdir=/etc
make
mkdir fuzzer
# finally run the fuzzer
qa/wirefuzz.pl -r=/home/victor/pcaps/*/* -c=suricata.yaml -e=0.02 \
    -p=src/suricata -l=fuzzer/ -S=rules/http-events.rules -N=1

What this command does is:

  • run from the source dir, output into fuzzer/
  • modify 2% of each pcap randomly while making sure the pcap itself stays valid (-e=0.02)
  • use the rules file rules/http-events.rules exclusively (-S)
  • use all the pcaps from /home/victor/pcaps/*/*
  • return success if a single pass over the pcaps was done (-N=1)

One thing to keep in mind is that the script creates a copy of the pcap when randomizing it. This means that very large files may cause problems depending on your disk space.

I would encourage everyone to fuzz Suricata using your private pcap collections. Then report issues to me… pretty please? 🙂

*UPDATE 2/15*: the updated wirefuzz.pl is now part of the master branch.

Fuzzing Suricata with AFL

AFL is a very powerful fuzzer, that tries to be smarter than random input generating fuzzers. It’s cool, but needs a bit more baby sitting. I’ve added some support to Suricata to assist AFL.

Here’s how to get started on fuzzing pcaps.

mkdir ~/tmp/fuzz
git clone https://github.com/inliniac/suricata -b dev-afl-v5
cd suricata
git clone https://github.com/OISF/libhtp -b 0.5.x
bash autogen.sh
export CFLAGS="-fsanitize=address"
export AFLDIR=/opt/afl-1.96b/bin/
export CC="${AFLDIR}/afl-gcc"
export CXX="${AFLDIR}/afl-g++"
./configure --disable-shared --sysconfdir=/etc --enable-afl

The configure output should show:
Compiler: /opt/afl-1.96b/bin//afl-gcc (exec name) / gcc (real)

make

# create tmp output dir for suricata
mkdir tmp/

# test the command to be fuzzed
src/suricata --runmode=single -k none -c suricata.yaml -l tmp/ \
    -S /dev/null \
    -r /opt/afl-1.96b/share/afl/testcases/others/pcap/small_capture.pcap

# start the fuzzer
export AFL_SKIP_CPUFREQ=1
/opt/afl-1.96b/bin/afl-fuzz -t 100000 -m none \
    -i /opt/afl-1.96b/share/afl/testcases/others/pcap/ -o aflout -- \
    src/suricata --runmode=single -k none -c suricata.yaml -l tmp/ \
    -S /dev/null -r @@

AFL should start running:

afl

Couple of things to keep in mind:

  • the above list assumes you have a /etc/suricata/ set up already, including a reference.config and classification.config
  • don’t skip the test step or you risk that AFL will just fuzz some basic error reporting by Suricata
  • the used ‘dev-afl-v5’ branch makes fuzzing faster and more reliable by disabling random, threading and a few other things
  • src/suricata –build-info should show the compiler is afl
  • keep your test cases small, even then runtime is going to be very long. AFL takes the input and modifies it to find as many unique code paths as possible

 

Fuzzing rules and YAMLs

For fuzzing rules and YAMLs the compilation steps are the same.

To fuzz rules, create a directory & test input:

mkdir testrules
echo 'alert http any any -> any any (content:"abc"; sid:1; rev:1;)' \
    > testrules/rules.txt

# test command
src/suricata -c suricata.yaml -l tmp/ --afl-parse-rules -T \
    -S testrules/rules.txt

# run AFL
export AFL_SKIP_CPUFREQ=1
/opt/afl-1.96b/bin/afl-fuzz -t 100000 -m none \
    -i testrules/ -o aflout -- \
    src/suricata -c suricata.yaml -l tmp/ --afl-parse-rules \
    -T -S @@

Finally, YAMLs:

mkdir testyamls/
cp suricata.yaml testyamls/

# test command
src/suricata -l tmp/ --afl-parse-rules -T -S testrules/rules.txt \
    -c testyamls/suricata.yaml

# run AFL
export AFL_SKIP_CPUFREQ=1
/opt/afl-1.96b/bin/afl-fuzz -t 100000 -m none \
    -i testyamls/ -o aflout -- \
    src/suricata -l tmp/ --afl-parse-rules \
    -T -S testrules/rules.txt -c @@

Note that the default YAML is HUGE for this purpose. It may be more efficient to use a sub set of it.

I plan to create some wrapper scripts to make things easier in the near future. Meanwhile, if you have crashes to report, please send them my way!

Suricata 3.0 is out!

suri-400x400Today, almost 2 years after the release of Suricata 2.0, we released 3.0! This new version of Suricata improves performance, scalability, accuracy and general robustness. Next to this, it brings a lot of new features.

New features are too numerous to mention here, but I’d like to highlight a few:

  • netmap support: finally a high speed capture method for our FreeBSD friends, IDS and IPS
  • multi-tenancy: single instance, multiple detection configs
  • JSON stats: making it much easier to graph the stats in ELK, etc
  • Much improved Lua support: many more fields/protocols available, output scripts

Check the full list here in the announcement: http://suricata-ids.org/2016/01/27/suricata-3-0-available/

New release model

As explained here, this is the first release of the new release model where we’ll be trying for 3 ‘major’ releases a year. We originally hoped for a month of release candidate cycles, but due to some issues found and the holidays + travel on my end it turned into 2 months.

My goal is to optimize our testing and planning to reduce this further, as this release cycle process is effectively an implicit ‘freeze’. Take a look at the number of open pull requests to see what I mean. For the next cycle I’ll also make the freeze explicit, and announce it.

Looking forward

While doing a release is great, my mind is already busy with the next steps. We have a bunch of things coming that are exciting to me.

Performance: my detection engine rewrite work has been tested by many already, and reports are quite positive. I’ve heard reports up to 25% increase, which is a great bonus considering the work was started to clean up this messy code.

ICS/SCADA: Jason Ish is finalizing a DNP3 parser that is very full featured, with detection, logging and lua support. Other protocols are also being developed.

Documentation: we’re in the process of moving our user docs from the wiki to sphinx. This means we’ll have versioned docs, nice pdf exports, etc. It’s already 180 pages!

Plus lots of other things. Keep an eye out on our mailing lists, bug tracker or IRC channel.

New Suricata release model

suri-400x400As the team is back from a very successful week in Barcelona, I’d like to take a moment on what we discussed and decided on with regards to development.

One thing no one was happy with is how the release schedules are working. Releases were meant to reasonably frequent, but the time between major releases was growing longer and longer. The 2.0 branch for example, is closing in on 2 years as the stable branch. The result is that many people are missing out on many of the improvements we’ve been doing. Currently many people using Suricata actually use a beta version, of even our git master, in production!

What we’re going to try is time based releases. Pretty much releases will be more like snapshots of the development branch. We think this can work as our dev branch is more and more stable due to our extensive QA setup.

Of course, we’ll have to make sure we’re not going to merge super intrusive changes just before a release. We’ll likely get into some pattern of merge windows and (feature) freezes, but how this will exactly play out is something we’ll figure out as we go.

We’re going to try to shoot for 3 of such releases per year.

In our redmine ticket tracker, I’ve also created a new pseudo-version ‘Soon’. Things we think should be addressed for the next release, will be added there. But we’ll retarget the tickets when they are actually implemented.

Since it’s already almost 2 years since we’ve done 2.0, we think the next release warrants a larger jump in the versioning. So we’re going to call it 3.0. The first release candidate will likely be released this week hopefully followed by a stable in December.

Get paid to work on Suricata?

If you like fiddling with Suricata development, maybe you can get paid to do it.

Companies ask me regularly if I can recommend Suricata developers. I’m going to assemble a list of people who are interested in such work. If you like me to consider you in such cases, drop me an email.

If you really want me to *recommend* you, it’s important that I actually know you somewhat. So becoming a (volunteer) contributor will help a lot.

Things to mention in your email:
– interests
– github profile
– open source contributions
– social media, blog links
– availability, whether you’re a contractor or looking for a real J-O-B

Who knows, maybe something good will come from it!

Btw, if you’re not a dev but great at research, or deployments and tuning, I know some ppl that are always looking for such new hires as well!