Migrating from ModSecurity 1.9.4 to 2.0.4

ModSecurity 2 has been out for a while now, and although I have played with it some, I never found some time to upgrade my own servers. The upgrading generally went quite smooth, even though ModSecurity 2 changed quite a bit.

First of all there are now 5 phases where you can filter. Actually, one of them only applies to the logging, so you can filter in 4 phases. The phases are headers and body for both request and response traffic. Filtering on specific URIs can be done in phase 1 (request headers), while inspecting a POST payload requires phase 2 (request body).

Next, some shortcuts where removed. In 1.9.4 there was a variable called POST_PAYLOAD, that enabled the user to match against payloads from POST requests easily. Now there is REQUEST_BODY, but since that can be part of non-POST requests as well, you have to use:

SecRule REQUEST_METHOD “POST” chain
SecRule REQUEST_BODY “evil”

instead of:

SecFilterSelective POST_PAYLOAD “evil”

One other change is visible above already. The keyword to create a rule has been changed from SecFilterSelective to SecRule. Many rules can be converted by just replacing the keyword, but certainly not all. A simple find/replace should not be done without a manual review!

I use a number of custom rules to protect certain parts of my server, so I needed to convert my rules. For most of them it was simply enough to replace SecFilterSelective with SecRule. For a few I had to replace the OUTPUT_STATUS variable with the RESPONSE_STATUS variable, as it is called now.

For one rule however, I had quite some problems to get it running correctly. This was the rule in 1.9.4 syntax:

# block wp-login.php
SecFilterSelective REMOTE_ADDR “!10.1.1.1” chain
SecFilterSelective REQUEST_URI “/wp-login.php”
log,deny,redirect:http://www.inliniac.net/nologin.html

This rule makes sure only 10.1.1.1 can open the login page, everyone else is redirected to a simple html page containing a ‘Access Denied – Logins disabled’ message. I converted it to the following:

SecRule REMOTE_ADDR “!10.1.1.1” chain
SecRule REQUEST_URI “/wp-login.php”
log,deny,redirect:http://www.inliniac.net/nologin.html

Guess what? It didn’t work. I’ve spend quite some time trying all kinds of variations of the rule, and finally I found out what the issue is. In 1.9.4 the rule actions, like deny, redirect etc could be in the final rule of a series of chained rules. With 2.0.4 this doesn’t work correctly. So when I changed the rules to the following, it worked:

# block wp-login.php
SecRule REMOTE_ADDR “!10.1.1.1” “chain,phase:1,log,deny,redirect:http://www.inliniac.net/nologin.html”
SecRule REQUEST_URI “/wp-login.php$”

I haven’t looked into this further to find out whether this is a bug or a feature.

The last thing that was interesting is the modsec2sguil script. There have been some changes to the alert files. So expect a new version of the script soon!

Setting up networking in Qemu for using OpenNSM

Geek00L has created a OpenBSD 4.0 based QEMU image called OpenNSM which I decided to give a try. Getting the networking part to work with my Linux host was a bit of a puzzle, so i’m writing it down here. Most of the steps were taken from the unofficial qemu wiki page here, but not all of them were necesarry.

First of all, I had to use the QEMU version 0.8.2 from the QEMU website, because the 0.8.0 version from my Ubuntu Dapper workstation didn’t work. With the latter version the OpenBSD bootup would just hang at the message ‘clock: Unknown CMOS layout’.

For the networking to work, I needed to setup a bridge between the actual NIC and the tap interface that QEMU was going to talk to. First the tap:

modprobe tun
chmod a+rwx /dev/net/tun

Next the bridge setup on eth1. The tap was added to it later.

brctl addbr br0
ifconfig eth1 0.0.0.0 up
brctl addif br0 eth1

Added an IP Address to the bridge in the subnet connected to the eth1 NIC.

ifconfig br0 192.168.2.12

After this I setup the file /etc/qemu-ifup. This file is called by QEMU on startup to setup the tap networking.

#!/bin/sh
sudo /sbin/ifconfig $1 0.0.0.0 promisc up
sudo /usr/sbin/brctl addif br0 $1

Now I was ready to start the image:

qemu -hda OpenNSM.img -m 256 -net nic -net tap

Once the image was booted, I made the corrections described by Geek00L here. Finally, I ran /root/nsm-scripts/net-config.sh in the OpenBSD installation to setup the system with a unique IP Address. I chose 192.168.2.13. After this I could use networking from my OpenNSM installation 🙂

Modsec2sguil 0.6 released

I’ve just release a new version of modsec2sguil, the set of Perl scripts that feeds ModSecurity alerts to Sguil. No new features, but many changes ‘under the hood’. I’ve created two modules, ModsecAlert and SguilBarnyardComms. These can be used in a Object Oriented way to parse ModSecurity events and communitcate a Sguil sensor agent.

It would be interesting to see if the SguilBarnyardComms module could be connected with the work of Jason Brevnik of SourceFire, who wrote a Barnyard replacement in Perl. If I have some spare time, I will have a look at this.

After this release, I want to look at bypassing the sensor_agent altogether and instead connect directly to the Sguil server. Bamm Visscher has plans to redesign parts of the agent – server communications. In the new ideas Sguil moves away from the one monolithic sensor_agent. Instead, different tasks will have their own agent which connects directly to the sguil server. For example a sancp sensor, a snort sensor, a pcap sensor, etc. Next, a number of the same sensors would be able to share a common id, called network id. This way the user can ask a transcript for an alert produced by a modsec sensor. It is my intention to create a Perl module or library that will make creating new agents for this setup easy.

Anyway, thats the future, modsec2sguil 0.6 is ready for your testing right now! Let me know how it works for you!

Download it here: http://www.inliniac.net/files/modsec2sguil-0.6.tar.gz

Sguil: renaming a sensor

This might be a no-brainer for some, but today I wanted to rename a sensor in Sguil, and decided to write the steps down.

Preparation

  1. backup the db
  2. stop sancp on the sensor
  3. stop barnyard on the sensor
  4. stop sensor_agent on the sensor
  5. stop snort on the sensor
  6. stop log_packets on the sensor
  7. stop sguild on the server

Steps on the Sguil server

Update the sensor name in the database.

# mysql -p
mysql> use sguildb;
mysql> UPDATE sensor SET hostname = “newname” WHERE hostname = “oldname”;
Query OK, 1 row affected (0.04 sec)
Rows matched: 1 Changed: 1 Warnings: 0

Rename the directory for the rules display

mv /sguild_data/rules/oldname /sguild_data/rules/newname

Steps on the sensor

Change the hostname in /etc/sensor_agent.conf

set HOSTNAME newname

Change the hostname in /etc/barnyard/barnyard.conf

config hostname: newname

Change the hostname in /usr/sbin/log_packets.sh

HOSTNAME=”newname”

Change the directory where sancp writes its files, for my Debian system in /etc/default/sancp

SANCP_ARCHIVE_DIR=”/snort_data/newname/sancp”

Change the Snort prefmonitor stats file location in /etc/snort/snort.conf

preprocessor perfmonitor: time 300 file /snort_data/newname/snort.stats pktcnt 10000

Rename the sensor directory

mv /snort_data/oldname /snort_data/newname

Finally

  1. start sguild on the server
  2. start log_packets.sh on the sensor
  3. start sancp on the sensor
  4. start snort on the sensor
  5. start sensor_agent on the sensor
  6. start barnyard on the sensor

That should be all!

UPDATE: forgot about sancp, snort stats and log_packets.sh

Sguil: adding support for ModSecurity alerts, continued

After the successful test with the Perl script to add ModSecurity alerts to Sguil, I have been working on a more robust implementation, also in Perl. Let me first describe the basic setup. The setup works with two scripts. The first places links to event files into a special queue directory. The second reads the links from that directory, parses them and sends the alerts among these events to Sguil. After that, the links are removed.

In ModSecurity there is a logging method called concurrent. What it does is that it creates a directory structure based on date and time. For each event a file is created in the directory structure, with the details of the event. It also writes a logfile called index, that adds one line for every event, with location of the file in the date/time directory layout.

ModSecurity also supports using a external script to take care of this indexing task, which is what the script modsec-auditlog-collector in the ModSecurity source tarball does. I have modified this Perl script to create a link to the actual event file in a special queue directory. The links all have a name like: modsec.log.-timestamp-, like Snort uses with unified logging. I have chosen to use links because I think it might be faster than copying, but I have yet to benchmark that.

The second script takes care of processing the links and communicating with Sguil. It basically has the following workflow. It gets a list of all the files in the queue. Each file in the queue is parsed, then it is checked whether it is an alert. If so, the data is preprocessed to make sure it is what Sguil expects. Next it is send to Sguil. Finally, all the files are removed.

ModSecurity has a different rule syntax than Snort and sees a different type of data. Snort sees packets, and thus knows about TCP checksums, sequence numbers, etc. ModSecurity works on a higher level, so it has no access to that data. One thing that ModSecurity alerts do include is the action taken: blocked or redirected and with what HTTP code. If available the rule msg is used for the Event Message, otherwise the ModSecurity message with (a part of) the rule that matched. Below is a screenshot that shows both, and also the HTTP code.

Sguil and Mod_Security msg view

The current implementation acts as a drop-in replacement of Barnyard. This way no changes to Sguil are necessary. Currently this means that Barnyard cannot run on the same sensor, although multiple instances of sensor_agent on a host could fix that. Sguil author Bamm Visscher suggested I can try to connect to the Sguil server directly instead of via the sensor_agent to fix this. I will probably have a look at that later.

Another issue is the way the alert data is presented in the Sguil interface. Currently I attach the entire alert file to the alert that is being send to Sguil, as the packet payload. However the Sguil packet payload view is not very usable for this kind of data, as can be seen on the screenshot below.

Sguil and Mod_Security payload view

So here is room for much improvement. Ideally Sguil would be able to show the uri, the post payload if applicable, useragent, etc. I’ve never worked with TCL/TK, but I did use Perl/TK in the past, so that might help me a little if I decide to give it a shot.

I expect a public beta version to be ready some time next week, although I have not yet decided how to release it. I could create a project on SourceForge or just have a simple webpage. I just haven’t decided yet. Stay tuned…

Sguil: adding support for ModSecurity alerts

I’m a huge fan of both Sguil and ModSecurity, but sadly the alerts generated by ModSecurity can’t show up in Sguil… or… can they? Well, if it all works out, soon they can!

Today I have hacked together a perl script that emulates barnyard for ModSecurity. It very much in a proof-of-concept phase, but it somewhat works already, at least good enough so i can show this screenshot.

Sguil screenshot showing experimental Mod_Security support
Nice huh? It is not ready for release yet, but when it is i’ll announce it here. I plan to release it under the GPL. Sguil author Bamm Visscher told me that the next release of Sguil will have support for having barnyard and PADS on the same sensor. By then, i hope that ModSecurity can be added to that list! 🙂

Sguil: detecting ICMP tunnels, continued

A few days ago i wrote about detecting ICMP tunnels. I came to the conclusion that for the two tunnels i tried the properties to detect on were:

  1. Non-standard average packet size. E.g. 81.81 bytes for the first connection is non-standard.
  2. Number of bytes in both directions are unequal.
  3. Average packet size in both directions is unequal.

I believe the third is the most important, because it must mean that 1 is also true and 2 is very likely to be true. If the average packet size is unequal in both directions, one of them should be non-standard. So far i have only seen echo-request and echo-reply being equal in size for pings. Unequal average packet sizes with exactly the same amount of data transfered is very unlikely, so i have ignored that for now.

This is the full report that shows all ICMP with unequal average packet sizes:

SUSPECTICMP||Suspect ICMP LAN->WAN: Top 10 ICMP users in the LAN talking to the WAN, with different packet sizes, with source and destination, volume and average packet size, sorted by packet size (fields: source ip, dest ip, from source MB, from source packet size, from dest MB, from dest packet size, total duration, average duration)||query||SELECT INET_NTOA(src_ip), INET_NTOA(dst_ip), my_src_bytes/1048576 AS my_src_mb, my_avg_src_pktsize, my_dst_bytes/1048576 AS my_dst_mb, my_avg_dst_pktsize, my_tot_duration, AVG(my_tot_duration) AS my_avg_duration FROM ( SELECT src_ip, dst_ip, ip_proto, SUM(src_bytes) AS my_src_bytes, SUM(dst_bytes) AS my_dst_bytes, start_time, end_time, SUM(src_bytes)/sum(src_pkts) AS my_avg_src_pktsize, SUM(dst_bytes)/sum(dst_pkts) AS my_avg_dst_pktsize, SUM(duration) AS my_tot_duration FROM ( SELECT sid AS my_sid, src_ip, dst_ip, ip_proto, src_bytes, dst_bytes, src_pkts, dst_pkts, start_time, end_time, duration FROM sancp IGNORE INDEX (p_key) WHERE ip_proto = 1 AND src_pkts > 0 AND dst_pkts > 0 AND start_time > %%STARTTIME%% AND end_time < %%ENDTIME%% ) AS step1_table INNER JOIN sensor ON my_sid=sensor.sid WHERE %%SENSORS%% AND ((src_ip between INET_ATON("192.168.0.0") and INET_ATON("192.168.255.255")) or (src_ip between INET_ATON("10.0.0.0") and INET_ATON("10.255.255.255")) or (src_ip between INET_ATON("172.16.0.0") and INET_ATON("172.31.255.255"))) and ((dst_ip not between INET_ATON("192.168.0.0") and INET_ATON("192.168.255.255")) and (dst_ip not between INET_ATON("10.0.0.0") and INET_ATON("10.255.255.255")) and (dst_ip not between INET_ATON("172.16.0.0") and INET_ATON("172.31.255.255"))) GROUP BY dst_bytes ) AS step2_table WHERE my_avg_src_pktsize != my_avg_dst_pktsize GROUP BY src_ip,dst_ip ORDER BY my_avg_dst_pktsize DESC, my_avg_src_pktsize DESC LIMIT 10||8||

This query only works on MySQL 4.1 and higher. It results in this report:

Sguil Sensor Report Showing an Suspect ICMP Report

This detects all tunnels I had setup, and shows nothing of the other non-tunnel tests I did, like the ping flood.

Sguil: using advanced queries to get more info on Snort events

Today, David Bianco showed me a way of creating SQL queries that I didn’t even know was possible. This is a technique with which it is possible to query the payload of Snort events in the Sguil database. These payloads are stored by Snort when it alerts and is the payload the actual rule triggered on. David showed a nice example of retrieving url’s for PHP url inclusion attacks.

I have written before about my usage of Mod_Security. I let Mod_Security respond with a 403 Forbidden message. Sadly, the alert generated by Mod_Security can not be displayed in Sguil. As somewhat of a replacement for that, I let Snort alert on the 403 Forbidden messages, so i can see in Sguil that Mod_Security blocked something. The disadvantage of this is that the 403 alert in itself does not contain much info. The sheer number of 403’s makes inspecting every single one with requesting a transcript a bit to much work.

This is where the new query comes in. The query creates a list of url’s that the server reported to be 403 Forbidden. Interesting is that we are not looking at an attack, but at an attack response. This means the the attackers IP is actually the destination IP.

SELECT COUNT(*) AS cnt, INET_NTOA(dst_ip) AS “Dest IP”, trim(LEADING “access ” FROM substring_index(substr(unhex(data_payload),locate(‘access ‘,unhex(data_payload))), ‘n’, 1)) AS url FROM event INNER JOIN data ON event.sid = data.sid and event.cid = data.cid where (signature like “%ATTACK-RESPONSES 403 Forbidden%”) GROUP BY dst_ip,url ORDER BY cnt DESC LIMIT 10;

The result is this:

MySQL result for query for 403 Forbidden event url's

As you can see, the first six results are from the comment spam I wrote about earlier. I left the source IP out because there is only one webserver. The query can easily be extended to show source IP’s as well.

Sguil: detecting ICMP tunnels

My earlier post about detecting ICMP anomolies was based on dry theory combined with experimenting with the ping command. The last couple of days I have been playing with real ICMP tunnels, to see how detection of those would work. This was easier said than done. Sure, running PingTunnel or itun between two hosts in my LAN worked fine. However, being an inline guy, i have Sancp looking at traffic passing my firewall only. And getting the ICMP tunnels to pass the firewall was the real trick. I won’t bore you with that now, because i intend to look at counter measures later, so i’ll handle that then. For now I will just assume that these ping tunnels will not be blocked by the firewall.

The query is largely the same as in my earlier post about detecting ICMP anomolies. However, i made one addition. Sancp also stores session duration information. In other words, how long did a session last. I figured that ICMP tunnels may have longer than average duractions, since they are likely to be used for hiding interactive sessions like ssh, irc or msn. One should remember however, that a continues ping command that (accidently) runs for a long period of time will also have a long duration.

Sguil Sensor Report Showing an ICMP Report

Five results. Three of which are (partly) ICMP tunnel traffic. Spotting them is easy. They all have the following properties:

  1. Non-standard average packet size. E.g. 81.81 bytes for the first connection is non-standard.
  2. Number of bytes in both directions are unequal.
  3. Average packet size in both directions is unequal.

The number of bytes transfered seems to be not a really good indicator in itself. The second result transfered 29 MB, the biggest number from these results, but was no tunnel. It was a ‘ping -f 192.168.0.102’ for a few minutes. The same goes for the duration. Sure the first tunnel has by far the longest duration, but if i keep a ping session running for a week it won’t misteriously turn into a tunnel 😉

Still, i think it is useful to have the information around because once you suspect a tunnel exists, volume and duration might be further indicators something is wrong.

This is the Sguil report:

TOPTENICMP||Top 10 ICMP LAN->WAN: Top 10 ICMP users in the LAN talking to the WAN, with source and destination, volume and average packet size, sorted by packet size (fields: source ip, dest ip, from source MB, from source packet size, from dest MB, from dest packet size, total duration, average duration)||query||select INET_NTOA(src_ip), INET_NTOA(dst_ip), sum(src_bytes)/1048576 as from_src_bytes, sum(src_bytes)/sum(src_pkts) as from_src_avg_pktsize, sum(dst_bytes)/1048576 as from_dst_bytes, sum(dst_bytes)/sum(dst_pkts) as from_dst_avg_pktsize, SUM(duration) as my_total_duration, SUM(duration)/COUNT(*) as my_avg_duration FROM sancp IGNORE INDEX (p_key) INNER JOIN sensor ON sancp.sid=sensor.sid WHERE start_time > %%STARTTIME%% AND end_time < %%ENDTIME%% AND %%SENSORS%% AND ip_proto = 1 AND ((src_ip between INET_ATON("192.168.0.0") and INET_ATON("192.168.255.255")) or (src_ip between INET_ATON("10.0.0.0") and INET_ATON("10.255.255.255")) or (src_ip between INET_ATON("172.16.0.0") and INET_ATON("172.31.255.255"))) and ((dst_ip not between INET_ATON("192.168.0.0") and INET_ATON("192.168.255.255")) and (dst_ip not between INET_ATON("10.0.0.0") and INET_ATON("10.255.255.255")) and (dst_ip not between INET_ATON("172.16.0.0") and INET_ATON("172.31.255.255"))) GROUP BY src_ip,dst_ip ORDER BY from_src_avg_pktsize DESC,from_src_bytes DESC LIMIT 10||8||

I also added ‘IGNORE INDEX (p_key)’. I saw default queries in Sguil use that, and it seems to improve performance. On my data these queries might return results in a few seconds, one user working with a bigger set of data reported it running an hour and a half… oops! Let’s see if this works.

A logical next step will be to create a query that actually compares the to_src and to_dst values average packet sizes and total transfered volume. I hope to get to that in a few days.

Btw, if you find this stuff interesting, i suggest you take a look at Extrusion Detection by Richard Bejtlich.