Sguil: adding support for ModSecurity alerts

I’m a huge fan of both Sguil and ModSecurity, but sadly the alerts generated by ModSecurity can’t show up in Sguil… or… can they? Well, if it all works out, soon they can!

Today I have hacked together a perl script that emulates barnyard for ModSecurity. It very much in a proof-of-concept phase, but it somewhat works already, at least good enough so i can show this screenshot.

Sguil screenshot showing experimental Mod_Security support
Nice huh? It is not ready for release yet, but when it is i’ll announce it here. I plan to release it under the GPL. Sguil author Bamm Visscher told me that the next release of Sguil will have support for having barnyard and PADS on the same sensor. By then, i hope that ModSecurity can be added to that list! 🙂

Sguil: detecting ICMP tunnels, continued

A few days ago i wrote about detecting ICMP tunnels. I came to the conclusion that for the two tunnels i tried the properties to detect on were:

  1. Non-standard average packet size. E.g. 81.81 bytes for the first connection is non-standard.
  2. Number of bytes in both directions are unequal.
  3. Average packet size in both directions is unequal.

I believe the third is the most important, because it must mean that 1 is also true and 2 is very likely to be true. If the average packet size is unequal in both directions, one of them should be non-standard. So far i have only seen echo-request and echo-reply being equal in size for pings. Unequal average packet sizes with exactly the same amount of data transfered is very unlikely, so i have ignored that for now.

This is the full report that shows all ICMP with unequal average packet sizes:

SUSPECTICMP||Suspect ICMP LAN->WAN: Top 10 ICMP users in the LAN talking to the WAN, with different packet sizes, with source and destination, volume and average packet size, sorted by packet size (fields: source ip, dest ip, from source MB, from source packet size, from dest MB, from dest packet size, total duration, average duration)||query||SELECT INET_NTOA(src_ip), INET_NTOA(dst_ip), my_src_bytes/1048576 AS my_src_mb, my_avg_src_pktsize, my_dst_bytes/1048576 AS my_dst_mb, my_avg_dst_pktsize, my_tot_duration, AVG(my_tot_duration) AS my_avg_duration FROM ( SELECT src_ip, dst_ip, ip_proto, SUM(src_bytes) AS my_src_bytes, SUM(dst_bytes) AS my_dst_bytes, start_time, end_time, SUM(src_bytes)/sum(src_pkts) AS my_avg_src_pktsize, SUM(dst_bytes)/sum(dst_pkts) AS my_avg_dst_pktsize, SUM(duration) AS my_tot_duration FROM ( SELECT sid AS my_sid, src_ip, dst_ip, ip_proto, src_bytes, dst_bytes, src_pkts, dst_pkts, start_time, end_time, duration FROM sancp IGNORE INDEX (p_key) WHERE ip_proto = 1 AND src_pkts > 0 AND dst_pkts > 0 AND start_time > %%STARTTIME%% AND end_time < %%ENDTIME%% ) AS step1_table INNER JOIN sensor ON my_sid=sensor.sid WHERE %%SENSORS%% AND ((src_ip between INET_ATON("") and INET_ATON("")) or (src_ip between INET_ATON("") and INET_ATON("")) or (src_ip between INET_ATON("") and INET_ATON(""))) and ((dst_ip not between INET_ATON("") and INET_ATON("")) and (dst_ip not between INET_ATON("") and INET_ATON("")) and (dst_ip not between INET_ATON("") and INET_ATON(""))) GROUP BY dst_bytes ) AS step2_table WHERE my_avg_src_pktsize != my_avg_dst_pktsize GROUP BY src_ip,dst_ip ORDER BY my_avg_dst_pktsize DESC, my_avg_src_pktsize DESC LIMIT 10||8||

This query only works on MySQL 4.1 and higher. It results in this report:

Sguil Sensor Report Showing an Suspect ICMP Report

This detects all tunnels I had setup, and shows nothing of the other non-tunnel tests I did, like the ping flood.

Sguil: using advanced queries to get more info on Snort events

Today, David Bianco showed me a way of creating SQL queries that I didn’t even know was possible. This is a technique with which it is possible to query the payload of Snort events in the Sguil database. These payloads are stored by Snort when it alerts and is the payload the actual rule triggered on. David showed a nice example of retrieving url’s for PHP url inclusion attacks.

I have written before about my usage of Mod_Security. I let Mod_Security respond with a 403 Forbidden message. Sadly, the alert generated by Mod_Security can not be displayed in Sguil. As somewhat of a replacement for that, I let Snort alert on the 403 Forbidden messages, so i can see in Sguil that Mod_Security blocked something. The disadvantage of this is that the 403 alert in itself does not contain much info. The sheer number of 403’s makes inspecting every single one with requesting a transcript a bit to much work.

This is where the new query comes in. The query creates a list of url’s that the server reported to be 403 Forbidden. Interesting is that we are not looking at an attack, but at an attack response. This means the the attackers IP is actually the destination IP.

SELECT COUNT(*) AS cnt, INET_NTOA(dst_ip) AS “Dest IP”, trim(LEADING “access ” FROM substring_index(substr(unhex(data_payload),locate(‘access ‘,unhex(data_payload))), ‘n’, 1)) AS url FROM event INNER JOIN data ON event.sid = data.sid and event.cid = data.cid where (signature like “%ATTACK-RESPONSES 403 Forbidden%”) GROUP BY dst_ip,url ORDER BY cnt DESC LIMIT 10;

The result is this:

MySQL result for query for 403 Forbidden event url's

As you can see, the first six results are from the comment spam I wrote about earlier. I left the source IP out because there is only one webserver. The query can easily be extended to show source IP’s as well.

Sguil: detecting ICMP tunnels

My earlier post about detecting ICMP anomolies was based on dry theory combined with experimenting with the ping command. The last couple of days I have been playing with real ICMP tunnels, to see how detection of those would work. This was easier said than done. Sure, running PingTunnel or itun between two hosts in my LAN worked fine. However, being an inline guy, i have Sancp looking at traffic passing my firewall only. And getting the ICMP tunnels to pass the firewall was the real trick. I won’t bore you with that now, because i intend to look at counter measures later, so i’ll handle that then. For now I will just assume that these ping tunnels will not be blocked by the firewall.

The query is largely the same as in my earlier post about detecting ICMP anomolies. However, i made one addition. Sancp also stores session duration information. In other words, how long did a session last. I figured that ICMP tunnels may have longer than average duractions, since they are likely to be used for hiding interactive sessions like ssh, irc or msn. One should remember however, that a continues ping command that (accidently) runs for a long period of time will also have a long duration.

Sguil Sensor Report Showing an ICMP Report

Five results. Three of which are (partly) ICMP tunnel traffic. Spotting them is easy. They all have the following properties:

  1. Non-standard average packet size. E.g. 81.81 bytes for the first connection is non-standard.
  2. Number of bytes in both directions are unequal.
  3. Average packet size in both directions is unequal.

The number of bytes transfered seems to be not a really good indicator in itself. The second result transfered 29 MB, the biggest number from these results, but was no tunnel. It was a ‘ping -f’ for a few minutes. The same goes for the duration. Sure the first tunnel has by far the longest duration, but if i keep a ping session running for a week it won’t misteriously turn into a tunnel 😉

Still, i think it is useful to have the information around because once you suspect a tunnel exists, volume and duration might be further indicators something is wrong.

This is the Sguil report:

TOPTENICMP||Top 10 ICMP LAN->WAN: Top 10 ICMP users in the LAN talking to the WAN, with source and destination, volume and average packet size, sorted by packet size (fields: source ip, dest ip, from source MB, from source packet size, from dest MB, from dest packet size, total duration, average duration)||query||select INET_NTOA(src_ip), INET_NTOA(dst_ip), sum(src_bytes)/1048576 as from_src_bytes, sum(src_bytes)/sum(src_pkts) as from_src_avg_pktsize, sum(dst_bytes)/1048576 as from_dst_bytes, sum(dst_bytes)/sum(dst_pkts) as from_dst_avg_pktsize, SUM(duration) as my_total_duration, SUM(duration)/COUNT(*) as my_avg_duration FROM sancp IGNORE INDEX (p_key) INNER JOIN sensor ON sancp.sid=sensor.sid WHERE start_time > %%STARTTIME%% AND end_time < %%ENDTIME%% AND %%SENSORS%% AND ip_proto = 1 AND ((src_ip between INET_ATON("") and INET_ATON("")) or (src_ip between INET_ATON("") and INET_ATON("")) or (src_ip between INET_ATON("") and INET_ATON(""))) and ((dst_ip not between INET_ATON("") and INET_ATON("")) and (dst_ip not between INET_ATON("") and INET_ATON("")) and (dst_ip not between INET_ATON("") and INET_ATON(""))) GROUP BY src_ip,dst_ip ORDER BY from_src_avg_pktsize DESC,from_src_bytes DESC LIMIT 10||8||

I also added ‘IGNORE INDEX (p_key)’. I saw default queries in Sguil use that, and it seems to improve performance. On my data these queries might return results in a few seconds, one user working with a bigger set of data reported it running an hour and a half… oops! Let’s see if this works.

A logical next step will be to create a query that actually compares the to_src and to_dst values average packet sizes and total transfered volume. I hope to get to that in a few days.

Btw, if you find this stuff interesting, i suggest you take a look at Extrusion Detection by Richard Bejtlich.

ModSecurity: rules against comment spam

Lately the wiki of my Vuurmuur project has been receiving quite a lot of comment spam. Although removing the spam manually is boring work, i still don’t really mind the spam, because it enables me to practice with ModSecurity rules to fight it off. So far, the spam seems to be following a pattern, in which the spam is posted by bots, and has the same general layout for longer periods of time. That makes it worthwhile to spend time on creating rules against it. Yesterday a new type of spam emerged on the wiki. The following audit_log is for one of them. I had to slightly edit it for layout reasons.

[22/Aug/2006:20:20:46 +0200] SPO4w5FhwZUAADItDEgAAAAC 34189 80
POST /tiki/tiki-index.php HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Content-Length: 1304
Content-Type: application/x-www-form-urlencoded

HTTP/1.1 200 OK
X-Powered-By: PHP/4.3.10-16
Set-Cookie: PHPSESSID=b2497fd593f56c5af4f3613ba78a7619; path=/tiki
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html; charset=utf-8

Stopwatch: 1156270844524739 1475657 (17717* 38503 0)
Producer: ModSecurity v1.9.2 (Apache 2.x)
Server: Apache/2.0.54 (Debian GNU/Linux) PHP/4.3.10-16 mod_ssl/2.0.54 OpenSSL/0.9.7e


A general rule against the comment spam would probably not be that hard. Just blocking “http://&#8221; would probably to the trick. However, I’m not quite willing to do this, since people might actually post real and interesting links to the wiki. I noticed that the above comment and the 30 other already posted messages all contained the uppercase word URL, so i decided to block on that. Below are the rules, which as you can see are quite an improvement over these rules, because they now only match on actual comments posts 😉

SecFilterSelective REQUEST_URI “/tiki/tiki-index.php” “chain,msg:’LOCAL comment spam'”
SecFilterSelective POST_PAYLOAD “URL” chain
SecFilterSelective POST_PAYLOAD “comments_postComment=post” log,deny,status:403

An Apache restart and just wait… but not for long:

[Tue Aug 22 21:35:44 2006] [error] [client] mod_security: Access denied with code 403. Pattern match “comments_postComment=post” at POST_PAYLOAD [msg “LOCAL comment spam”] [hostname “”] [uri “/tiki/tiki-index.php”] [unique_id “VTGBGJFhwZUAADMnAWAAAAAA”]

This morning there were 59 attempts blocked already which i would have to remove manually without these rules. So taking the time to setup the rules really pays off.

Sguil: detecting ICMP anomolies

Recently Websense discovered malware that caused quite a lot of media attention, because it used ICMP to send personal data out of the victims pc or network. In many networks, outgoing ICMP is unrestricted because it can aid in solving connectivity and routing problems. While (partially) blocking outgoing ICMP would certainly solve this problem, not everone will be willing to do so. Note however that the idea of using ICMP as a covert channel is hardly new.

Yesterday I wrote about creating custom Sancp queries and adding them to Sguil as canned reports. Today i tried to create a query that can be used to detect anomolies in ICMP traffic leaving your network. Like yesterday, i did this in the form of a ‘canned report’. One limitation is that there seems to be no way to get any information on exact ICMP codes and types.

I tried to create a query that would do the following. I assumed that there are at least two ways the traffic can stand out. Either by it’s volume, or by non-default packet sizes. Normally ping uses a fixed packet size of 64 bytes on my Linux system, Windows 2000 defaults to 32 bytes. If the packet size is not what you would expect, something fishy might be going on. So the query tries to find the execptional cases in this respect.

Volume is another factor. To tranfser substantial amounts of data through ICMP, the volume of the packets will need to be quite high as well. Note however that Linux’ ping utility for example pings until you tell it to stop, and it happened to me before that i accidently left it running an entire night, so a large volume doesn’t have to mean there is something wrong. The target IP is also very interesting. If it is IP belonging to Google there is a big chance that it is just a test. In the example below most volume is to which is the server in my ISP’s network that can be reached under ‘’.

Sguil Sensor Report Showing an ICMP Report

In this report you can see that pinged with an average packet size of 1480 bytes, which is certainly not normal. It transfered 0.08MB to this IP. Another amount of 2.24MB was transfered to the IP, with a average packet size of 64.33 bytes. This is weird because it is not exactly 64 bytes, what I would expect. This can be explained by me running a default ping for a long time and after that doing a few tests with larger sizes.

Continue reading

Sguil: creating custom reports

In the Sguil NSM system, Sancp plays a vital role. Sancp records session data, in which all connections are recorded. For all connections (and pseudo connections, think udp, icmp), Sancp records the number of bytes transfered, number of packets, start and end time, etc. This is very much useful information, of which Sguil only makes a subset accessable. Because the information is stored in a MySQL database, nothing prevents you from querying the database manually, which is what i did. However, David Bianco suggested that i could also add them as ‘canned reports’ to Sguil, which i did.

At this stage i am mostly interested in the information from Sancp about the traffic volume. Which host(s) use the most bandwidth? Questions like this. Below i explain one of the ‘canned reports’ i created.

Report View

This is the output of the query below. What it shows is that in my lan downloaded 367MB from, and that it did this via http (protocol 6 is tcp, port is 80). Let’s have a look at the query.

TOPTENWAN2LAN_SRC_DST_SER||Top b/w per serv. from WAN to LAN (downloads), flow from WAN-IPs to LAN-IPs, volume in MB||query||select sum(dst_bytes)/1048576 as my_dst_bytes, INET_NTOA(dst_ip), INET_NTOA(src_ip), ip_proto, dst_port from sancp INNER JOIN sensor ON sancp.sid=sensor.sid WHERE start_time > %%STARTTIME%% AND end_time 0 AND %%SENSORS%% and ((src_ip between INET_ATON(“”) and INET_ATON(“”)) or (src_ip between INET_ATON(“”) and INET_ATON(“”)) or (src_ip between INET_ATON(“”) and INET_ATON(“”))) and ((dst_ip not between INET_ATON(“”) and INET_ATON(“”)) and (dst_ip not between INET_ATON(“”) and INET_ATON(“”)) and (dst_ip not between INET_ATON(“”) and INET_ATON(“”))) GROUP BY src_ip,dst_ip,ip_proto,dst_port ORDER BY my_dst_bytes DESC LIMIT 10||5||

Copy-paste this into your /etc/sguild/sguild.reports and restart sguild. It should be available in the Sensor Reports window. The parts between percent signs are Sguil variables. The double pipes are separators between different sections of the line. The line starts with a sort of internal name for Sguil. What follows is the description as it will be show in the report selection screen and in the report itself. Next it is indicated that his is a query. After that the query itself. After the query in the last field it is indicated how many columns of data Sguil can expect from MySQL.

Now the query. First, i select dst_bytes, dst_ip and src_ip, ip_proto and dst_port. The dst_bytes field contains the number of bytes flowing from dst_ip to src_ip. Because we use group later, we SUM dst_bytes. And because it is in bytes, we divide it by 1024*1024(=1048576) so the result will be megabytes. The result is stored in my_dst_bytes. We then make sure that the src_ip is in a private ip range and that dst_ip is not in a private ip-range. Then the grouping is done, followed by the ordering by my_dst_bytes. Easy huh!

Naturally, this only works for networks that actually use private ip-ranges, but it should be easy to adapt if you use something else. Below, i have added another seven of these reports.

Continue reading

ModSecurity: more security by obscurity

Yesterday, Philippe Baumgart showed me that my obscurity setup is not yet perfect. In fact, he could very easily enter an URL that didn’t exist and caused the webserver behind my proxy to respond with a 404. In this 404 the name and the version of the webserver were exposed.

After some testing i found that adding the following to my config worked very well.

# enable output scanning in Mod Security.
SecFilterScanOutput On

# hide outgoing 404 by webserver behind proxy
SecFilterSelective OUTPUT_STATUS 404 deny,status:404

This catches outgoing 404 errors, and replaces them by the 404 from the proxy. For some reason, this still didn’t exaclty look like the 404 from the proxy itself, because it contained a message that an additional 404 was encountered. I solved this by changing the ErrorDocument in de Apache config:

ErrorDocument 404 “<html><head><title>404 Not Found</title></head><body><h1>Not Found</h1><p>The requested URL was not found on this server.</p></body></html>”

After this, there was no longer any difference between 404’s produced by the proxy and by the webserver behind it.

Next, Phil showed me that i also leaked my version number of PHP. By using WordPress hiding the fact that I use PHP is impossible and pointless, but hiding the exact version still looks like a good idea. The version was leaked in the header from a server response: X-Powered-By: PHP/4.4.1build1. Solving this requires the mod_header module again:

# unset X-Powered-By to prevent leaking the PHP version
Header unset X-Powered-By

This hides this header. Thanks to Phil for doing some pen-testing 🙂

Vuurmuur: first baby steps in traffic shaping

Quite a while ago a placed a poll on the Vuurmuur Wiki, asking for the most important feature Vuurmuur needs. It turns out most people want traffic shaping. Traffic shaping has been on my todo list for a long time, but i never really got into using it, let alone understand it enough to integrate it into a GUI. So the last couple of days i had some spare time, and i have been checking it out. So far i am distinguishing the following types of traffic shaping.

First there is simple priorization: my VoIP/Gaming traffic should have priority over people up- and downloading ISO images. This is quite simple, and could be done using setting the ToS and creating some rules for that. In Vuurmuur, setting ToS could for example be done in a service.

Second, there is some more advanced priorization, with soft limits. This could for example be used in a situation where you say VoIP/Gaming should have a guaranteed bandwith of 20kb/s, but if on VoIP is active or not using the full 20kb/s, other traffic is allowed to use the 20kb/s in addition to the other bandwidth. Most important here is to keep interactive traffic responsive.

Third, the hard limits. For example saying that p2p should never be able to use more than 40kb/s, no matter how many bandwidth you have available. This could also be useful for assigning speed limits to specific hosts, for example in hosting companies.

Fourth and last, there is some misc stuff for which shaping can be useful, like limiting the rate of SYN packets, giving smaller packets priority over larger, etc.

Okay, on to my baby steps:

$TC qdisc add dev $DEV root handle 1: htb default 30

This creates a htb root qdisc, which defaults to class 1:30.

$TC class add dev $DEV parent 1: classid 1:1 htb rate $MAX_PPP_UP

Here we create the class, where we define the max speed of this class to be MAX_PPP_UP.

$TC class add dev $DEV parent 1:1 classid 1:10 htb rate 100kbit ceil $MAX_PPP_UP burst 4k prio 1
$TC class add dev $DEV parent 1:1 classid 1:20 htb rate $MAX_PPP_UP_BULK burst 0k prio 2
$TC class add dev $DEV parent 1:1 classid 1:30 htb rate 200kbit burst 1k prio 3
$TC class add dev $DEV parent 1:1 classid 1:40 htb rate 100kbit burst 0k prio 4

Here we define the actual limits. The first rule gets 100kbit guaranteed, but as a ceiling of the maximum speed. It has the highest priority. The second class is for bulk uploading, and it is set to about 70% of the maximum upload speed. With this set, i can upload at about 40kb/s, and still have a ping of around 80-100ms, while setting this higher will cause it to go into the 200+ms levels very quickly. I’ve also disabled burst for this class, because i don’t want ping spikes. The other two have hard limits, and can be used to really rate limit some traffic.

Like stated above, unclassified traffic gets into the default class 1:30. To have some traffic go into another class, i’m using the iptables CLASSIFY target:

# icmp gets high prio
$IPTABLES -t mangle -A FORWARD -i eth0 -o ppp0 -p icmp -j CLASSIFY –set-class 1:10

# http data from dmz to wan is bulk
$IPTABLES -t mangle -A FORWARD -i eth1 -o ppp0 -p tcp –sport 80 -j CLASSIFY –set-class 1:20

I like the CLASSIFY iptables target over the tc filter rules because it enables me to use the netfilter connection tracking helpers. This way i could for example say:

$IPTABLES -t mangle -A FORWARD -i eth0 -o ppp0 -m helper –helper sip -j CLASSIFY –set-class 1:10

So this way all traffic related to the SIP VoIP connection (the RTP stream) gets into the highest priority class.

This is all egress shaping, the only shaping that works well, or so everyone says. Ingress shaping is not as easy, for a number of reasons. First, we have no real control over the rate of traffic being sent to us. Second, linux shaping is not as advanced for ingress shaping as it is for egress shaping. IMQ and later IFB have been invented to deal with this, but i have never looked at it yet.

However, in a gateway setup, egress shaping can do all we need. Traffic that comes in on my ppp0 interface, gets out on my eth1 interface. At this eth1 interface we can do egress shaping for incoming traffic coming in on ppp0.

This about sums up my first attempt at understanding and using traffic shaping. So far i think it works very well. The next big question will be how this can be integrated into Vuurmuur. If i get some ideas about this, i’ll post them here.

Oh yeah, if anyone is interested in participating in this process, i have setup a page on the wiki to help determine what we want and how we want to do it: here. Feel free to add to it, edit, etc.

ModSecurity: further improvements to the reverse proxy

People can reach my webserver in three ways: by my domain, by the hostname connected to my dsl, and by my ipaddress. What i now wanted is setup the proxy in such a way, that only people visiting would be proxied to the webserver.

Blocking requests that are IP based instead of name based have an important advantage. IP based requests are mostly used by scantools and other forms of malicious traffic that just attempt connecting to port 80 on large IP-ranges. So this way one should be able to keep a lot of crap like worm traffic out.

Implementing this turned out to be a little more complicated than i thought. Essentially the only way i got it working was by creating three virtual hosts and put the proxy configuration in the virtual host for The other two virtual hosts just deny access, so visiting them gives a 403.

Willam Metcalf told me that the server signature rewrite function from ModSecurity does not hide the original server signature in all cases. In normal 200 and 300 series responses, the original signature is still in the header. Enabling the ‘header’ module and adding this to your config, helps:

Header set Server “Apache”

This way people will know you run Apache, but won’t know which version.