Sguil: adding support for ModSecurity alerts

I’m a huge fan of both Sguil and ModSecurity, but sadly the alerts generated by ModSecurity can’t show up in Sguil… or… can they? Well, if it all works out, soon they can!

Today I have hacked together a perl script that emulates barnyard for ModSecurity. It very much in a proof-of-concept phase, but it somewhat works already, at least good enough so i can show this screenshot.

Sguil screenshot showing experimental Mod_Security support
Nice huh? It is not ready for release yet, but when it is i’ll announce it here. I plan to release it under the GPL. Sguil author Bamm Visscher told me that the next release of Sguil will have support for having barnyard and PADS on the same sensor. By then, i hope that ModSecurity can be added to that list! 🙂

Sguil: detecting ICMP tunnels, continued

A few days ago i wrote about detecting ICMP tunnels. I came to the conclusion that for the two tunnels i tried the properties to detect on were:

  1. Non-standard average packet size. E.g. 81.81 bytes for the first connection is non-standard.
  2. Number of bytes in both directions are unequal.
  3. Average packet size in both directions is unequal.

I believe the third is the most important, because it must mean that 1 is also true and 2 is very likely to be true. If the average packet size is unequal in both directions, one of them should be non-standard. So far i have only seen echo-request and echo-reply being equal in size for pings. Unequal average packet sizes with exactly the same amount of data transfered is very unlikely, so i have ignored that for now.

This is the full report that shows all ICMP with unequal average packet sizes:

SUSPECTICMP||Suspect ICMP LAN->WAN: Top 10 ICMP users in the LAN talking to the WAN, with different packet sizes, with source and destination, volume and average packet size, sorted by packet size (fields: source ip, dest ip, from source MB, from source packet size, from dest MB, from dest packet size, total duration, average duration)||query||SELECT INET_NTOA(src_ip), INET_NTOA(dst_ip), my_src_bytes/1048576 AS my_src_mb, my_avg_src_pktsize, my_dst_bytes/1048576 AS my_dst_mb, my_avg_dst_pktsize, my_tot_duration, AVG(my_tot_duration) AS my_avg_duration FROM ( SELECT src_ip, dst_ip, ip_proto, SUM(src_bytes) AS my_src_bytes, SUM(dst_bytes) AS my_dst_bytes, start_time, end_time, SUM(src_bytes)/sum(src_pkts) AS my_avg_src_pktsize, SUM(dst_bytes)/sum(dst_pkts) AS my_avg_dst_pktsize, SUM(duration) AS my_tot_duration FROM ( SELECT sid AS my_sid, src_ip, dst_ip, ip_proto, src_bytes, dst_bytes, src_pkts, dst_pkts, start_time, end_time, duration FROM sancp IGNORE INDEX (p_key) WHERE ip_proto = 1 AND src_pkts > 0 AND dst_pkts > 0 AND start_time > %%STARTTIME%% AND end_time < %%ENDTIME%% ) AS step1_table INNER JOIN sensor ON my_sid=sensor.sid WHERE %%SENSORS%% AND ((src_ip between INET_ATON("192.168.0.0") and INET_ATON("192.168.255.255")) or (src_ip between INET_ATON("10.0.0.0") and INET_ATON("10.255.255.255")) or (src_ip between INET_ATON("172.16.0.0") and INET_ATON("172.31.255.255"))) and ((dst_ip not between INET_ATON("192.168.0.0") and INET_ATON("192.168.255.255")) and (dst_ip not between INET_ATON("10.0.0.0") and INET_ATON("10.255.255.255")) and (dst_ip not between INET_ATON("172.16.0.0") and INET_ATON("172.31.255.255"))) GROUP BY dst_bytes ) AS step2_table WHERE my_avg_src_pktsize != my_avg_dst_pktsize GROUP BY src_ip,dst_ip ORDER BY my_avg_dst_pktsize DESC, my_avg_src_pktsize DESC LIMIT 10||8||

This query only works on MySQL 4.1 and higher. It results in this report:

Sguil Sensor Report Showing an Suspect ICMP Report

This detects all tunnels I had setup, and shows nothing of the other non-tunnel tests I did, like the ping flood.

Sguil: using advanced queries to get more info on Snort events

Today, David Bianco showed me a way of creating SQL queries that I didn’t even know was possible. This is a technique with which it is possible to query the payload of Snort events in the Sguil database. These payloads are stored by Snort when it alerts and is the payload the actual rule triggered on. David showed a nice example of retrieving url’s for PHP url inclusion attacks.

I have written before about my usage of Mod_Security. I let Mod_Security respond with a 403 Forbidden message. Sadly, the alert generated by Mod_Security can not be displayed in Sguil. As somewhat of a replacement for that, I let Snort alert on the 403 Forbidden messages, so i can see in Sguil that Mod_Security blocked something. The disadvantage of this is that the 403 alert in itself does not contain much info. The sheer number of 403’s makes inspecting every single one with requesting a transcript a bit to much work.

This is where the new query comes in. The query creates a list of url’s that the server reported to be 403 Forbidden. Interesting is that we are not looking at an attack, but at an attack response. This means the the attackers IP is actually the destination IP.

SELECT COUNT(*) AS cnt, INET_NTOA(dst_ip) AS “Dest IP”, trim(LEADING “access ” FROM substring_index(substr(unhex(data_payload),locate(‘access ‘,unhex(data_payload))), ‘n’, 1)) AS url FROM event INNER JOIN data ON event.sid = data.sid and event.cid = data.cid where (signature like “%ATTACK-RESPONSES 403 Forbidden%”) GROUP BY dst_ip,url ORDER BY cnt DESC LIMIT 10;

The result is this:

MySQL result for query for 403 Forbidden event url's

As you can see, the first six results are from the comment spam I wrote about earlier. I left the source IP out because there is only one webserver. The query can easily be extended to show source IP’s as well.

Sguil: detecting ICMP tunnels

My earlier post about detecting ICMP anomolies was based on dry theory combined with experimenting with the ping command. The last couple of days I have been playing with real ICMP tunnels, to see how detection of those would work. This was easier said than done. Sure, running PingTunnel or itun between two hosts in my LAN worked fine. However, being an inline guy, i have Sancp looking at traffic passing my firewall only. And getting the ICMP tunnels to pass the firewall was the real trick. I won’t bore you with that now, because i intend to look at counter measures later, so i’ll handle that then. For now I will just assume that these ping tunnels will not be blocked by the firewall.

The query is largely the same as in my earlier post about detecting ICMP anomolies. However, i made one addition. Sancp also stores session duration information. In other words, how long did a session last. I figured that ICMP tunnels may have longer than average duractions, since they are likely to be used for hiding interactive sessions like ssh, irc or msn. One should remember however, that a continues ping command that (accidently) runs for a long period of time will also have a long duration.

Sguil Sensor Report Showing an ICMP Report

Five results. Three of which are (partly) ICMP tunnel traffic. Spotting them is easy. They all have the following properties:

  1. Non-standard average packet size. E.g. 81.81 bytes for the first connection is non-standard.
  2. Number of bytes in both directions are unequal.
  3. Average packet size in both directions is unequal.

The number of bytes transfered seems to be not a really good indicator in itself. The second result transfered 29 MB, the biggest number from these results, but was no tunnel. It was a ‘ping -f 192.168.0.102’ for a few minutes. The same goes for the duration. Sure the first tunnel has by far the longest duration, but if i keep a ping session running for a week it won’t misteriously turn into a tunnel 😉

Still, i think it is useful to have the information around because once you suspect a tunnel exists, volume and duration might be further indicators something is wrong.

This is the Sguil report:

TOPTENICMP||Top 10 ICMP LAN->WAN: Top 10 ICMP users in the LAN talking to the WAN, with source and destination, volume and average packet size, sorted by packet size (fields: source ip, dest ip, from source MB, from source packet size, from dest MB, from dest packet size, total duration, average duration)||query||select INET_NTOA(src_ip), INET_NTOA(dst_ip), sum(src_bytes)/1048576 as from_src_bytes, sum(src_bytes)/sum(src_pkts) as from_src_avg_pktsize, sum(dst_bytes)/1048576 as from_dst_bytes, sum(dst_bytes)/sum(dst_pkts) as from_dst_avg_pktsize, SUM(duration) as my_total_duration, SUM(duration)/COUNT(*) as my_avg_duration FROM sancp IGNORE INDEX (p_key) INNER JOIN sensor ON sancp.sid=sensor.sid WHERE start_time > %%STARTTIME%% AND end_time < %%ENDTIME%% AND %%SENSORS%% AND ip_proto = 1 AND ((src_ip between INET_ATON("192.168.0.0") and INET_ATON("192.168.255.255")) or (src_ip between INET_ATON("10.0.0.0") and INET_ATON("10.255.255.255")) or (src_ip between INET_ATON("172.16.0.0") and INET_ATON("172.31.255.255"))) and ((dst_ip not between INET_ATON("192.168.0.0") and INET_ATON("192.168.255.255")) and (dst_ip not between INET_ATON("10.0.0.0") and INET_ATON("10.255.255.255")) and (dst_ip not between INET_ATON("172.16.0.0") and INET_ATON("172.31.255.255"))) GROUP BY src_ip,dst_ip ORDER BY from_src_avg_pktsize DESC,from_src_bytes DESC LIMIT 10||8||

I also added ‘IGNORE INDEX (p_key)’. I saw default queries in Sguil use that, and it seems to improve performance. On my data these queries might return results in a few seconds, one user working with a bigger set of data reported it running an hour and a half… oops! Let’s see if this works.

A logical next step will be to create a query that actually compares the to_src and to_dst values average packet sizes and total transfered volume. I hope to get to that in a few days.

Btw, if you find this stuff interesting, i suggest you take a look at Extrusion Detection by Richard Bejtlich.

ModSecurity: rules against comment spam

Lately the wiki of my Vuurmuur project has been receiving quite a lot of comment spam. Although removing the spam manually is boring work, i still don’t really mind the spam, because it enables me to practice with ModSecurity rules to fight it off. So far, the spam seems to be following a pattern, in which the spam is posted by bots, and has the same general layout for longer periods of time. That makes it worthwhile to spend time on creating rules against it. Yesterday a new type of spam emerged on the wiki. The following audit_log is for one of them. I had to slightly edit it for layout reasons.

–9075cb7a-A–
[22/Aug/2006:20:20:46 +0200] SPO4w5FhwZUAADItDEgAAAAC 195.225.177.131 34189 192.168.1.101 80
–9075cb7a-B–
POST /tiki/tiki-index.php HTTP/1.1
Host: wiki.vuurmuur.org
Referer: http://wiki.vuurmuur.org/tiki/tiki-index.php?page=Vuurmuur
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Content-Length: 1304
Content-Type: application/x-www-form-urlencoded

–9075cb7a-C–
page=Vuurmuur&cols=60&comments_sort_mode=points_desc&
comments_parentId=0&rows=6&comments_postComment=
post&comments_threshold=0&comments_threadId=0&
comments_data=Chevrolet+-+%5BURL%3Dhttp%3A%2F%2F
canonprnbf.kickme.to%2F+%5D+Chevrolet+%5B%2FURL%5D+
home+based+businesses+-+%5BURL%3Dhttp%3A%2F%2F
1964cadulj.blog.kataweb.it%2F1964cadulj%2Findex-home-
based-businesses.html+%5D+home+based+businesses+
%5B%2FURL%5D+diet+pills+-+%5BURL%3Dhttp%3A%2F%2F
aemerge4li.web1000.com%2Findex-diet-pills.html+%5D+diet+
pills+%5B%2FURL%5D+Cash+Advance+-+%5BURL%3D
http%3A%2F%2Fseamlesmnp.su.pl%2F+%5D+Cash+Advance
+%5B%2FURL%5D+Azithromycin+-+%5BURL%3D
http%3A%2F%2Fthelacefab7ne.kickme.to%2F+%5D+
Azithromycin+%5B%2FURL%5D+allergy+-+%5BURL%3D
http%3A%2F%2Fgeo.ya.com%2Faeggcelpzg%2Findex-
allergy.html+%5D+allergy+%5B%2FURL%5D+Cash+Advance+
-+%5BURL%3Dhttp%3A%2F%2Faofficeuov.ir.pl%2F+%5D+
Cash+Advance+%5B%2FURL%5D+free+games+-+%5BURL
%3Dhttp%3A%2F%2Fwww.geocities.com%2Fthebestjwb
%2Findex-free-games.html+%5D+free+games+%5B%2F
URL%5D+Ambien+-+%5BURL%3Dhttp%3A%2F%2F
beadedluwu.su.pl%2F+%5D+Ambien+%5B%2FURL%5D+altace
+-+%5BURL%3Dhttp%3A%2F%2Facellphonehy4.tripod.com
%2Findex-altace.html+%5D+altace+%5B%2FURL%5D
+&comments_reply_threadId=0&comments_title=Chevrolet&
comments_offset=0&comments_previewComment=preview&
comments_grandParentId=
–9075cb7a-F–
HTTP/1.1 200 OK
X-Powered-By: PHP/4.3.10-16
Set-Cookie: PHPSESSID=b2497fd593f56c5af4f3613ba78a7619; path=/tiki
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html; charset=utf-8

–9075cb7a-H–
Stopwatch: 1156270844524739 1475657 (17717* 38503 0)
Producer: ModSecurity v1.9.2 (Apache 2.x)
Server: Apache/2.0.54 (Debian GNU/Linux) PHP/4.3.10-16 mod_ssl/2.0.54 OpenSSL/0.9.7e

–9075cb7a-Z–

A general rule against the comment spam would probably not be that hard. Just blocking “http://&#8221; would probably to the trick. However, I’m not quite willing to do this, since people might actually post real and interesting links to the wiki. I noticed that the above comment and the 30 other already posted messages all contained the uppercase word URL, so i decided to block on that. Below are the rules, which as you can see are quite an improvement over these rules, because they now only match on actual comments posts 😉

SecFilterSelective REQUEST_URI “/tiki/tiki-index.php” “chain,msg:’LOCAL comment spam'”
SecFilterSelective POST_PAYLOAD “URL” chain
SecFilterSelective POST_PAYLOAD “comments_postComment=post” log,deny,status:403

An Apache restart and just wait… but not for long:

[Tue Aug 22 21:35:44 2006] [error] [client 195.225.177.131] mod_security: Access denied with code 403. Pattern match “comments_postComment=post” at POST_PAYLOAD [msg “LOCAL comment spam”] [hostname “wiki.vuurmuur.org”] [uri “/tiki/tiki-index.php”] [unique_id “VTGBGJFhwZUAADMnAWAAAAAA”]

This morning there were 59 attempts blocked already which i would have to remove manually without these rules. So taking the time to setup the rules really pays off.

Sguil: detecting ICMP anomolies

Recently Websense discovered malware that caused quite a lot of media attention, because it used ICMP to send personal data out of the victims pc or network. In many networks, outgoing ICMP is unrestricted because it can aid in solving connectivity and routing problems. While (partially) blocking outgoing ICMP would certainly solve this problem, not everone will be willing to do so. Note however that the idea of using ICMP as a covert channel is hardly new.

Yesterday I wrote about creating custom Sancp queries and adding them to Sguil as canned reports. Today i tried to create a query that can be used to detect anomolies in ICMP traffic leaving your network. Like yesterday, i did this in the form of a ‘canned report’. One limitation is that there seems to be no way to get any information on exact ICMP codes and types.

I tried to create a query that would do the following. I assumed that there are at least two ways the traffic can stand out. Either by it’s volume, or by non-default packet sizes. Normally ping uses a fixed packet size of 64 bytes on my Linux system, Windows 2000 defaults to 32 bytes. If the packet size is not what you would expect, something fishy might be going on. So the query tries to find the execptional cases in this respect.

Volume is another factor. To tranfser substantial amounts of data through ICMP, the volume of the packets will need to be quite high as well. Note however that Linux’ ping utility for example pings until you tell it to stop, and it happened to me before that i accidently left it running an entire night, so a large volume doesn’t have to mean there is something wrong. The target IP is also very interesting. If it is IP belonging to Google there is a big chance that it is just a test. In the example below most volume is to 194.109.21.51 which is the server in my ISP’s network that can be reached under ‘ping.xs4all.nl’.

Sguil Sensor Report Showing an ICMP Report

In this report you can see that 192.168.1.2 pinged 145.97.193.148 with an average packet size of 1480 bytes, which is certainly not normal. It transfered 0.08MB to this IP. Another amount of 2.24MB was transfered to the IP 194.109.21.51, with a average packet size of 64.33 bytes. This is weird because it is not exactly 64 bytes, what I would expect. This can be explained by me running a default ping for a long time and after that doing a few tests with larger sizes.

Continue reading

Sguil: creating custom reports

In the Sguil NSM system, Sancp plays a vital role. Sancp records session data, in which all connections are recorded. For all connections (and pseudo connections, think udp, icmp), Sancp records the number of bytes transfered, number of packets, start and end time, etc. This is very much useful information, of which Sguil only makes a subset accessable. Because the information is stored in a MySQL database, nothing prevents you from querying the database manually, which is what i did. However, David Bianco suggested that i could also add them as ‘canned reports’ to Sguil, which i did.

At this stage i am mostly interested in the information from Sancp about the traffic volume. Which host(s) use the most bandwidth? Questions like this. Below i explain one of the ‘canned reports’ i created.

Report View

This is the output of the query below. What it shows is that 192.168.1.2 in my lan downloaded 367MB from 145.97.193.148, and that it did this via http (protocol 6 is tcp, port is 80). Let’s have a look at the query.

TOPTENWAN2LAN_SRC_DST_SER||Top b/w per serv. from WAN to LAN (downloads), flow from WAN-IPs to LAN-IPs, volume in MB||query||select sum(dst_bytes)/1048576 as my_dst_bytes, INET_NTOA(dst_ip), INET_NTOA(src_ip), ip_proto, dst_port from sancp INNER JOIN sensor ON sancp.sid=sensor.sid WHERE start_time > %%STARTTIME%% AND end_time 0 AND %%SENSORS%% and ((src_ip between INET_ATON(“192.168.0.0”) and INET_ATON(“192.168.255.255”)) or (src_ip between INET_ATON(“10.0.0.0”) and INET_ATON(“10.255.255.255”)) or (src_ip between INET_ATON(“172.16.0.0”) and INET_ATON(“172.31.255.255”))) and ((dst_ip not between INET_ATON(“192.168.0.0”) and INET_ATON(“192.168.255.255”)) and (dst_ip not between INET_ATON(“10.0.0.0”) and INET_ATON(“10.255.255.255”)) and (dst_ip not between INET_ATON(“172.16.0.0”) and INET_ATON(“172.31.255.255”))) GROUP BY src_ip,dst_ip,ip_proto,dst_port ORDER BY my_dst_bytes DESC LIMIT 10||5||

Copy-paste this into your /etc/sguild/sguild.reports and restart sguild. It should be available in the Sensor Reports window. The parts between percent signs are Sguil variables. The double pipes are separators between different sections of the line. The line starts with a sort of internal name for Sguil. What follows is the description as it will be show in the report selection screen and in the report itself. Next it is indicated that his is a query. After that the query itself. After the query in the last field it is indicated how many columns of data Sguil can expect from MySQL.

Now the query. First, i select dst_bytes, dst_ip and src_ip, ip_proto and dst_port. The dst_bytes field contains the number of bytes flowing from dst_ip to src_ip. Because we use group later, we SUM dst_bytes. And because it is in bytes, we divide it by 1024*1024(=1048576) so the result will be megabytes. The result is stored in my_dst_bytes. We then make sure that the src_ip is in a private ip range and that dst_ip is not in a private ip-range. Then the grouping is done, followed by the ordering by my_dst_bytes. Easy huh!

Naturally, this only works for networks that actually use private ip-ranges, but it should be easy to adapt if you use something else. Below, i have added another seven of these reports.

Continue reading