Suricata profiling per keyword

Last week I’ve added some more profiling options to Suricata. It’s part of the current git master. It’s enabled only when --enable-profiling and then through the suricata.yaml:

profiling:
  # per keyword profiling
  keywords:
    enabled: yes
    filename: keyword_perf.log
    append: yes

This will output a table similar to below:

--------------------------------------------------------------------------
Date: 11/7/2013 -- 15:13:11
--------------------------------------------------------------------------
Stats for: total
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
threshold        355324491   190574   409      72276       1864.00     3625.00     1860.00    
content          1274592063  534328   196738   312321      2385.00     2424.00     2362.00    
pcre             56626031    11149    824      254562      5079.00     12234.00    4507.00    
byte_test        153287955   128254   32109    67989       1195.00     1658.00     1040.00    
byte_jump        3676404     2041     2041     15939       1801.00     1801.00     0.00       
flow             38276182    22842    22842    63987       1675.00     1675.00     0.00       
isdataat         580764      558      556      2427        1040.00     1040.00     1017.00    
dsize            2212029     2062     2061     3711        1072.00     1072.00     789.00     
flowbits         1677209     874      870      9873        1919.00     1923.00     884.00     
itype            1653        2        1        1386        826.00      267.00      1386.00    
icode            27383781    93827    2        25545       291.00      1021.00     291.00     
flags            192751968   245519   189709   255639      785.00      753.00      892.00     
urilen           6149297     6142     1099     28299       1001.00     1395.00     915.00     
byte_extract     143091      78       78       7743        1834.00     1834.00     0.00       
--------------------------------------------------------------------------
Stats for: packet
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
flow             38276182    22842    22842    63987       1675.00     1675.00     0.00       
dsize            2212029     2062     2061     3711        1072.00     1072.00     789.00     
flowbits         351171      294      290      5526        1194.00     1198.00     884.00     
itype            1653        2        1        1386        826.00      267.00      1386.00    
icode            27383781    93827    2        25545       291.00      1021.00     291.00     
flags            192751968   245519   189709   255639      785.00      753.00      892.00     
--------------------------------------------------------------------------
Stats for: packet/stream payload
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
content          1203990910  512902   183628   312321      2347.00     2365.00     2337.00    
pcre             28087301    6598     54       254562      4256.00     12279.00    4190.00    
byte_test        153287955   128254   32109    67989       1195.00     1658.00     1040.00    
byte_jump        3676404     2041     2041     15939       1801.00     1801.00     0.00       
isdataat         578172      556      554      2427        1039.00     1039.00     1017.00    
byte_extract     143091      78       78       7743        1834.00     1834.00     0.00       
--------------------------------------------------------------------------
Stats for: http uri
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
content          44775802    13102    8351     60993       3417.00     3257.00     3698.00    
pcre             18284421    3646     97       61338       5014.00     8916.00     4908.00    
isdataat         2592        2        2        1725        1296.00     1296.00     0.00       
urilen           6149297     6142     1099     28299       1001.00     1395.00     915.00     
--------------------------------------------------------------------------
Stats for: http raw uri
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
pcre             9534        2        0        4953        4767.00     0.00        4767.00    
--------------------------------------------------------------------------
Stats for: http client body
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
content          1556904     441      181      58476       3530.00     2874.00     3986.00    
pcre             63924       6        6        17358       10654.00    10654.00    0.00       
--------------------------------------------------------------------------
Stats for: http headers
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
content          23688244    7631     4348     31098       3104.00     3311.00     2829.00    
pcre             9998970     859      667      71904       11640.00    12727.00    7862.00    
--------------------------------------------------------------------------
Stats for: http stat code
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
content          80052       39       20       3699        2052.00     2199.00     1898.00    
--------------------------------------------------------------------------
Stats for: http method
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
content          476334      203      201      27240       2346.00     2351.00     1846.00    
--------------------------------------------------------------------------
Stats for: http cookie
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
content          23817       10       9        2763        2381.00     2384.00     2358.00    
pcre             181881      38       0        13095       4786.00     0.00        4786.00    
--------------------------------------------------------------------------
Stats for: post-match
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
flowbits         1326038     580      580      9873        2286.00     2286.00     0.00       
--------------------------------------------------------------------------
Stats for: threshold
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
threshold        355324491   190574   409      72276       1864.00     3625.00     1860.00

The first part has the totals for all keywords. After this the stats are broken down per buffer type.

Part of this work was sponsored by Emerging Threats.