For my Suricata QA setup, I’m using tcpreplay on a dual port gigabit NIC. The idea is to blast out packets on one port and then have Suricata listen on the other part.

For the traffic replay I’m using tcpreplay 3.4.4 from the Ubuntu archive. As I have a lot of pcaps to process I intend to use the –topspeed option to keep runtimes as low as possible. This will result in approximately ~500Mbps on this box, as the pcaps come from a nas.

While validating the replay results, I noticed that there was a lot of packet reordering going on. This seemed odd as tcpreplay replays packets in order. The docs seemed to suggest the driver/NIC does this: http://tcpreplay.synfin.net/wiki/FAQ#tcpreplayissendingpacketsoutoforder

It turned out that this is caused by the driver using multiple tx-queues.

dmesg:

[    1.143444] igb 0000:03:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)

With the help of Luca Deri I was able to reduce the number of queues.

To do this, the igb driver module needs to be passed an option, RSS=1. However, the igb driver that comes with Ubuntu 13.10 (which has version 5.0.5k) does not support this option.

The latest version is needed, which can be downloaded from http://sourceforge.net/projects/e1000/files/igb%20stable/5.1.2/

After installing it, remove the current module and load the new module with the RSS option:

modprobe -r igb
modprobe igb RSS=1

Confirm the result in dmesg:

[  834.376632] igb 0000:03:00.1: Using MSI-X interrupts. 1 rx queue(s), 1 tx queue(s)

With this, tcpreplay at topspeed will not result in reordered packets.

Many thanks to Luca Deri for putting me on the right track here.