[Tfug] Packet delays

[John Hubbard]

I am having a problem with TCP packets being delayed.  I am working on a 
distributed control system, and we are sending messages as TCP packets 
(using the Internet Communications Engine ICE).  Running Wireshark on 
both the machine origination the packet, and on the machine receiving 
the packet I have observed delays of >150ms in the time between when 
Wireshark reports the sending machine has sent the packet and when the 
other Wireshark reports the machine receiving the packet.  I have looked 
through the ICE code and it looks like they are properly using 
TCP_NO_DELAY to disable Nagel's algorithm when sending packets.  This 
delay doesn't happen all of the time.  I would say that it happens on 
the order of 1/10000 packets.

We are running everything on commodity machines.  They are running 
CentOS 5 (a free build of RHEL 5) with Intel CPUs and network 
controllers.  There is a switch between the two machines (likely an hp 
pro-curve or an Extreme Networking Aspen 88xx).  In theory these 
machines are on a virtual private sub-net but I don't have total control 
over that.  I do know that the results are similar when I run tests in 
the middle of the night or the middle of the day so I don't think that 
other network activity is the cause.

So my questions for the community are: Has anyone else seen delays like 
these?  Does anyone have any ideas what could be causing my delays (and 
how to confirm/eliminate that as a cause)?

Thanks in advance and if anyone needs more information I can give it to 
you.  We aren't doing anything proprietary/closed source so I can share 
almost anything.

-- 
-john

To be or not to be, that is the question
                 2b || !2b
(0b10)*(0b1100010) || !(0b10)*(0b1100010)
         0b11000100 || !0b11000100
         0b11000100 || 0b00111011
                0b11111111
         255, that is the answer.