[Tfug] Packet delays

[Ammon Lauritzen]

On Sat, Feb 19, 2011 at 9:59 AM, John Hubbard <ender8282 at yahoo.com> wrote:
> I am having a problem with TCP packets being delayed.  I am working on a
> distributed control system, and we are sending messages as TCP packets
> (using the Internet Communications Engine ICE).  Running Wireshark on both
> the machine origination the packet, and on the machine receiving the packet
> I have observed delays of >150ms in the time between when Wireshark reports
> the sending machine has sent the packet and when the other Wireshark reports
> the machine receiving the packet.  I have looked through the ICE code and it
> looks like they are properly using TCP_NO_DELAY to disable Nagel's algorithm
> when sending packets.  This delay doesn't happen all of the time.  I would
> say that it happens on the order of 1/10000 packets.

Do ICMP or UDP packets have similar lag - is it just TCP?

> We are running everything on commodity machines.  They are running CentOS 5
> (a free build of RHEL 5) with Intel CPUs and network controllers.  There is
> a switch between the two machines (likely an hp pro-curve or an Extreme
> Networking Aspen 88xx).  In theory these machines are on a virtual private
> sub-net but I don't have total control over that.  I do know that the
> results are similar when I run tests in the middle of the night or the
> middle of the day so I don't think that other network activity is the cause.

Virtual private subnet? You mean 802.1q? In theory that shouldn't
introduce delays, but in practice I have seen >100ms pings between
hosts that were supposedly only separated by a few switches and some
vlan tagging.

Are you sure there isn't anything silly like a router in the path
between the hosts?

Are any other projects on the network, could they actually be
stressing the switches? :)

Are the boxes close enough to run a big long cable between them to
isolate this sort of infrastructure issue as a problem?

> So my questions for the community are: Has anyone else seen delays like
> these?  Does anyone have any ideas what could be causing my delays (and how
> to confirm/eliminate that as a cause)?

Other than that, the standard answer is to make sure nothing's running
half duplex anywhere ;)

You've probably already considered most of this and sorry I can't be
more helpful, but that's all I've got right now.

Ammon

-- 
Ammon Lauritzen