[Tfug] Funky Apache glitch...

[Adrian]

Anyone seen a funky glitch in Apache were the following happens over a period 
of about 10minutes:
1) stops writing to logs (but still running)
2) individual threads stop responding at various times
3) everything hangs for a few minutes
4) Apache suddenly dumps all previous 10min entries to the log out of order
5) everything continues as if nothing had happened?

I ran into this the other week on one of my servers and have been completely 
unable to find a cause. The machine (though under light-moderate HTTP/SQL 
load) did not, as far as I can tell, lock up in an OOM or spike in load. 
There is no indication that any exploit was used against the machine. 
Although the office said they could get to the server (via HTTP), I was 
shelled in with no problems and they could get to the internet (also serves 
as a firewall). There are no indications of any hardware/network/drive 
problem. Everything is running just fine now, haven't even rebooted the 
machine (though I did restart Apache later that evening).

The logs look like this:
x.x.x.x - - [28/Oct/2005:13:44:43 -0700] "GET
x.x.x.x - - [28/Oct/2005:13:44:43 -0700] "GET
x.x.x.x - - [28/Oct/2005:13:44:43 -0700] "GET
x.x.x.x - - [28/Oct/2005:13:44:44 -0700] "GET
x.x.x.x - - [28/Oct/2005:13:44:44 -0700] "GET
x.x.x.x - - [28/Oct/2005:13:44:44 -0700] "GET
x.x.x.x - - [28/Oct/2005:13:45:23 -0700] "GET
x.x.x.x - - [28/Oct/2005:13:45:29 -0700] "GET
x.x.x.x - - [28/Oct/2005:13:45:30 -0700] "POST
x.x.x.x - - [28/Oct/2005:13:45:30 -0700] "GET
x.x.x.x - - [28/Oct/2005:13:45:44 -0700] "POST
x.x.x.x - - [28/Oct/2005:13:45:45 -0700] "GET
 (some clients not working up to this point, then hang)
x.x.x.x - - [28/Oct/2005:13:38:43 -0700] "POST
x.x.x.x - - [28/Oct/2005:13:43:27 -0700] "GET
x.x.x.x - - [28/Oct/2005:13:43:40 -0700] "GET
x.x.x.x - - [28/Oct/2005:13:44:20 -0700] "GET
x.x.x.x - - [28/Oct/2005:13:44:31 -0700] "GET
x.x.x.x - - [28/Oct/2005:13:44:42 -0700] "GET
x.x.x.x - - [28/Oct/2005:13:44:42 -0700] "GET
x.x.x.x - - [28/Oct/2005:13:42:49 -0700] "POST
x.x.x.x - - [28/Oct/2005:13:42:49 -0700] "POST
x.x.x.x - - [28/Oct/2005:13:37:50 -0700] "POST
x.x.x.x - - [28/Oct/2005:13:46:04 -0700] "GET
...
x.x.x.x - - [28/Oct/2005:13:43:30 -0700] "GET
x.x.x.x - - [28/Oct/2005:13:46:45 -0700] "GET
x.x.x.x - - [28/Oct/2005:13:46:47 -0700] "GET
x.x.x.x - - [28/Oct/2005:13:46:49 -0700] "POST
x.x.x.x - - [28/Oct/2005:13:46:50 -0700] "GET
x.x.x.x - - [28/Oct/2005:13:37:27 -0700] "POST
x.x.x.x - - [28/Oct/2005:13:38:11 -0700] "POST
 (about here suddenly everything starts working again)
x.x.x.x - - [28/Oct/2005:13:46:51 -0700] "POST
x.x.x.x - - [28/Oct/2005:13:46:51 -0700] "GET
x.x.x.x - - [28/Oct/2005:13:47:18 -0700] "POST
x.x.x.x - - [28/Oct/2005:13:47:18 -0700] "GET
x.x.x.x - - [28/Oct/2005:13:47:27 -0700] "GET
x.x.x.x - - [28/Oct/2005:13:47:27 -0700] "GET
x.x.x.x - - [28/Oct/2005:13:47:27 -0700] "GET
x.x.x.x - - [28/Oct/2005:13:47:27 -0700] "GET
x.x.x.x - - [28/Oct/2005:13:47:28 -0700] "GET
x.x.x.x - - [28/Oct/2005:13:47:28 -0700] "GET
x.x.x.x - - [28/Oct/2005:13:47:28 -0700] "GET
x.x.x.x - - [28/Oct/2005:13:47:28 -0700] "GET

Anybody got a clue what might have happened?

Adrian