Author Topic: memory leak? HELP!  (Read 4392 times)

Offline lxgwx4

  • Jr. Member
  • **
  • Posts: 10
memory leak? HELP!
« on: January 31, 2007, 09:11:07 PM »
After run Roundcube under linux a few hours later,the number of 'httpd' processes(use the linux 'ps -e | grep httpd | wc -l' command)didn't decrease to MaxSpareServers in the httpd-mpm.conf file.And the used memory was not freed.
I use the prefork module.details are:

  StartServers     5
  MinSpareServers    5
  MaxSpareServers   10
  ServerLimit    200
  MaxClients    200
  MaxRequestsPerChild  2000

HELP!!
HELP!!!

Offline lxgwx4

  • Jr. Member
  • **
  • Posts: 10
Re: memory leak? HELP!
« Reply #1 on: January 31, 2007, 09:18:15 PM »
When where are no request ,according to the apache manual,the httpd processes should be decreased to MaxSpareServers.But I still have 60-70 processes.
What's the problem?

Offline chillll

  • Newbie
  • *
  • Posts: 1
Re: memory leak? HELP!
« Reply #2 on: January 31, 2007, 09:27:53 PM »
It's a so serious problem.

but, I can't answer anything.

Offline jpingle

  • Jr. Member
  • **
  • Posts: 77
Re: memory leak? HELP!
« Reply #3 on: February 01, 2007, 08:10:20 PM »
If you can post the output of apache's server status page someone might be able to guess as to what is going on. It is probably something specific to your server that is not timing out as it should, or getting stuck. It would help a lot if Apache's "extended status" is turned on.

Offline lxgwx4

  • Jr. Member
  • **
  • Posts: 10
Re: memory leak? HELP!
« Reply #4 on: February 01, 2007, 08:43:56 PM »
Thank you jpingle. My Apache's status are:
============================

Apache Server Status for 192.168.3.15
Server Version: Apache/2.2.3 (Unix) PHP/5.1.6
Server Built: Nov 10 2006 20:02:26

--------------------------------------------------------------------------------

Current Time: Friday, 02-Feb-2007 09:38:14 CST
Restart Time: Friday, 02-Feb-2007 09:36:31 CST
Parent Server Generation: 0
Server uptime: 1 minute 42 seconds
1 requests currently being processed, 7 idle workers
_W______........................................................
................................................................
................................................................
...............................................................
Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

PID Key:


  19470 in state: _ ,  19471 in state: W ,  19472 in state: _
  19473 in state: _ ,  19474 in state: _ ,  19475 in state: _
  19487 in state: _ ,  19490 in state: _ ,


--------------------------------------------------------------------------------
To obtain a full report with current status information you need to use the ExtendedStatus On directive.
--------------------------------------------------------------------------------

Apache/2.2.3 (Unix) PHP/5.1.6 Server at 192.168.3.15 Port 80

============================
Is this information enough for tracing this problem?

Offline lxgwx4

  • Jr. Member
  • **
  • Posts: 10
Re: memory leak? HELP!
« Reply #5 on: February 01, 2007, 09:47:33 PM »
The top post is the status when Apache was started.
Below is the status after Stress Test.
========================================

Apache Server Status for 192.168.3.15
Server Version: Apache/2.2.3 (Unix) PHP/5.1.6
Server Built: Nov 10 2006 20:02:26

--------------------------------------------------------------------------------

Current Time: Friday, 02-Feb-2007 10:42:43 CST
Restart Time: Friday, 02-Feb-2007 09:36:31 CST
Parent Server Generation: 0
Server uptime: 1 hour 6 minutes 11 seconds
227 requests currently being processed, 9 idle workers
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW._W
WWWWWWWWWWWWWWW.WWWWWWWW.WWWWWWWWWWWWW.WWWWWWWWWWWWWWW_WWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWW_W_WWW.WWWWWWWWWWWW...WWW_WWWWWWW.WW.WWW.
WWWWWWWWWW.W._WWW.http://WWW.WWWWWWWWWWWWW_WWW.WWWW.W.W.WWWWWW_W_WWWWW
Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

PID Key:


  19470 in state: W ,  19471 in state: W ,  19472 in state: W
  19473 in state: W ,  19474 in state: W ,  19475 in state: W
  19487 in state: W ,  19490 in state: W ,  19551 in state: W
  19564 in state: W ,  19565 in state: W ,  19566 in state: W
  19567 in state: W ,  19568 in state: W ,  19569 in state: W
  21537 in state: W ,  19571 in state: W ,  19572 in state: W
  19573 in state: W ,  19574 in state: W ,  19575 in state: W
  19576 in state: W ,  19577 in state: W ,  19578 in state: W
  19588 in state: W ,  19589 in state: W ,  19590 in state: W
  19592 in state: W ,  19593 in state: W ,  19594 in state: W
  19595 in state: W ,  19605 in state: W ,  19606 in state: W
  19607 in state: W ,  19608 in state: W ,  19609 in state: W
  19610 in state: W ,  19611 in state: W ,  19614 in state: W
  19615 in state: W ,  19616 in state: W ,  19617 in state: W
  19618 in state: W ,  19619 in state: W ,  21886 in state: W
  19621 in state: W ,  19677 in state: W ,  19678 in state: W
  19679 in state: W ,  19680 in state: W ,  19681 in state: W
  21541 in state: W ,  19683 in state: W ,  19898 in state: W
  19901 in state: W ,  19902 in state: W ,  19903 in state: W
  19904 in state: W ,  19905 in state: W ,  19906 in state: W
  19978 in state: W ,  22367 in state: _ ,  21890 in state: W
  19987 in state: W ,  19988 in state: W ,  21893 in state: W
  20018 in state: W ,  21894 in state: W ,  20020 in state: W
  20021 in state: W ,  20022 in state: W ,  21132 in state: W
  21193 in state: W ,  20039 in state: W ,  20040 in state: W
  20041 in state: W ,  20042 in state: W ,  20043 in state: W
  20045 in state: W ,  20063 in state: W ,  20064 in state: W
  20065 in state: W ,  20066 in state: W ,  20067 in state: W
  20068 in state: W ,  20069 in state: W ,  20085 in state: W
  20087 in state: W ,  20088 in state: W ,  21253 in state: W
  20090 in state: W ,  20092 in state: W ,  20093 in state: W
  20094 in state: W ,  20095 in state: W ,  21895 in state: W
  20097 in state: W ,  20098 in state: W ,  20099 in state: W
  20118 in state: W ,  20119 in state: W ,  20120 in state: W
  20121 in state: W ,  20122 in state: W ,  20145 in state: W
  20146 in state: W ,  20147 in state: W ,  20148 in state: W
  20149 in state: W ,  20150 in state: W ,  20151 in state: W
  20194 in state: W ,  20195 in state: W ,  20196 in state: W
  21896 in state: _ ,  20198 in state: W ,  20199 in state: W
  21211 in state: W ,  20295 in state: W ,  20296 in state: W
  20297 in state: W ,  20298 in state: W ,  20300 in state: W
  20301 in state: W ,  21899 in state: W ,  21781 in state: W
  20305 in state: W ,  20306 in state: W ,  20307 in state: W
  20308 in state: W ,  20309 in state: W ,  20349 in state: W
  20350 in state: W ,  20351 in state: W ,  20352 in state: W
  20353 in state: W ,  20354 in state: W ,  20355 in state: W
  20421 in state: W ,  20422 in state: W ,  20423 in state: W
  20424 in state: W ,  20425 in state: W ,  21900 in state: W
  21657 in state: W ,  20444 in state: W ,  20445 in state: W
  20446 in state: _ ,  20447 in state: W ,  21901 in state: _
  20449 in state: W ,  20450 in state: W ,  20468 in state: W
  20473 in state: W ,  20474 in state: W ,  21929 in state: W
  20476 in state: W ,  20477 in state: W ,  21137 in state: W
  20482 in state: W ,  21660 in state: W ,  20484 in state: W
  20485 in state: W ,  20486 in state: W ,  20555 in state: W
  20559 in state: W ,  21197 in state: W ,  21662 in state: W
  21663 in state: _ ,  20587 in state: W ,  20588 in state: W
  20589 in state: W ,  20590 in state: W ,  21675 in state: W
  21932 in state: W ,  20747 in state: W ,  20749 in state: W
  21678 in state: W ,  21679 in state: W ,  20753 in state: W
  20771 in state: W ,  20773 in state: W ,  21138 in state: W
  20775 in state: W ,  21257 in state: W ,  20777 in state: W
  21201 in state: W ,  20793 in state: W ,  20794 in state: W
  20795 in state: W ,  20796 in state: W ,  21681 in state: W
  21141 in state: _ ,  20828 in state: W ,  20829 in state: W
  20830 in state: W ,  21142 in state: W ,  20911 in state: W
  20912 in state: W ,  20914 in state: W ,  20915 in state: W
  20916 in state: W ,  20917 in state: W ,  21204 in state: W
  22378 in state: W ,  20935 in state: W ,  20936 in state: W
  20938 in state: W ,  20940 in state: W ,  20941 in state: W
  20942 in state: W ,  20943 in state: W ,  21704 in state: _
  21938 in state: W ,  20946 in state: W ,  20947 in state: W
  21046 in state: W ,  21047 in state: W ,  22380 in state: W
  21205 in state: W ,  21051 in state: W ,  21070 in state: W
  21941 in state: W ,  21708 in state: W ,  21074 in state: W
  21075 in state: W ,  21151 in state: W ,  21088 in state: W
  21942 in state: _ ,  21972 in state: W ,  21973 in state: _
  21260 in state: W ,  21093 in state: W ,  21208 in state: W
  21740 in state: W ,  21974 in state: W ,


--------------------------------------------------------------------------------
To obtain a full report with current status information you need to use the ExtendedStatus On directive.
--------------------------------------------------------------------------------

Apache/2.2.3 (Unix) PHP/5.1.6 Server at 192.168.3.15 Port 80

========================================

Does it mean no connection was released?

Offline jpingle

  • Jr. Member
  • **
  • Posts: 77
Re: memory leak? HELP!
« Reply #6 on: February 01, 2007, 10:30:09 PM »
That means that Apache is still actively writing to every process listed with a "W"

You may have to sniff the connection to see what is still being read after all that time, if anything. It could be that there is something tripping the process up or it could be stuck reading a large file, it's hard to say. I've also heard of people having large % of CPU time when their login mechanism for IMAP was misconfigured.

I'd say this is not a memory leak in RoundCube, but rather something causing the process not to terminate. It's more likely that it's happening on the IMAP end, but without more information it's hard to say. You might check the process list on the server, see if there are any processes chewing up 100% (or a large %) of CPU time, or check other system logs for errors. The problem is likely elsewhere on the server, and it's only being brought out by RoundCube because of its interaction with whatever is broken.

You can also try watching the connection with tcpflow if you have access to the server at that level.

Offline lxgwx4

  • Jr. Member
  • **
  • Posts: 10
Re: memory leak? HELP!
« Reply #7 on: February 02, 2007, 12:53:59 AM »
Yes, now I also think that this is not the memory leak problem.
Do you mean that Apache will not close the connection when IMAP connections exist of this httpd process? I'm confused about the relationship between imap and http.

Offline D.L

  • Newbie
  • *
  • Posts: 1
Re: memory leak? HELP!
« Reply #8 on: February 02, 2007, 01:07:57 AM »
What's about the roundcube' code.
Does it has something with this problem.

Offline jpingle

  • Jr. Member
  • **
  • Posts: 77
Re: memory leak? HELP!
« Reply #9 on: February 02, 2007, 02:55:30 PM »
It really depends on what part of the whole workflow is blocking on where the problem might lie.

For example: If the IMAP server is stuck sending data from a huge file, Roundcube is probably reading it and Apache is writing that data back to the clients.

Usually, it would hit a timeout and then terminate. I believe by default PHP will stop executing after 60 seconds, but I'm not sure that is the case if the data is actively being read by a client.

You might want to reconfigure apache with "ExtendedStatus On" so you can get a little more detail out of the server status page. It will tell you exactly what page is being requested for each process. That would go a long way to figuring out what request is causing the problem.

Also, you might check/post some of the contents of Apache's error log, and/or RoundCube's log files such as console and errors.

Offline lxgwx4

  • Jr. Member
  • **
  • Posts: 10
Re: memory leak? HELP!
« Reply #10 on: February 05, 2007, 12:24:43 AM »
In the RoundCube's errors log file, there are many messages like:
1. "IMAP Error: Invalid password"
2. "PHP Fatal error: Maximum execution time of 120 seconds exceeded in /usr/local/apache/htdocs/roundcube/program/lib/imap.inc on line 132"
3. "DB Error: DB Error: connect failed in /usr/local/apache/htdocs/roundcube/program/include/rcube_db.inc on line 105"

So, I think maybe it's the MySQL or IMAP Server's problem. Maybe they are out of service and the PHP script are still waiting the result data.

Offline lxgwx4

  • Jr. Member
  • **
  • Posts: 10
Re: memory leak? HELP!
« Reply #11 on: February 08, 2007, 02:59:27 AM »
There're still many httpd processes exist in my linux system. By using netstat and lsof linux command and the Apache's server-status module, I found that the existed httpd processes looks like still connect with the IMAP server and not killed by the linux.
More details are in the attachments.

Offline lxgwx4

  • Jr. Member
  • **
  • Posts: 10
Re: memory leak? HELP!
« Reply #12 on: February 28, 2007, 01:27:47 AM »
These days I found that there's some bugs in the function:
function iil_ReadLine($fp, $size){
   $line="";
   if ($fp){
      do{
         $buffer = fgets($fp, 2048);
         $line.=$buffer;
      }while($buffer[strlen($buffer)-1]!="\n");
   $buffer = fgets($fp, 2048);
   $line=$buffer;
   }
   return $line;
}

maybe not just in this function.
when there is 'while' or 'for' statements, under the stress test, this function will not return and be dead.