Roundcube times out under heavy load/What performance to expect?

[masc]

Hello,

I am trying to set up a medium sized Roundcube cluster in front of Dovecot. Unfortunately I encounter timeouts on some requests involving the IMAP connections. Roundcube returns "504 - Gateway Timeout" after 30 seconds in its Apache2 mpm_prefork default Docker-container.

The Dovecot nodes are HW with 24C/48T, 96G memory and SSD storage.
The HAProxy nodes handling ingress traffic are running in L7 mode and additionally do SSL-offloading and have enough resources (30% loaded while running the tests).
The Roundcube-nodes have 4 Cores with 2,8 GHz and 4G memory each.
The DB-nodes are running a separate MySQL-cluster based on Percona XtraDB.

Overview

The structure looks as follows

HAProxy (A) <-> HAProxy (P)
|                      \                     \                     \
Roundcube 1 Roundcube 2 Roundcube 3 Roundcube 4
|   |                  /                     /                     /
|   imapproxy  imapproxy     imapproxy      imapproxy
|   |                 /                    /                      /
|   Dovecot 1 <-> Dovecot 2
|   |                      /
MySQL DB

In words, HAProxy active/passive cluster node processes the ingress traffic, balances over the 4 Roundcube nodes, which use a MySQL DB and Dovecot as an IMAP backend.

Testing

For testing I wrote a JMeter testfile that does cycles of

• Open page
• Login
• Load mail lists
• Load some mails
• Open some pages (like "compose mail")
• Logout

The testfile does not include loading static resources for obvious reasons. The testfile alternates between a set of 16 IMAP accounts that are spread over the two Dovecot nodes equally.

If you are interested in the JMeter testfile, I drop me a PM, I am willing to clean the file up and share it with others if there is a demand for it.

Problem

With the testfile and the Apache2 mpm_prefork based Docker-container with the 4 Roundcube-nodes I achieve fairly good responsetimes, but some requests run into timeouts. These are exclusively requests that involve IMAP transactions (but see the "Additional information about the Dovecot and its performance"!). It seems that the Apache and/or Roundcube inside the containers does not answer in time and some 0,0x percent of the requests get stuck for too long. The performance of requests per second (of actual content, again, no static resources) is between 600 and 850 per second.

Questions

• What performance to expect from Roundcube with Apache2 mpm_prefork when the IMAP server is no limiting factor
• What could be the reason for some requests getting stuck so much that they run into the 30 second timeout
• Does anyone around here run a similar setup with 800+ concurrent users and how did you design the setup and why

Additional information about the Dovecot and its performance

To investigate the problem and exclude the Dovecot from being at fault, I enabled statistics of IMAP commands in the Dovecot nodes. The statistics show that the answering times of Dovecot are very good, one second at max. - when running the tests, the Dovecot nodes are barely showing any load, neither on CPU nor on the storage.

In the hopes for a lively discussion,

masc