Having trouble with load balanced setup

[erik_fugletaint]

I have two load balanced roundcube EC2 instances running in docker containers with the 1.4.x-apache image from https://hub.docker.com/r/roundcube/roundcubemail.  I'm using a mariadb (mysql) database on a single db server.  The instances should be using the db for session storage, and as far as I can tell they are doing so.  There are hundreds of rows in the session table from both front end IPs.  The load balancer is using a round robin algorithm.  Every other request to roundcube is failing, with one of two messages:

• Connection to storage server failed.
• Could not load message from server.

I've been able to eliminate the errors by configuring stickiness (aka persistence) on the load balancer.  When a client's requests are handled by a single instance, they are all successful.  It does not matter which instance handles the requests as long as they continue going to the same one.  Therefore it seems to me like a problem with the session storage/retrieval.  But I can't figure out how to troubleshoot this further.

With stickiness removed again I've enabled imap_debug and session_debug and explicitly set session_storage to 'db'.  I don't see any IMAP failures logged and HTTP requests are getting 200 responses.  I do see some "Session auth check failed" messages, followed by "Send new auth cookie" messages, but according to https://github.com/roundcube/roundcubemail/issues/8024, these are normal.  Those log messages do not occur at the same time that I get errors in the browser, or nearly as frequently.  It seems like when I get the browser error "Could not load message from server", only an HTTP request is logged but no IMAP request/response.

Any idea what else I should check?  These are my non-default roundcube configs, with private info redacted in caps:

config.inc.php

Code Select Expand

<?php
    $config['plugins'] = [];
    $config['log_driver'] = 'stdout';
    $config['zipdownload_selection'] = true;
    $config['des_key'] = 'MYDESKEY';
    include(__DIR__ . '/config.docker.inc.php');

config.docker.inc.php

Code Select Expand

<?php
  $config['db_dsnw'] = 'mysql://roundcube:[email protected]:3306/roundcubemail';
  $config['db_dsnr'] = '';
  $config['default_host'] = 'ssl://imapserver.mydomain.com';
  $config['default_port'] = '993';
  $config['smtp_server'] = 'ssl://smtpserver.mydomain.com';
  $config['smtp_port'] = '465';
  $config['temp_dir'] = '/tmp/roundcube-temp';
  $config['skin'] = 'larry';
  $config['plugins'] = array_filter(array_unique(array_merge($config['plugins'], ['archive', 'zipdownload', 'database_attachments'])));

include('/var/roundcube/config/config.imap_debug.php');
include('/var/roundcube/config/config.session_debug.php');
include('/var/roundcube/config/config.session_storage.php');

config.imap_debug.php

Code Select Expand

<?php
  $config['imap_debug'] = true;

config.session_debug.php

Code Select Expand

<?php
  $config['session_debug'] = true;

config.session_storage.php

Code Select Expand

<?php
  $config['session_storage'] = 'db';

[erik_fugletaint]

I may have figured this out.

The IP addresses in the session db table were a clue.  At first glance I thought they were my EC2 instance private IP addresses, but in fact they are the addresses of the application load balancer in the two availability zones with my instances.  So I thought perhaps the session check was using that IP address and failing when authentication happened using the other IP address.  But after looking through the default configs, I saw that they include this:

Code Select Expand

// check client IP in session authorization
$config['ip_check'] = false;

So the IP address shouldn't be checked.  Nonetheless, it made me realize that I'm not recording the client's IP address which is not great.  So then I found this option in the defaults:

Code Select Expand

// List of trusted proxies
// X_FORWARDED_* and X_REAL_IP headers are only accepted from these IPs
$config['proxy_whitelist'] = array();

That allows the X-Forwarded-For header set by the load balancer to be used to determine the actual client IP address.  I set proxy_whitelist with the two IP addresses for the load balancer, restarted the containers, and boom: My real client IP address is showing in the session database entries.  After disabling stickiness on the load balancer, everything seems to be working.  I need to give it a bit longer to be sure, but things are looking good.

This makes me wonder if the ip_check configuration is not working as intended.  Because when set to the default of 'false' I would expect to not have the problem I was having.

[erik_fugletaint]

Unfortunately I was too optimistic.  I'm still seeing the load balancer IPs in the sessions table, and still getting the error messages in my browser when not using sticky sessions.  There are periods where I end up reaching the same instance for a while, and the errors go away.  But I can't yet explain why this happens or why it stops.

[erik_fugletaint]

I'm just going to continue this conversation with myself here 

Those load balancer IP addresses in the session table were mostly from health checks.  I created a separate page to receive the health checks and that, along with adding all potential load balancer interface IPs to the proxy_whitelist config has eliminated them.  Now I only see real client IPs in that table, and far fewer rows.  The original problem still exists but I have a load balancer issue to follow up on that could be related.  Once I've eliminated that I'll continue fiddling with roundcube if necessary.

[erik_fugletaint]

Hello dear silent friends.

I figured it out.  When using load balanced instances it's necessary to manually configure the des_key to be identical on both instances.  Otherwise the session cookie data cannot be decrypted and requests on every instance but the one that set the cookie will fail.  It's been quite a journey sorting this out.  I hope this helps someone in the distant future struggling with a similar setup.

Cheers!