Roundcube discards entire message index whenever a new message arrives?

[larrymcp]

I did some further research into Roundcube's slow performance when displaying folders with a lot of messages in them, since our first theory didn't work.  I ran some tests while monitoring the IMAP conversations.

I've now found the "culprit" and I'll describe it here.

Here's the issue:  If the contents of a folder have changed in any way (even a single message added or removed), then Roundcube doesn't use its existing message index.  It fetches the entire message index all over again, from the server each time.  Yikes!

This causes really long delays if you have a folder with lots of messages in it.  Every time something changes in that folder, you have to sit there for several seconds while Roundcube re-fetches the whole folder index.  Huge drain on the mail server, too.

Why doesn't Roundcube behave like other mail clients in this respect?  I know that IMAP must have a way to retrieve only what's changed in a folder (new or removed messages) without having to request the entire list, because I verified that Thunderbird doesn't re-fetch the whole folder.

Has this issue been previously discussed, or submitted for fixing?  If not, I would certainly like to put it in the hopper.

[alec]

Do you have enabled caching in Roundcube? What IMAP server? What sorting order? Provide imap_debug log.

[larrymcp]

Ok-  gladly!  Thank you so much for taking a look at this.  If there's a solution to be found, it will save me much headaches.

We've got the following caching settings in effect:

Code Select Expand


$rcmail_config['imap_cache'] = 'db';
$rcmail_config['messages_cache'] = true;
$rcmail_config['message_cache_lifetime'] = '30d';
$rcmail_config['session_storage'] = 'db';


Sorting is set blank, as follows:

Code Select Expand


$rcmail_config['message_sort_col'] = '';


The IMAP server is hMailServer, current version.

IMAP log files:

Here's the IMAP log showing what happened when I switched to the Trash folder a few minutes ago.   Log #1   There were 4 messages added to Trash since the cache was refreshed, so Roundcube downloaded the whole message index again, causing a long delay.

I also tried turning off IMAP SORT on the server.  That makes the problem even worse, though:  Roundcube then requests each message date individually.  Here's a log showing what happened after 2 messages were added to Trash:  Log #2

I then tried changing message_sort_col to 'arrival' with the IMAP sorting still turned off.  But got the same results as Log #2 above; RC requested each message date individually.

Again, thanks so much...  deepest gratitude if there is some workaround that can correct this behavior.

[alec]

Here's the IMAP log showing what happened when I switched to the Trash folder a few minutes ago.   Log #1   There were 4 messages added to Trash since the cache was refreshed, so Roundcube downloaded the whole message index again, causing a long delay.

Your server is so f*ing slow. Sorting 4k messages takes a second on my dovecot installation. Something's wrong with the IMAP server. As you see in the log sorting is set to date. If you want to get rid of SORT call, you need to set sorting to None. Setting message_sort_col is not enough (it's a default for new users), you need to set it in message list menu.

[larrymcp]

Ok-  this sounds very promising.  I don't see where that setting is located; can you tell me where that is?  I'm in the Roundcube "Settings" menu, and I'm looking at each section of settings, and I don't see anything about sorting in any of those menus.

Your server is so f*ing slow.

You got that right.     When sorting by date it is a beast...  but that wasn't a problem before Roundcube, because other mail clients only ask for the message list once.

hMailServer's author actually acknowledged the slow sorting in a forum post, saying "If you sort on date, hMailServer must open all messages in your mailbox and read the Date header from them.  However, sorting on arrival date gives much better performance than sorting on Date.  If you sort on arrival, hMailServer uses meta data stored in the database."

So apparently, if we could get Roundcube to request sorting by arrival date, that would solve the problem.  Is this the same setting that I'm having trouble finding in Roundcube?  Thanks again for your help.

[alec]

Ok-  this sounds very promising.  I don't see where that setting is located; can you tell me where that is?  I'm in the Roundcube "Settings" menu, and I'm looking at each section of settings, and I don't see anything about sorting in any of those menus.

It's not in Settings, it's on the messages list. There's an icon in table header.

[larrymcp]

Oh wow, this is SO AWESOME.

I changed it to sort on arrival date, and now it comes back in just one second, every time.

Solved, thanks!