The login attempt through Roundcube takes a very long time to fail. In fact it always takes exactly 60 seconds to fail which sounds to me like some kind of timeout is firing...
The only thing that appears in the mail-info.log file is this one line which occurs at the moment the Roundcube login GUI reports the failure:
Dec 09 08:50:07 imap-login: Info: Disconnected (no auth attempts in 60 secs): user=<>, rip=::1, lip=::1, TLS handshaking: Disconnected
There is no entry on the mail-err.log file for this login attempt.
It looks to me like Roundcube is connecting to the IMAP server but is not then sending a valid login request or if it is the server is not seeing it for some reason. Is there any additional level of logging that can be enabled in Roundcube? Or anything I can try outside of Roundcube to investigate what may be going on. This used to work prior to upgrading to macOS Server 5.2 and macOS 10.12.1 (I was on the previous 5.x release and OS X 10.11.6 before that). And this mail server is continuously used by many clients, including some on the same machine, so I don't think there is any fundamental issue with the mail server itself.