Solved:
I'm using vestacp, I tried re-installing vesta and an error appeared:
ERROR 1071 (42000) at line 51: Specified key was too long; max key length is 767 bytes
This is a db error which I assumed was related to roundcube. So after verifying the roundcube db was incomplete, I deleted the roundcube db and re-created an empty roundcube db, then tried to re-install from initial roundcube sql:
mysql roundcube < /usr/share/roundcubemail/SQL/mysql.initial.sql
ERROR: 1071 (42000) at line 51: Specified key was too long; max key length is 767 bytes
I added: innodb_large_prefix = 1;
Still didn't work.
Mariadb version 5.5.6 has issues with max key length. So I could either try and fix this or install a newer version of mariadb.
Upgraded to mariadb 10.4 -
mysql roundcube < /usr/share/roundcubemail/SQL/mysql.initial.sql
That worked! In addition, mysql performs better.
I still have an smtp problem which I'll start a new thread.