Congratulations on installing on Mountain Lion Server. I for one think it shoes everything wrong with Mountain Lion Server. So many ways Apple makes OSX Server hard to work with....
Anyways, I think your challenge is with SSL...so just to make sure you can connect, go to the server admin, certs section, and select custom...then for Mail (IMAP and POP) choose no cert, or make sure you pick the right one. I would start with none, just to make sure it's all working.