By accident (or, more correctly, trying different settings!), I've found the problem.
RoundCube setting "smtp_server".
If set to nothing (to use PHP mail()), that works OK.
If set to %h or a domain, it fails with the errors indicated.
Either setting works fine on hMail 5.1.2.
Now which end is wrong is another matter entirely, but it's working again!
Jim