You could use hmail_spamfilter. This plugin will copy spam/ham messages in speparate folders; in my case "c:/spamassassin/train_bayes/ham_to_learn" and "c:/spamassassin/train_bayes/spam_to_learn". After that it is not a job for Roundcube anymore. You have to process these message f.e. by sa-learn.exe. (Spamassassin) separately. It depends on which kind of spam detection application you are using. Of course you have to configure hMailServer to scan messages by Spamassassin. There is a very comfortable interface to do so.
To get this working on Windows along with Spamassassing I have the following tasks scheduled:
C:\spamassassin\sa-learn.exe --ham --showdots c:/spamassassin/train_bayes/ham_to_learn
C:\spamassassin\sa-learn.exe --spam --showdots c:/spamassassin/train_bayes/spam_to_learn
As you are using hMailserver I guess your Server runs on a Windows OS. So, this should guide you into the right direction.