hi,
As far i know roundcube is a fronted, you will need the use sieverules and dovecot, and for automatic filtering you will need spamassassin/amavis or similar of similar packages.
Know Gmail is not only an interface but an entire huge collections of tools and servers that goggle runs (including similar spam hunting systems).
But as you are using roundcube then you already have your own email server and spamkiller which should not be hard to configure correctly.
If you just wish to categorize on some keywords (and/or known senders) then the sieve rules with dovecot are enough (which you can configure through round cube if correctly setup) (requires dovecot+dovecot pigeonhole plugin).
Just what i know.