Ok I think its just a mysql format problem, try this:
$rcmail_config['password_query'] = 'UPDATE `mail`.`virtual_users` SET `password` = MD5( %p ) WHERE `virtual_users`.`id` = (SELECT `id` FROM `mail`.`virtual_users` WHERE `domain_id` = (SELECT `id` FROM `mail`.`virtual_users` WHERE `name` = %d) AND `user` = %l AND `password` = MD5( %o )) LIMIT 1';