One way to solve this is to use the virtusertable to only accept mail for the specific valid users and reject mail for anyone else.

First list the valid users:

      joe@virt.dom        juser@isp.net
      susan@virt.dom      suser@isp.net
      al@virt.dom         auser@isp.net

Don't forget postmaster and abuse (which should go to you)

      postmaster@virt.dom    postmaster
      abuse@virt.dom         abuse

And then finally have a default entry which calls the error mailer:

      @virt.dom              error:5.1.1:"550 A-ha, you loose"

This last entry will reject mail to any address other than the 5 listed address.