Need to enable inbound recipient verification, then the option is available
As Barry mentioned. On the configuration
document. Router/SMTP -> Restrictions and
Controls... -> SMTP Inbound Controls tab.
In the Inbound Intended Recipient Controls
section.
Enable "Verify that local
domain recipients exist in the Domino Directory: "
Then the option will appear to Deny
mail to groups.