I know my customer had a similar issue in Domino 8.5.3 running on Windows when using a scheduled LotusScript agent which sends an email to a group containing duplicate groups, but it was fixed in Domino 9.0 as seen below. I assume it should work on AIX Domino 9.01 FP4. Maybe a regression bug?
FYI
SPR # RRAL8M5MGS fixed in 8.5.3 FP2
9.0 release
Product Area: Client - Basic; - Standard Technical Area: Mail Platform: Cross Platform
Regression in 8.5.3
SPR# RRAL8M5MGS - Fixed an issue which occurred when sending a confirmation of an Event Announcement to a group that contains duplicate subgroups. The message would not be sent after receiving the following error: ""An error occurred while saving: Document has invalid structure"