The only time I've seen that is if I'm trying to set a string variable to a hard-coded value (such as HTML to print out to the page) but the string is too big. If you provide more detail on what you're trying to do that causes the error then I may be able to help.

If it does, then it is the expected behaviour. Even when set in Domino Administrator it will give an error to when said database is already over the quota. You need to have the size reduced (delete / archive documents and do compact) before you can set the quota.