If you've called recycle() obsessively & this turns out to be heap memory ...
I've noticed the garbage-collector doesn't get triggered often enough when there are Java agents / packages being called by LotusScript.
I rewrote the Java agents to 1) handle multiple docs in one agent launch, then 2) call garbage-collection explicitly after every 100-150 docs.
With the packages this was tougher. So I just added an agent to wake up every so often and garbage-collect.