You could check to see whether the issue occurs only for replication to a specific server and clustermate replica for this application, or wheher replication with an additional server replica produces the same result.

You could try running maintenance on the application and check the server's log to confirm if there were any issues reported that might have to be addressed.

However, since the formula works on another application, it may not be the fault directly of the formula itself which produces this error, but instead focus might be needed towards a specific document that may be corrupted or contain a large attachment which produces the result.  We have seen in some cases that this message is reported due to a problem experienced when handling a certain document(s) and once that is corrected the replication cycle completes without issue.

To try and shed some light on which document might be causing the issue, you could look at adding some debug to the local client's ini file, restart the replication reproducing the issue, then take a look at the local console log for additional thoughts.  The log should show you the same error and at or around this error, you would be able to grab the noteid of the document that was being replicated at the time.  You can then use the Admin Client to open this application and search for the docment with that noteid to better determine what that document might be or contain.  If you were to move that document aside, maybe to another application or folder within the same applicaion not used by the replication formula, then restart the replication and confirm if that improves the result.

With Notes closed, add the following debug parameters,

log_replication=3   debug_repl=1  debug_repl_all=2  logsatusbar=1  console_log_enabled=1

For additional ideas on log_replication, you can take a look at this technote, http://www.ibm.com/support/docview.wss?uid=swg27002968

In case the error message was witnessed within the Notes status bar, the debug parameter to output those messages to the console log is included above.

Take a look at whether this helps to assist you on your troubleshooting of this issue, but should you need more immediate assistance on the debug output or searching for the noteid, you may prefer to open a pmr with us, but you can attach the console log file and description of the issue and we can assist you further from there.

Thank you Richard and John for your reply,

on my admin PC I see on the replicator page the counter update 42.622 of 42.622 updates, and then the error Memory allocation request exceed 65,000 bytes.

The log.nsf now has 13.732 log documents regarding this one replication ending with:

28-04-2015 23:15:02   REPLICA: ...Skipping note in Destination list (UNID OF9A49991A:7BEF6480-ONC1257913:005562C6 SrcNote ID 0x1F94BE;DestNote ID 0x1A291E;Class 0x0001);
28-04-2015 23:15:02   REPLICA: ... (con't) Latest note is already in destination
28-04-2015 23:15:02   Updated Data Note - DestID 1698182, SrcID 1934586
28-04-2015 23:15:02   REPLICA: ...Skipping note in Destination list (UNID OF2AB10914:C3D76E69-ONC1257A0D:004A46FF SrcNote ID 0x1D84FA;DestNote ID 0x19E986;Class 0x0001);
28-04-2015 23:15:02   REPLICA: ... (con't) Latest note is already in destination
28-04-2015 23:15:02   Unable to replicate activities.nsf: Memory allocation request exceeded 65,000 bytes
28-04-2015 23:15:02   REPLICA: Replicator Debug Enabled
28-04-2015 23:15:02   REPLICA: Watching for ALL notes

log file snip:

[08E0:0006-0AC8:wrepl] 28-04-2015 23:15:02   REPLICA: ...Skipping note in Destination list (UNID OF2AB10914:C3D76E69-ONC1257A0D:004A46FF SrcNote ID 0x1D84FA;DestNote ID 0x19E986;Class 0x0001);
[08E0:0006-0AC8:wrepl] 28-04-2015 23:15:02   REPLICA: ... (con't) Latest note is already in destination
[08E0:0006-0AC8:wrepl] NoteOpenCallback: n=29, TotalNotes=42622, NotesToCopy=3
[08E0:0006-0AC8:wrepl] NoteOpenCallback: hNote=0, (source)NoteID=20C726, status=0x13B=Memory allocation request exceeded 65,000 bytes, hDestDB=22
[08E0:0006-0AC8:wrepl] NoteOpenCallback: n=30, TotalNotes=42622, NotesToCopy=3
[08E0:0006-0AC8:wrepl] NoteOpenCallback: hNote=0, (source)NoteID=1E1082, status=0x13B=Memory allocation request exceeded 65,000 bytes, hDestDB=22
[08E0:0006-0AC8:wrepl] StreamNotes:  NSFDbGetNotes()->ERROR: 13B=Memory allocation request exceeded 65,000 bytes
[08E0:0006-0AC8:wrepl] StreamNotes: Requested 31, got 28
[08E0:0006-0AC8:wrepl] [08E0:0006-0AC8:wrepl] ReplicateNotefile: StreamNotes->0x13B=Memory allocation request exceeded 65,000 bytes, stop=1
[08E0:0006-0AC8:wrepl] 28-04-2015 23:15:02   Unable to replicate activities.nsf: Memory allocation request exceeded 65,000 bytes
[08E0:0006-0AC8:wrepl] [08E0:0006-0AC8:wrepl] ReplicateNotefile: ReplUpdateLog->0x13B=Memory allocation request exceeded 65,000 bytes
[08E0:0006-0AC8:wrepl] 28-04-2015 23:15:02   REPLICA: Replicator Debug Enabled
[08E0:0006-0AC8:wrepl] 28-04-2015 23:15:02   REPLICA: Watching for ALL notes

ODS = 52 on local and server.

Then performed a fixup on the database on the Domino server (Also 901 FP3).

28-04-2015 23:32:56   Warning: Fixup purged corrupt document UNID (48520C58:707E3E21:C1257824:002F6EAD) from D:\IBM\Domino\data\activities.nsf
28-04-2015 23:32:57   Document NT001E1082 in database D:\IBM\Domino\data\activities.nsf is damaged: Document has invalid structure
28-04-2015 23:32:57   Document (UNID OF48520C58:707E3E21-ONC1257824:002F6EAD) in database D:\IBM\Domino\data\activities.nsf has been deleted
28-04-2015 23:32:57   Warning: Fixup purged corrupt document UNID (DE8C4309:248615EB:C1257689:0041ECC9) from D:\IBM\Domino\data\activities.nsf
28-04-2015 23:32:57   Document NT0020C726 in database D:\IBM\Domino\data\activities.nsf is damaged: Document has invalid structure
28-04-2015 23:32:57   Document (UNID OFDE8C4309:248615EB-ONC1257689:0041ECC9) in database D:\IBM\Domino\data\activities.nsf has been deleted

After that the replication works fine!

Strange, the database replicated without any error with no replication formula, only gave the error with the formula enabled, never thought the error could be a corrupt document, I feel stupid now.....

Thank you for your help solving this problem!!