SPR

Technote

Problem:
This issue was reported to Quality Engineering as SPR #CMCY5Z3RC6 and has been
fixed in 6.5.5, 6.5.4 Fix Pack 3 (FP3), and Domino 7.0.1.
Excerpt from the Lotus Notes and Domino Release 6.5.5, 7.0.1 MR, and 6.5.4 FP3
fix lists MR fix list (available at http://www.ibm.com/developerworks/lotus):
Replication
SPR# CMCY5Z3RC6 - Fixed a memory leak in replication.

Refer to the Upgrade Central site for details on upgrading Notes/Domino.


This problem is caused by a memory leak in the Domino server code. If you
review the Top 10 section of the memcheck you will find that block 0x880D (the
extra 8 on the front means it is a shared block) is very high. In this case
over 450 megabytes.

#------ Top 10 Shared Memory Block Usage:
BY SIZE | BY HANDLE COUNT
Type TotalSize Handles | Type Handles TotalSize
---------------------------- | ----------------------------
0x82cd 783314944 199 | 0x8604 32259 796210
0x880d 484854678 7413 | 0x841c 7999 21285876
0x8252 48234496 46 | 0x841b 7979 327218
0x8219 22472162 1609 | 0x880d 7413 484854678
0x841c 21285876 7999 | 0x8001 5005 6776770
0x8291 20864514 319 | 0x8405 4105 13148662
0x824b 20077392 518 | 0x815f 3115 619082
0x834a 17301504 17 | 0x8219 1609 22472162
0x9122 15893658 243 | 0x82d2 1554 5402608
0x8311 14680064 14 | 0xab04 1420 51908
-----------------------------------------------------------

This information can also be found in a memory dump

CLIENT (0x0800)
+3 1 592 SHRMEM BLK_CLIENT - CLIENT static data
+13 14973 979413876 SHRHDL BLK_CL_OPENED_NOTE - Client-opened note
pool

In order to debug this crash, the following ini parameters must be added to the
notes.ini and the server needs to be restarted.
DEBUG_DUMP_FULL_HANDLE_TABLE=1
DEBUG_SHOWLEAKS=1
DEBUG_THREADID=1
DEBUG_TRAPLEAKS=080D
DEBUG_TRAPLEAKS_SHOWSTACK=1
Console_log_enabled=1

When collecting this debug it is important that the server be shut down cleanly
before it crashes. For example, if the server is crashing every two days, let
the server run for one day and cleanly shut it down at the end of the day. You
will see the following output in the console.log if block 0x080D is leaking.

[0CE0:0002-05B0] Leaked block: PID=0xCE0 Handle=0x1D1E Type=0x080D(PKG_0x8+13)
Length=65412 Addr=1B96FBA0
[0CE0:0002-05B0] NOTE: This is a shared block that 'may' get freed later by
another process.
[0CE0:0002-05B0] When the last Notes process terminates, this block
will be reported again if it is still allocated.
[0CE0:0002-05B0] Allocated at 6/13/2005 20:47:58 by PID 0xCE0 - thread 0x2
[0CE0:0002-05B0] Called from [01]: 6019F8D7 nnotes! OSDisableGlobalEvents +999
[0CE0:0002-05B0] Called from [02]: 600070D6 nnotes! OSMemAllocExtended +390
[0CE0:0002-05B0] Called from [03]: 6000765C nnotes! OSMemAlloc +28
[0CE0:0002-05B0] Called from [04]: 60015903 nnotes! OSCreatePool +547
[0CE0:0002-05B0] Called from [05]: 6001A624 nnotes! Cstrtrim +2708
[0CE0:0002-05B0] Called from [06]: 600A52DE nnotes! NSFNoteCreatePool +30
[0CE0:0002-05B0] Called from [07]: 60180C21 nnotes!
ClientGetServerAvailabilityInfo +4065
[0CE0:0002-05B0] Called from [08]: 601838CB nnotes!
ClientGetServerAvailabilityInfo +15499
[0CE0:0002-05B0] Called from [09]: 601834E0 nnotes!
ClientGetServerAvailabilityInfo +14496
[0CE0:0002-05B0] Called from [10]: 6001B7C2 nnotes! NSFDbIsRemote +930
[0CE0:0002-05B0] Called from [11]: 6001FF6D nnotes! NSFNoteOpenExtended +365
[0CE0:0002-05B0] Called from [12]: 607D90C1 nnotes! NSFNoteReplicate +369
[0CE0:0002-05B0] Called from [13]: 608F0075 nnotes! ReplicateNotefile +23429
[0CE0:0002-05B0] Called from [14]: 608EAB2A nnotes! ReplicateNotefile +1594
[0CE0:0002-05B0] Called from [15]: 608F9B87 nnotes! ReplicateWithServerExt2
+9591
[0CE0:0002-05B0] Called from [16]: 608F7CE8 nnotes! ReplicateWithServerExt2
+1752
[0CE0:0002-05B0] Called from [17]: 004017AA nReplica! unknown +0
[0CE0:0002-05B0] Called from [18]: 00401513 nReplica! unknown +0

The crash stack for this issue will vary because the crash is due to the
process running out of memory. However, the top of the stack will always
represent a failure to allocate memory.


############################################################
### FATAL THREAD 13/57 [ nHTTP:0210:0dc8]
### FP=0x0ba1f450, PC=0x600019be, SP=0x0ba1f428, stksize=40
### EAX=0x60c1d6e4, EBX=0x000004fa, ECX=0x0000ff84, EDX=0x60c1d6e4
### ESI=0x00000000, EDI=0x60c104fb, CS=0x0000001b, SS=0x00000023
### DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00010212
Exception code: c0000005 (ACCESS_VIOLATION)
############################################################
@[ 1] 0x600019be nnotes._AllocDBlock@12+14 (60c1d6e4,ff84,ba1f4e4,ba1f510)
@[ 2] 0x600019a4 nnotes._OSAllocDBlock@12+20 (60c1d6e4,ff84,ba1f4e4,ff84)
@[ 3] 0x60001dfb nnotes._DPoolAlloc@16+603 (b70014,ff84,824b,4bee0a05)
@[ 4] 0x6000415b nnotes._AllocObject@24+75 (b70014,60c40a40,4bee09f7,ff84)
@[ 5] 0x60007021 nnotes._OSMemAllocExtended@16+209 (824b,ff7e,0,ba1f5a0)
@[ 6] 0x6000765c nnotes._OSMemAlloc@12+28 (824b,ff7e,ba1f5a0,6415824b)
@[ 7] 0x60015903 nnotes._OSCreatePool@28+547 (6415c24b,0,20e,400)
@[ 8] 0x6001a624 nnotes._NSFNoteCreatePoolExtended@24+164 (0,6415824b,0,0)
@[ 9] 0x600a52de nnotes._NSFNoteCreatePool@20+30 (0,6415824b,0,0)
@[10] 0x600aabe8 nnotes._NSFNoteCreateType@12+24 (0,6415824b,ba1f668,5841)
@[11] 0x600bafd1 nnotes._NSFNoteCreateCloneExt@12+65 (0,0,ba1f690,9)
@[12] 0x600baebe nnotes._NSFNoteCopyExt@12+94 (5841,0,ba1f804,ba1f7ec)
@[13] 0x600bae52 nnotes._NSFNoteCopy@8+18 (5841,ba1f804,0,0)
@[14] 0x60076fc7 nnotes._NSFProfileOpenExt2@36+295 (2d1,6d0ccc,a,0)
@[15] 0x600a2cea nnotes._NSFProfileOpenExt@32+42 (2d1,6d0ccc,a,0)
@[16] 0x600b0fe6 nnotes._NSFProfileOpen@28+38 (2d1,6d0ccc,a,0)
@[17] 0x005285bb ninotes.WebAdmin::SetACL+603
(ba1fe24,61626577,6e696d64,66736e2e)
@[18] 0x00528145 ninotes.WebAdmin::Refresh+133
(1000f4ed,4a771ba8,4a065d34,10042778)
@[19] 0x100136e5 nhttpstack.HTInotesExtension::RefreshConfiguration+5
(3,10042778,10027c40,100136b2)
@[20] 0x10001873 nhttpstack.HTConfigurationThread::ThreadMain+355
(0,ba1ffb8,60115334,10042778)
@[21] 0x10027c6a nhttpstack._HTThreadBeginProc@4+42 (10042778,0,0,0)
@[22] 0x60115334 nnotes._ThreadWrapper@4+212 (0,0,0,0)
[23] 0x77e4a990 KERNEL32.FlsSetValue+1913



############################################################
### FATAL THREAD 13/58 [ nHTTP:0b1c:0cd0]
### FP=0x0a71deac, PC=0x600019be, SP=0x0a71de84, stksize=40
### EAX=0x60c1d6ac, EBX=0x000004ec, ECX=0x000004dc, EDX=0x60c1d6ac
### ESI=0x00000000, EDI=0x60c104ed, CS=0x0000001b, SS=0x00000023
### DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00010206
Exception code: c0000005 (ACCESS_VIOLATION)
############################################################
@[ 1] 0x600019be nnotes._AllocDBlock@12+14 (60c1d6ac,4dc,a71df40,a71df6c)
@[ 2] 0x600019a4 nnotes._OSAllocDBlock@12+20 (60c1d6ac,4dc,a71df40,4dc)
@[ 3] 0x60001dfb nnotes._DPoolAlloc@16+603 (b70014,4dc,8219,495048a)
@[ 4] 0x6000415b nnotes._AllocObject@24+75 (b70014,60c40a40,495047c,4dc)
@[ 5] 0x600073fd nnotes._OSMemoryAllocate@12+141 (b70014,4d6,52d20060,0)
@[ 6] 0x600a021c nnotes._DbBDBRead@20+2092 (4dbd0000,0,0,1)
@[ 7] 0x6009e3fb nnotes._DbLoad@4+635 (52d1fee8,a71fe24,0,0)
@[ 8] 0x6002d480 nnotes._NSFDbOpenExtended3@48+14944 (a71fe24,0,0,0)
@[ 9] 0x6007fea6 nnotes._NSFDbOpenExtended2@44+54 (a71fe24,0,0,0)
@[10] 0x6009c54a nnotes._NSFDbOpenExtended@28+58 (a71fe24,0,0,0)
@[11] 0x600b0f9a nnotes._NSFDbOpen@8+26 (a71fe24,a71fdd4,100c208,1004ccfa)
@[12] 0x00528467 ninotes.WebAdmin::SetACL+263
(a71fe24,61626577,6e696d64,66736e2e)
@[13] 0x00528145 ninotes.WebAdmin::Refresh+133
(1000f4ed,5d80cfc8,4cdb948c,10042778)
@[14] 0x100136e5 nhttpstack.HTInotesExtension::RefreshConfiguration+5
(3,10042778,10027c40,100136b2)
@[15] 0x10001873 nhttpstack.HTConfigurationThread::ThreadMain+355
(0,a71ffb8,60115334,10042778)
@[16] 0x10027c6a nhttpstack._HTThreadBeginProc@4+42 (10042778,0,0,0)
@[17] 0x60115334 nnotes._ThreadWrapper@4+212 (0,0,0,0)
[18] 0x77e4a990 KERNEL32.FlsSetValue+1913 (60115260,0,0,dddd04d2)
More >