SPR

Technote

Problem:
This issue was reported to Quality Engineering and has been addressed via SPR#
ASAR5SMKG9 in Domino 6.0.4 / 6.5.1 and 7.0 for UNIX/AIX.

Excerpt from the Lotus Notes and Domino Release 6.5.1 MR fix list (available at
http://www.ibm.com/developerworks/lotus):

Database
SPR# ASAR5SMKG9 - An occasional error condition would leave a semaphore locked
that would in turn cause many other threads and tasks to wait forever for the
semaphore to be unlocked, resulting in many console error messages from each
blocked thread complaining about excessive wait time. This fix performs the
necessary unlocking operation on the semaphore, allowing other threads to use
it.

This issue was reported to Quality Engineering and has been addressed via SPR#
TACN5QKJBA in Domino 6.0.4 / 6.5.2 and 7.0 for all other platforms.

Excerpt from the Lotus Notes and Domino Release 6.0.4 / 6.5.2 MR fix list
(available at http://www.ibm.com/developerworks/lotus):

SPR #TACN5QKJBA - Fixed a potential server hang when a thread does not properly
unlock the NSF Buffer Pool semaphore on an error path.


Supporting Information:
The call stack for the program at the time of the hang is as follows:

Job: 111535/QNOTES /SERVER
|
|
Thread ID: 00000000000001A9
|
Pgm Name Mod Name Statement Procedure Name
---------- ---------- ---------- ----------
------------------------------------------------------------
QLESPI QLECRTTH 0000001112
LE_Create_Thread2__FP12crtth_parm_t
QP0WPINT QP0WSPTHR 0000002459 pthread_create_part2
LIBNOTES THREAD 0000000016 ThreadWrapper
SERVER SCHED 0000000057 Scheduler
SERVER WORKTHRD 0000000158 WorkThreadTask
SERVER DBSERVER 0000000282 DbServer
SERVER MISCNSF 0000000020 ServerMiscFunction
LIBNOTES DBCLOSE 0000000001 NSFDbClose
LIBNOTES DBCLOSE 0000000001 NSFDbCloseExtended
LIBNOTES DBCLOSE 0000000001 NSFDbCloseExtended2
LIBNOTES DBCLOSE 0000000009 NSFDbCloseExtended3
LIBNOTES DBCLOSE 0000000219 DbClose
LIBNOTES DOCCACHE 0000000002 DocCacheClose
LIBNOTES DOCCACHE 0000000001 DocCacheDeaccess
LIBNOTES DOCCACHE 0000000020 DocCacheEnter
LIBNOTES NSFSEM5 0000000021 NSFBufCloseStorage
LIBNOTES DBBUF 0000000009 iNSFBufCloseStorage
LIBNOTES OSSEM 0000000001 OSLockSem
LIBNOTES OSSEM 0000000045 WaitOnNativeSemaphore
LIBNOTES OSSEM 0000000013 WaitForThreadSem
QP0WPTHR QP0WCOND 0000002475 pthread_cond_timedwait
QP0WPINT QP0WSCOND 0000002113
wait__20Qp0wPthreadConditionFP7Qp0wTcbP9Qp0wMutexiT3
QP0WPINT QP0WTCB 0000002251 blockMyThread__7Qp0wTcbFiT1


Also, the following tracebacks and Log snippet were seen on Domino 6.5.1 for
zSeries, z/OS 1.4:

Tracebacks pulled from the SVC dump:
*** waiting for "directory manager queue" semaphore (owner=chronos) ***
__semop_timed
OSLockSemInt +440
OSLockWriteSem +A0
DbDirManQueueUpdate +E4
RemoveFromCache +8D4
StateMachine +168
LookupInDbCache +C4C
DbUnReserveInDbCache +30
NSFDbOpenExtended3 +6124
OpenDB +6BE
ServerOpenDb +2F8
DbServer +FF0

*** waiting for "NSF buffer pool container" semaphore (owner=server)
__semop_timed
OSLockSemInt +440
OSLockSem +16
iNSFBufCloseStorage +78
NSFBufCloseStorage +190
DocCacheClose +EC
DbClose +F76
NSFDbClose +9A
DbServerTerm +C10
DeleteTask +10E
WorkThreadTask +C9E

*** waiting for "Open database queue" semaphore (owner=adminp)
__semop_timed
OSLockWriteFRWSemInt +5B4
OSLockWriteFRWSem +1A
LockDbQueue +1C
NSFDbOpenExtended3 +9A6
ReloadDirAssist +4EA
EnsureDesignCollectionsOpen +DC
Lookup +33E
NAMELookupThread +2B0

Notes Log Snippet:
~~~
33554658:00002--503250943¨ 01-10-2004 16:38:42 Chronos: Error full
text indexing mail12/dmdflu.nsf: Message Queue is full.
~~~
[50331973:00002-1157693441] 01-10-2004 18:38:03 Chronos: Performing
hourly full text indexing
[16777352:00007--2013200379] 01-10-2004 18:38:04 Router: Message
005B40E4 delivered to JOBS
ti="005B692C-C1256F20" sq="0007052F" THREAD [16777263:00266-788594698]
WAITING FOR SEM 0x0256 NSF buffer pool container (@2D840F40)
(OWNER=16777263:788594698) FOR 30000 ms
~~~
FOR RWSEM 0x0294 Directory manager queue semaphore (@15FF14A4)
(R=1,W=2,WRITER=50331973:1157693441,1STREADER=50331973:1157693441) FOR
~~~
'....@..V........'LkMgr BEGIN Long Held Lock Dump ------------------
[16777263:00133-788594816] Lock(Mode=SIX* LockID(DB
DB=/notesdata/mail2.box)) Waiters countNonIntentLocks = 1
countIntentLocks = 0, queuLength = 1
~~~
ti="005B8740-C1256F20" sq="0007055B" THREAD [00063:00004-1057030146]
WAITING FOR RWSEM 0x0294 Directory manager queue semaphore (@15FF14A4)
(R=1,W=2,WRITER=50331973:1157693441,1STREADER=50331973:1157693441) FOR
30000 ms

Tran=0 Func=N/A [16777352:00002--2013200383] Delay=112min)
[16777263:00133-788594816] LkMgr END Long Held Lock Dump
----------------------- More >