Skip Ribbon Commands
Skip to main content
Navigate Up
Sign In

Quick Launch

Average Rating:

facebook Twitter
Email
Print Bookmark Alert me when this article is updated

Feedback

ERROR: “WRT_8165-TIMEOUT BASED COMMIT POINT” while running PowerCenter session on AIX 6.1
Problem Description

In PowerCenter, the following performance issues with AIX/Oracle are observed and all stacks point to Read/Write with I/O:

  • The server does not respond and the Integration Service is lost to the domain. Even though the Integration Service and node are running, the domain considers that they have stopped because the server was unresponsive for 20+ minutes. Also, the Integration Service running in the grid for that node is considered not running in Administration Console.
  • Workflow fails with the following error:

 “WRT_8165-TIMEOUT BASED COMMIT POINT”.

  • Informatica workflows run very slowly following an upgrade to the AIX server Operating System.

 Stack Trace:

iosl.select(??, ??, ??, ??, ??) at 0x900000000121dac
iceos.ICEOS::select(ICESelectHandle*,int,const ICETimeValue*)(??, ??, ??) at 0x900000005ac2474
connmodule.IAConnModule::run()(??) at 0x900000008e95f1c
connmodule.IAConnModule::startFunc(void*)(??) at 0x900000008e93d84
icethrmgr.mainThreadFcnWrapper(void*)(??) at 0x900000005ab9da4
Thread_Adapter.ACE_Thread_Adapter::invoke_i()() at 0x900000009c46188
Thread_Adapter.ACE_Thread_Adapter::invoke()() at 0x900000009c451bc
Base_Thread_Adapter.ace_thread_adapter() at 0x900000009c44c98

 

write.write(??, ??, ??) at 0x90000000002a8c4
ntt.nttwr(??, ??, ??, ??, ??) at 0x90000000e323988
nsnt.nsntwrn(??, ??, ??, ??, ??, ??) at 0x90000000e4bda20
nsp.nspsend(??, ??, ??, ??) at 0x90000000e4b8514
nsdo.nsdofls(0x112beb8d0, 0x112bed1c0, 0x112c1c150, 0x112c1c610, 0x0, 0x0) at 0x90000000e29234c
nsdo.nsdo(0x112bed078, 0x5400000054, 0x112bed1c0, 0x112bd1eb8, 0x112bd2660, 0x20, 0x300000003) at 0x90000000e296698
nsdo.nsdosend(0x112bed078, 0x5400000054, 0x112bed1c0, 0x0, 0x100000000000001, 0x2000000020, 0x300000003) at 0x90000000e2929ac
nio.nioqrc(??, ??, ??, ??, ??) at 0x90000000ec01c68
ttcdrv.ttcdrv(??, ??) at 0x90000000ec1f744
nioqwa.nioqwa(??, ??, ??, ??, ??, ??) at 0x90000000ec044f8
upiprv.upirtrc(0x112bd5bf0, 0x0, 0x0, 0x112bf13b8, 0x112bf2328, 0x9001000a2932bf0, 0x112bf2b28, 0x112bf4390) at 0x90000000e2786c0
kpurc.kpurcsc(??, ??, ??, ??, ??, ??, ??, ??) at 0x90000000e28b8ec
kpuex.kpuexecv8(??, ??, ??, ??, ??, ??) at 0x90000000e88d4e8
kpuex.kpuexec(??, ??, ??, ??, ??, ??, ??, ??) at 0x90000000e88ee60
oci8.OCIStmtExecute(??, ??, ??, ??, ??, ??, ??, ??) at 0x90000000e283100
oradriver.TOracleStatement::executenormal(int,long&)(??, ??, ??) at 0x90000000dfa9f18
oradriver.TOracleStatement::execute(int,long&)(??, ??, ??) at 0x90000000dfac790
stmtimpl.TStatementImpl::Execute(int,long&)(??, ??, ??) at 0x900000008db5e74
statemnt.TStatement::nExecute(unsigned int,unsigned int,long&)(??, ??, ??, ??) at 0x900000008d9a918
wrtreltgt.RelTarget::WriteInsertBatch(const SBlkDesc*,unsigned int,unsigned int,int,SNopArray*)(??, ??, ??, ??, ??, ??) at 0x9000000051a9874
wrtreltgt.RelTarget::Execute(const SBlkDesc*,int,unsigned long,int,const unsigned short*,int,int,int&,SNopArray*)(??, ??, ??, ??, ??, ??, ??, ??) at 0x9000000051a7274
wrt.Writer::run()(??) at 0x100525760
wrtrun.SWriterRunnable::handleRequest(const SManagerRequest&)(??, ??) at 0x10051efb0
worker.SWorker::run()(??) at 0x10038905c
runnable.SPETLRunnable::mainLoop()(??) at 0x100382c5c
dtmrun.SDTMRunnable::MainEntryPoint()(??) at 0x1003844c4
thread.runit(void*)(??) at 0x90000000f2250f8

Cause

This issue occurs when the following setting is done in the Oracle machine:

vm.nr_hugepages = 27008 in /etc/sysctl.conf

This setting reserves 50 GB of memory for hugepages and the resources are not sufficient for other server operations.

 Another issue observed: AIX 6.1 TL4 has performance issue.

Solution

To resolve this issue, remove the vm.nr_hugepages = 27008 in /etc/sysctl.conf setting and restore back the normal behavior. This will resolve if you see similar stack trace and behavior.

- Upgrade AIX 6.1 TL4 to AIX 6.1 TL6

More Information

More Stack trace:

  3 Thread 1426233664 (LWP 18080)  0x00002aae878026e2 in select ()
   from /lib64/libc.so.6
  2 Thread 1417840960 (LWP 6733)  0x00002aae878005f6 in poll ()
   from /lib64/libc.so.6
  1 Thread 46929110265104 (LWP 25036)  0x00002aae878026e2 in select ()
   from /lib64/libc.so.6

 

Thread 3 (Thread 1426233664 (LWP 18080)):
#0  0x00002aae878026e2 in select () from /lib64/libc.so.6
#1  0x00002aae8450c812 in ICEOS::select ()
   from /informatica/pmart861/server/bin/libpmICE.so
#2  0x00002aae8286d6d2 in IAConnModule::run ()
   from /informatica/pmart861/server/bin/libpminfapi.so
#3  0x00002aae8286f82d in IAConnModule::startFunc ()
   from /informatica/pmart861/server/bin/libpminfapi.so
#4  0x00002aae84363d95 in ACE_Thread_Adapter::invoke_i ()
   from /informatica/pmart861/server/bin/libACE.so.5.4.7
#5  0x00002aae84363cfa in ACE_Thread_Adapter::invoke ()
   from /informatica/pmart861/server/bin/libACE.so.5.4.7
#6  0x00002aae81481143 in start_thread () from /lib64/libpthread.so.0
#7  0x00002aae878088cd in clone () from /lib64/libc.so.6
#8  0x0000000000000000 in ?? ()

 

  3 Thread 1275164992 (LWP 7080)  0x00002b98426d193b in read ()
   from /lib64/libpthread.so.0
  2 Thread 1283557696 (LWP 7081)  0x00002b98426d1cc1 in recv ()
   from /lib64/libpthread.so.0
  1 Thread 47933094737872 (LWP 4371)  0x00002b98426cf1c6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

 

Thread 3 (Thread 1275164992 (LWP 7080)):
#0  0x00002b98426d193b in read () from /lib64/libpthread.so.0
#1  0x00002b98439111c3 in pmsys_read ()
   from /informatica/pmart861/server/bin/libpmuti.so
#2  0x00000000006891a9 in PSDiskFile::read ()
#3  0x0000000000672feb in PmCacheMgr::pin ()
#4  0x000000000065efcb in PmBPlusTree<PmRwBtreeCtrlTs>::search ()
#5  0x00000000007d5de4 in cacheAccess::lookupExactKey ()
#6  0x000000000085d117 in SLookupWidget::rowExec ()
#7  0x000000000085d841 in SLookupWidget::execRows ()
#8  0x000000000085dbac in SLookupWidget::blkExecute ()
#9  0x00000000008248be in SWidget::blkPush ()
#10 0x0000000000825190 in SWidget::blkPushDown ()
#11 0x000000000081183f in SExprWidget::blkExecute ()
#12 0x00000000008248be in SWidget::blkPush ()
#13 0x0000000000825190 in SWidget::blkPushDown ()
#14 0x00000000008255c4 in SWidget::blkFormSegmentAndPushDown ()
#15 0x00000000007c7f6f in TLeafExchangeWidget::PushSegment ()
#16 0x000000000079ba5b in STransformationRunnable::processOneBlk ()
#17 0x000000000079c3c2 in STransformationRunnable::run ()
#18 0x000000000079c535 in STransformationRunnable::handleRequest ()
#19 0x00000000005fef44 in SWorker::run ()
#20 0x00000000005d698d in SDTMRunnable::MainEntryPoint ()
#21 0x00002b98427e888a in runit ()
   from /informatica/pmart861/server/bin/libpmcef.so
#22 0x00002b98426cb143 in start_thread () from /lib64/libpthread.so.0
#23 0x00002b9849a0c8cd in clone () from /lib64/libc.so.6
#24 0x0000000000000000 in ?? ()

Applies To
Product: PowerCenter
Problem Type:
User Type: Administrator
Project Phase:
Product Version: PowerCenter
Database:
Operating System: AIX 6.1
Other Software:

Reference
Attachments
Last Modified Date:4/9/2014 2:02 AMID:120758
People who viewed this also viewed

Feedback

Did this KB document help you?



What can we do to improve this information (2000 or fewer characters)